Gardeners' World presenter Monty Don opened up about the "huge stick" he received, just days before announcing his exit from the BBC show