BBC Gardeners' World presenter Monty Don revealed details of the awkward exchange after announcing his surprise exit from the show