As the food writer joined guests at the King's private residence to celebrate his Chelsea garden, she revealed memories of her own time on the Gloucestershire estate