The Top Gear and Clarkson's Farm host made the shock admission during the final two episodes of season five