The Clarkson's Farm star defended himself after a fan claimed he is "not strictly a farmer" and "everybody knows it"