Presenter revealed that he and former co-hosts Jeremy Clarkson and James May had a pact with their executive producer to all leave together, even if one of them didn’t want to go