The former I’m A Celebrity… Get Me Out Of Here! star shared his true thoughts on presenters Ant and Dec after many viewers claimed Ant was "harsh" on him