The Celebrity Traitors star opened up about how it was 'weird' that he wasn't liked by fellow cast members on the show's second series