Adam Boulton has been urged to retract the ‘vile remarks’ that he made about the former Tory minister