The Tory leader insisted that voters were not interested ‘in having a row’ over the debate to rejoin the EU