PM suggested this weekend that a return to freedom of movement could be in Labour’s next election manifesto