Campaigners told the Home Affairs Select Committee that voters are 'up in arms' about the proposals and do not believe it is just about tackling illegal migration