A 'yes' vote would force the government to restrict asylum, residency permits, and scrap EU deal on free movement.