PM’s comments cast doubt on Labour’s election pledge to ‘smash the gangs’ and curb illegal Channel crossings