Labour, the Conservatives, the Lib Dems and Restore have all ruled out standing, in an effective boycott of the by-election.