Artist Chris Ashmore said the criticism ‘didn’t feel fair’ with three weeks still to go on the project