The Prime Minister expressed an openness to an apology regarding slavery, while firmly ruling out reparations