The US president said he was unsure if the state visit will repair his fraught relationship with the prime minister