‘At the time, it was a very difficult decision to make,’ actor and writer said during Henley Literary Festival panel