The former Tottenham Hotspur boss has been speaking about his time in north London and how it eventually came to an end