The actor-producer told Theo Von that the late night shows "started feeling like I was in a f***ing class I didn’t want to take."