"The only thing I know right now is that there’s never a good time to fire an 86-year-old who’s still pretty good at his job," the beloved singer said.