Sir Tom expressed his anger last week as he revealed it hadn't been his choice to leave The Voice, saying there's 'never a good time to fire an 86-year-old'