Britain's Got Talent judge Amanda Holden claimed her dilemma was "blatantly ignored" by the American singer after being invited onto his show