It's not the first time the comedian has spoken out about his friendship with Britain’s Got Talent judge Amanda Holden, after fans raised the same question