One of the famous faces from Celebrity Traitors has opened up about what one of their major worries was while they were on the show