‘I don't go to places where I’m not welcome,’ said ‘Pose’ actor, who hasn’t attended the star-studded event since 2019