Top Dems are expected to skip his speech but invited Epstein accusers and ICE detainees in their place