The president could be on track to lose major gains he made with two key groups of voters in 2024, the poll indicated