Contrastive focus has been shown to enhance memory for focused elements and their contextual alternatives. However, recent work has shown that the memory effects of contrastive accenting vary across individuals. The present study investigates whether individual differences in memory for focus-related information reflect distinct recall patterns and which linguistic or cognitive abilities underlie these patterns. Native speakers of German completed a delayed-recall task in which contrastive focus was marked by pitch accenting, along with a battery of predictor tasks targeting different aspects of focus processing as well as broader linguistic and cognitive abilities. Using a probabilistic clustering approach, we identified two recall profiles that differed primarily in overall recall performance for focused elements and their alternatives. Participants who recalled more focus-related information also performed better on inference-based tasks, whereas prosodic perception and domain-general cognitive abilities did not predict recall profiles. These findings suggest that individual variability in the memory effects of contrastive focus is closely associated with inference-making ability and indicate that the encoding of contrast sets may be influenced by listener-specific inferential abilities.