Background We aimed to characterize substance use patterns, and their correlates, among adults with and without HIV at two healthcare facilities in Cape Town, South Africa. Methods We recruited adults with and without HIV from two primary healthcare facilities. We assessed alcohol use with the AUDIT-C and other substance use with the WHO-ASSIST questionnaires. We classified participants with no lifetime drug use and no or low-risk alcohol use as "low risk" and fitted latent class models among remaining participants to identify additional substance use patterns. We used multinomial logistic regression to examine factors associated with substance use patterns. Results Among 602 participants, 265 (44.0%) were "low risk". Latent class analysis among the remaining 337 identified three patterns: "higher-risk alcohol use" (N=168; 27.9%), "cannabis use" (universal cannabis use with mixed alcohol risk; N=114; 18.9%), and "mixed illicit drug use" (predominantly no or low-risk alcohol use with use of multiple drugs; N=55; 9.1%). Participants younger than 45 years were less likely to be in the low-risk group. Male sex was associated with lower probability of the low-risk group and higher probability of the higher-risk alcohol use and cannabis use classes. HIV was associated with higher probability of higher-risk alcohol use. One facility had a substantially higher probability of mixed illicit drug use. Conclusions We identified distinct higher-risk substance use profiles that varied strongly by age, sex, and facility. These findings support integrating alcohol and substance use screening and management into primary care and tailoring responses to local substance use patterns.