Influenza antivirals can directly benefit treated individuals by reducing the risk of some influenza-associated complications when initiated soon after symptom onset. Antivirals may also reduce onward transmission, and previous modeling studies including such indirect effects have estimated large population-level reductions in influenza-associated disease outcomes. However, these studies often assume an optimistic number of people initiate, and adhere to, prompt treatment. Using a compartmental framework, we modeled population-level reductions in influenza-associated hospitalizations given potential direct and indirect effects of influenza antivirals in the United States. From reported patterns of care-seeking, antiviral prescribing, and treatment adherence, we estimated 1.1-9.7% of symptomatic individuals initiate and adhere to prompt antiviral treatment. We estimated small direct effects of antivirals in example seasons, with 0.9% of hospitalizations prevented when antivirals reduced hospitalization risk by 20%. Assuming antivirals had direct and indirect effects generated slightly larger impacts. For example, up to 3.8% of hospitalizations were prevented when antivirals reduced hospitalization risk and onward transmission by 20%. More substantial impacts (for example, preventing >20% of hospitalizations) were only achieved by increasing initiation and adherence to prompt treatment beyond typically observed levels. Therefore, prompt treatment initiation and adherence are critical factors shaping potential population-level impacts of influenza antivirals.