AbstractArtificial intelligence (AI) is increasingly used in healthcare to identify high-risk patients. Laboratory studies are mixed regarding patient attitudes towards AI use. Here we field-tested whether informing patients of their risk for influenza and its complications can serve as an effective behaviour-change intervention to increase vaccination, and whether disclosing AI use in the risk determination or providing personalized reasons for the algorithm’s prediction (a form of explainable AI) influence message effectiveness. We ran three preregistered randomized controlled trials with over 90,000 unique healthcare system patients in the USA (ClinicalTrials.gov identifiers: NCT04323137, NCT05009251 and NCT05509283). Patients identified by a previously validated machine learning algorithm as being at high risk for influenza and related complications were randomized to be sent no message or one of several different messages encouraging influenza vaccination (the primary outcome). High-risk nudges increased vaccination: among patients informed of their high risk, vaccination was 1.1–1.4 percentage points (3.3–5.4%) higher than those simply reminded to get a vaccine, and 1.7–3.5 percentage points (3.3–14.7%) higher than non-messaged patients. Vaccination was similar across message arms that did versus did not disclose ‘algorithm’ involvement or its risk explanations, indicating that patients are neither averse to nor appreciative of algorithm use or explainability in this realistic health application. This work was partially funded by the National Institute on Aging of the National Institutes of Health under award number P30AG034532.This is a preview of subscription content, access via your institutionAccess options Access through your institutionAccess Nature and 54 other Nature Portfolio journalsGet Nature+, our best-value online-access subscription27,99 € / 30 dayscancel any timeLearn moreSubscribe to this journalReceive 12 digital issues and online access to articles118,99 € per yearonly 9,92 € per issueLearn moreBuy this articlePurchase on SpringerLinkInstant access to the full article PDF.39,95 €Prices may be subject to local taxes which are calculated during checkoutFig. 1: Study 1 vaccination timing and differences in vaccination by study arm.Fig. 2: Study 2 vaccination increases compared with passive control.Fig. 3: Difference in vaccination rates by arm and number of risk reasons in study 2.Fig. 4: Study 3 vaccination rate increases compared with passive control.Data availabilityDe-identified data necessary to reproduce all results reported here are posted on OSF at https://osf.io/mqbux/. All data are posted at the individual level, except vaccination timing data (that is, vaccination dates) which are aggregated by experimental condition because dates of service are Protected Health Information under 45 CFR §164.514(b). Related documents available include study protocols and statistical analysis plans (available on ClinicalTrials.gov registrations, study 1: https://clinicaltrials.gov/ct2/show/NCT04323137; study 2: https://clinicaltrials.gov/ct2/show/NCT05009251; study 3: https://clinicaltrials.gov/ct2/show/NCT05509283; study 4: https://clinicaltrials.gov/ct2/show/NCT05509270).Code availabilityCode necessary to reproduce all results reported here is available via OSF at https://osf.io/mqbux/.ReferencesBronsoler, A., Doyle, J. J. & Van Reenen, J. M. The impact of healthcare it on clinical quality, productivity and workers. SSRN Electron. J. 14, 23–46 (2021).Google Scholar Rajpurkar, P., Chen, E., Banerjee, O. & Topol, E. J. AI in health and medicine. Nat. Med. 28, 31–38 (2022).Article CAS PubMed Google Scholar Goh, K. H. et al. Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare. Nat. Commun. 12, 711 (2021).Article CAS PubMed PubMed Central Google Scholar Tomašev, N. et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature 572, 116–119 (2019).Article PubMed PubMed Central Google Scholar Young, A. T., Amara, D., Bhattacharya, A. & Wei, M. L. Patient and general public attitudes towards clinical artificial intelligence: a mixed methods systematic review. Lancet Digit. Health 3, e599–e611 (2021).Article CAS PubMed Google Scholar Logg, J. M., Minson, J. A. & Moore, D. A. Algorithm appreciation: people prefer algorithmic to human judgment. Organ. Behav. Hum. Decis. Process. 151, 90–103 (2019).Article Google Scholar Dietvorst, B. J., Simmons, J. P. & Massey, C. Algorithm aversion: people erroneously avoid algorithms after seeing them err. J. Exp. Psychol. Gen. 144, 114–126 (2015).Article PubMed Google Scholar Longoni, C., Bonezzi, A. & Morewedge, C. K. Resistance to medical artificial intelligence. J. Consum. Res. 46, 629–650 (2019).Article Google Scholar Fridman, A., Gershon, R. & Gneezy, A. COVID-19 and vaccine hesitancy: a longitudinal study. PLoS ONE 16, e0250123 (2021).Article CAS PubMed PubMed Central Google Scholar MacIntyre, C. R., Mahimbo, A., Moa, A. M. & Barnes, M. Influenza vaccine as a coronary intervention for prevention of myocardial infarction. Heart 102, 1953–1956 (2016).Article PubMed PubMed Central Google Scholar Warren-Gash, C., Smeeth, L. & Hayward, A. C. Influenza as a trigger for acute myocardial infarction or death from cardiovascular disease: a systematic review. Lancet Infect. Dis. 9, 601–610 (2009).Article PubMed Google Scholar People at increased risk for flu complications. Centers for Disease Control and Prevention https://www.cdc.gov/flu/highrisk/index.htm (2022).Goeijenbier, M. et al. Benefits of flu vaccination for persons with diabetes mellitus: a review. Vaccine 35, 5095–5101 (2017).Article CAS PubMed Google Scholar Nichol, K. L. Efficacy and effectiveness of influenza vaccination. Vaccine 26, D17–D22 (2008).Article CAS PubMed Google Scholar Troeger, C. E. et al. Mortality, morbidity, and hospitalisations due to influenza lower respiratory tract infections, 2017: an analysis for the Global Burden of Disease Study 2017. Lancet Respir. Med. 7, 69–89 (2019).Article Google Scholar Bonezzi, A., Ostinelli, M. & Melzner, J. The human black-box: the illusion of understanding human better than algorithmic decision-making. J. Exp. Psychol. Gen. 151, 2250–2258 (2022).Article PubMed Google Scholar Cadario, R., Longoni, C. & Morewedge, C. K. Understanding, explaining, and utilizing medical artificial intelligence. Nat. Hum. Behav. 5, 1636–1642 (2021).Article PubMed Google Scholar Pezzo, M. V. & Beckstead, J. W. Patients prefer artificial intelligence to a human provider, provided the AI is better than the human: a commentary on Longoni, Bonezzi and Morewedge (2019). Judgm. Decis. Mak. 15, 443–445 (2020).Adadi, A. & Berrada, M. Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018).Article Google Scholar Miller, T. Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2019).Article Google Scholar Mercado, J. E. et al. Intelligent agent transparency in human-agent teaming for multi-UxV management. Hum. Factors 58, 401–415 (2016).Article PubMed Google Scholar Stubbs, K., Wettergreen, D. & Hinds, P. J. Autonomy and common ground in human–robot interaction: a field study. IEEE Intell. Syst. 22, 42–50 (2007).Article Google Scholar Benartzi, S. et al. Should governments invest more in nudging?. Psychol. Sci. 28, 1041–1055 (2017).Article PubMed PubMed Central Google Scholar Patel, M. S., Volpp, K. G. & Asch, D. A. Nudge units to improve the delivery of health care. New Engl. J. Med. 378, 214–216 (2018).Article PubMed PubMed Central Google Scholar Dai, H. et al. Behavioural nudges increase COVID-19 vaccinations. Nature 597, 404–409 (2021).Article CAS PubMed PubMed Central Google Scholar Milkman, K. L. et al. A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment. Proc. Natl Acad. Sci. USA 118, 10–12 (2021).Article Google Scholar Milkman, K. L. et al. A 680,000-person megastudy of nudges to encourage vaccination in pharmacies. Proc. Natl Acad. Sci. USA 119, e2115126119 (2022).Article CAS PubMed PubMed Central Google Scholar Santos, H. C., Goren, A., Chabris, C. F. & Meyer, M. N. Effect of targeted behavioral science messages on COVID-19 vaccination registration among employees of a large health system: a randomized trial. JAMA Netw. Open 4, 5–8 (2021).Article Google Scholar Schneider, F. H. et al. Financial incentives for vaccination do not have negative unintended consequences. Nature 613, 526–533 (2023).Article CAS PubMed PubMed Central Google Scholar Yokum, D., Lauffenburger, J. C., Ghazinouri, R. & Choudhry, N. K. Letters designed with behavioural science increase influenza vaccination in Medicare beneficiaries. Nat. Hum. Behav. 2, 743–749 (2018).Article PubMed Google Scholar Chen, N., Trump, K.-S., Hall, S. & Le, Q. The effect of postcard reminders on vaccinations among the elderly: a block-randomized experiment. Behav. Public Policy 7, 240–265 (2020).Article Google Scholar Rabb, N. et al. Evidence from a statewide vaccination RCT shows the limits of nudges. Nature 604, E1–E7 (2022).Article CAS PubMed Google Scholar Szilagyi, P. G. et al. Effect of personalized messages sent by a health system’s patient portal on influenza vaccination rates: a randomized clinical trial. J. Gen. Intern. Med. 37, 615–623 (2022).Article PubMed Google Scholar Modin, D. et al. Effect of electronic nudges on influenza vaccination rate in older adults with cardiovascular disease: prespecified analysis of the NUDGE-FLU trial. Circulation 147, 1345–1354 (2023).Article PubMed Google Scholar Johansen, N. D. et al. Electronic nudges to increase influenza vaccination uptake among patients with heart failure: a prespecified analysis of the NUDGE-FLU trial. Eur. J. Heart Fail. 25, 1450–1458 (2023).Brewer, N. T., Chapman, G. B., Rothman, A. J., Leask, J. & Kempe, A. Increasing vaccination: putting psychological science into action. Psychol. Sci. Public Interest 18, 149–207 (2017).Article PubMed Google Scholar Sheeran, P., Harris, P. R. & Epton, T. Does heightening risk appraisals change people’s intentions and behavior? A meta-analysis of experimental studies. Psychol. Bull. 140, 511–543 (2014).Article PubMed Google Scholar Motta, M., Sylvester, S., Callaghan, T. & Lunz-Trujillo, K. Encouraging COVID-19 vaccine uptake through effective health communication. Front. Polit. Sci. 3, 630133 (2021).Article Google Scholar Reñosa, M. D. C. et al. Nudging toward vaccination: a systematic review. BMJ Glob. Health 6, e006237 (2021).Article PubMed PubMed Central Google Scholar French, D. P., Cameron, E., Benton, J. S., Deaton, C. & Harvie, M. Can communicating personalised disease risk promote healthy behaviour change? A systematic review of systematic reviews. Ann. Behav. Med. 51, 718–729 (2017).Article PubMed PubMed Central Google Scholar Schulberg, S. D. et al. Cardiovascular risk communication strategies in primary prevention. A systematic review with narrative synthesis. J. Adv. Nurs. 78, 3116–3140 (2022).Article PubMed PubMed Central Google Scholar Brewer, N. T. What works to increase vaccination uptake. Acad. Pediatr. 21, S9–S16 (2021).Article PubMed Google Scholar Tuckerman, J. et al. Short message service reminder nudge for parents and influenza vaccination uptake in children and adolescents with special risk medical conditions. JAMA Pediatr. 177, 337–344 (2023).Article PubMed PubMed Central Google Scholar Johansen, N. D. et al. Electronic nudges to increase influenza vaccination uptake in Denmark: a nationwide, pragmatic, registry-based, randomised implementation trial. Lancet 401, 1103–1114 (2023).Article PubMed Google Scholar Sääksvuori, L. et al. Information nudges for influenza vaccination: evidence from a large-scale cluster-randomized controlled trial in Finland. PLoS Med. 19, e1003919 (2022).Article PubMed PubMed Central Google Scholar Wolk, D. M. et al. Prediction of influenza complications: development and validation of a machine learning prediction model to improve and expand the identification of vaccine-hesitant patients at risk of severe influenza complications. J. Clin. Med. 11, 4342 (2022).Article PubMed PubMed Central Google Scholar La Macchia, S. T., Louis, W. R., Hornsey, M. J. & Leonardelli, G. J. In small we trust: lay theories about small and large groups. Pers. Soc. Psychol. Bull. 42, 1321–1334 (2016).Article PubMed Google Scholar Rudert, S. C., Reutner, L., Greifeneder, R. & Walker, M. Faced with exclusion: perceived facial warmth and competence influence moral judgments of social exclusion. J. Exp. Soc. Psychol. 68, 101–112 (2017).Article Google Scholar Clark, J. K., Thiemb, K. C., Hoover, A. E. & Habashi, M. M. Gender stereotypes and intellectual performance: stigma consciousness as a buffer against stereotype validation. J. Exp. Soc. Psychol. 68, 185–191 (2017).Article Google Scholar Mann, T. C. & Ferguson, M. J. Reversing implicit first impressions through reinterpretation after a two-day delay. J. Exp. Soc. Psychol. 68, 122–127 (2017).Article PubMed Google Scholar Filiz, I., Judek, J. R., Lorenz, M. & Spiwoks, M. The extent of algorithm aversion in decision-making situations with varying gravity. PLoS ONE 18, e0278751 (2023).Waters, E. A. et al. Translating cancer risk prediction models into personalized cancer risk assessment tools: stumbling blocks and strategies for success. Cancer Epidemiol. Biomarkers Prev. 29, 2389–2394 (2020).Article CAS PubMed PubMed Central Google Scholar Hasselblad, V. & Hedges, L. V. Meta-analysis of screening and diagnostic tests. Psychol. Bull. 117, 167–178 (1995).Article CAS PubMed Google Scholar Castelo, N., Bos, M. W. & Lehmann, D. R. Task-dependent algorithm aversion. J. Mark. Res. 56, 809–825 (2019).Article Google Scholar Saccardo, S. et al. Field testing the transferability of behavioural science knowledge on promoting vaccinations. Nat. Hum. Behav. 8, 878–890 (2024).Article PubMed PubMed Central Google Scholar DellaVigna, S. & Linos, E. RCTs to scale: comprehensive evidence from two nudge units. Econometrica 90, 81–116 (2022).Article Google Scholar Service, O. et al. EAST: four simple ways to apply behavioural insights. Annu. Rev. of Policy Des. 5, 1–53 (2017).Jacobson, M., Chang, T. Y., Shah, M., Pramanik, R. & Shah, S. B. Can financial incentives and other nudges increase COVID-19 vaccinations among the vaccine hesitant? A randomized trial. Vaccine 40, 6235–6242 (2022).Article PubMed PubMed Central Google Scholar Campos-Mercade, P. et al. Monetary incentives increase COVID-19 vaccinations. Science 374, 879–882 (2021).Article CAS PubMed PubMed Central Google Scholar Bryan, C. J., Tipton, E. & Yeager, D. S. Behavioural science is unlikely to change the world without a heterogeneity revolution. Nat. Hum. Behav. 5, 980–989 (2021).Article PubMed PubMed Central Google Scholar Brody, I. et al. Targeting behavioral interventions based on past behavior: evidence from vaccine uptake. Organ. Behav. Hum. Decis. Process. 192, 104465 (2026).Article Google Scholar de Ridder, D., Kroese, F. & van Gestel, L. Nudgeability: mapping conditions of susceptibility to nudge influence. Perspect. Psychol. Sci. 17, 346–359 (2022).Article PubMed Google Scholar Chater, N. & Loewenstein, G. The i-frame and the s-frame: how focusing on individual-level solutions has led behavioral public policy astray. Behav. Brain Sci. 46, e147 (2022).Funk, C., Hefferon, M., Kennedy, B. & Johnson, C. Trust and Mistrust in Americans’ Views of Scientific Experts. https://www.pewresearch.org/science/2019/08/02/trust-and-mistrust-in-americans-views-of-scientific-experts/ (Pew Research Center, 2019).Pasquini, G., Stocking, G., Kikuchi, E., Pula, I. & Kirzinger, A. Where Do Americans Get Health Information, and What Do They Trust? https://www.pewresearch.org/science/2026/04/07/where-do-americans-get-health-information-and-what-do-they-trust/ (Pew Research Center, 2026).Saccardo, S. et al. Scaling Nudges: Who Moves and How. https://doi.org/10.2139/ssrn.3971192 (SSRN, 2025).R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing https://www.r-project.org (2021).Coppock, A. randomizr: easy-to-use tools for common forms of random assignment and sampling. Comprehensive R Archive Network (CRAN) https://cran.r-project.org/web/packages/randomizr/index.html (2024).Colling, L. J. bayesplay: the bayes factor playground. R package version 0.9.3. GitHub https://github.com/bayesplay/bayesplay (2023).Rich, B. table1: tables of descriptive statistics in HTML. CRAN: Contributed Packages https://doi.org/10.32614/CRAN.package.table1 (2025).Wickham, H. stringr: simple, consistent wrappers for common string operations. Comprehensive R Archive Network (CRAN) https://cran.r-project.org/web/packages/stringr/index.html (2025).Lüdecke, D. sjPlot: data visualization for statistics in social science. Comprehensive R Archive Network (CRAN) https://CRAN.R-project.org/package=sjPlot (2025).Coppock, A. randomizr: easy-to-use tools for common forms of random assignment and sampling. Comprehensive R Archive Network (CRAN) https://cran.r-project.org/web/packages/randomizr/index.html (2025).Champely, S. pwr: basic functions for power analysis. GitHub https://github.com/heliosdrm/pwr (2023).Wickham, H. The split-apply-combine strategy for data analysis. J. Stat. Softw. 40, 1–29 (2011).Article Google Scholar Sievert, C. Interactive Web-Based Data Visualization with R, Plotly, and Shiny (CRC, 2020).Rizopoulos, D. ltm: an R package for latent variable modeling and item response theory analyses. J. Stat. Softw. 17, 1–25 (2006).Grolemund, G. & Wickham, H. Dates and times made easy with {lubridate}. J. Stat. Softw. 40, 1–25 (2011).Article Google Scholar Long, J. A. jtools: analysis and presentation of social scientific data. J. Open Source Softw. 9, 6610 (2024).Article Google Scholar Hugh-Jones, D. ggmagnify: create a magnified inset of part of a ‘Ggplot’ object. GitHub https://github.com/hughjonesd/ggmagnify (2026).Wickham, H. et al. Welcome to the Tidyverse. J. Open Source Softw. 4, 1686 (2019).Article Google Scholar Lenth, R. V. & Piaskowski, J. emmeans: estimated marginal means, aka least-squares means. Comprehensive R Archive Network (CRAN) https://cran.r-project.org/web/packages/emmeans/index.html (2026).Gomila, R. Logistic or linear? Estimating causal effects of experimental treatments on binary outcomes using regression analysis. J. Exp. Psychol. Gen. 150, 700–709 (2021).Article PubMed Google Scholar Hellevik, O. Linear versus logistic regression when the dependent variable is a dichotomy. Qual. Quant. 43, 59–74 (2009).Article Google Scholar Wetzels, R. et al. Statistical evidence in experimental psychology: an empirical comparison using 855 t tests. Perspect. Psychol. Sci. 6, 291–298 (2011).Article PubMed Google Scholar Dienes, Z. Using Bayes to get the most out of non-significant results. Front. Psychol. 5, 781 (2014).Download referencesAcknowledgementsWe thank A. Lanyado, R. Yesharim, E. Hertzman, E. Choman, C. Metzger, E. Cohen and T. Sagiv (originally at Medial EarlySign) for discussions about study design and data analysis in relation to the AI algorithm, and for their assistance in generating reasons that patients were considered to be high risk. We thank S. Martin and K. Murphy for their invaluable support of this work, C. Walter and P. Juthani for research assistance, and S. Brietzke for help de-identifying survey data. We also thank Geisinger Marketing for help preparing and sending the messages. Finally, we thank members of Geisinger Business Intelligence and Advanced Analytics, especially C. Hartranft, G. Strevig, C. Cauthorn and E. Reich, for data assistance, and the Geisinger Data Core and R. Maff for their assistance liaising between Geisinger and Medial EarlySign.FundingStudies 1 and 2 were supported by the National Institute on Aging of the National Institutes of Health under award number P30AG034532 (which provided salary support to G.M.R., A.G., M.S., J.J.D., M.N.M. and C.F.C.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the paper, beyond approval of the initially proposed designs for the field trials.Author informationAuthor notesThese authors contributed equally: Michelle N. Meyer, Christopher F. Chabris.These authors jointly supervised this work: Michelle N. Meyer, Christopher F. Chabris.Authors and AffiliationsBehavioral Insights Team, Geisinger, Danville, PA, USAGail M. Rosenbaum, Amir Goren, Maheen Shermohammed, Michelle N. Meyer & Christopher F. ChabrisDepartment of Laboratory Medicine, Diagnostic Medicine Institute, Geisinger, Danville, PA, USADonna M. Wolk & Ann Marie TiceGeisinger Commonwealth School of Medicine, Danville, PA, USADonna M. WolkSloan School of Management, Massachusetts Institute of Technology, Cambridge, MA, USAJoseph J. Doyle Jr.National Bureau of Economic Research, Cambridge, MA, USAJoseph J. Doyle Jr.Department of Bioethics and Decision Sciences, Geisinger, Danville, PA, USAMichelle N. Meyer & Christopher F. ChabrisAuthorsGail M. RosenbaumView author publicationsSearch author on:PubMed Google ScholarAmir GorenView author publicationsSearch author on:PubMed Google ScholarMaheen ShermohammedView author publicationsSearch author on:PubMed Google ScholarDonna M. WolkView author publicationsSearch author on:PubMed Google ScholarAnn Marie TiceView author publicationsSearch author on:PubMed Google ScholarJoseph J. Doyle Jr.View author publicationsSearch author on:PubMed Google ScholarMichelle N. MeyerView author publicationsSearch author on:PubMed Google ScholarChristopher F. ChabrisView author publicationsSearch author on:PubMed Google ScholarContributionsG.M.R., A.G., M.S., J.J.D., M.N.M. and C.F.C. conceptualized the project; J.J.D., M.N.M. and C.F.C. acquired funding; G.M.R., M.S., A.G., A.M.T., D.M.W., J.J.D., M.N.M. and C.F.C. designed methodology; G.M.R., M.S. and A.G. performed the investigations and administrated the project; G.M.R. and M.S. curated the data, undertook formal analysis and visualized the data; A.G., J.J.D., M.N.M. and C.F.C. supervised the work; G.M.R., A.G. and M.S. wrote the original draft; G.M.R., M.S., A.G., A.M.T., D.M.W., J.J.D., M.N.M. and C.F.C. contributed to the review and editing of the paper.Corresponding authorCorrespondence to Christopher F. Chabris.Ethics declarationsCompeting interestsGeisinger paid Medial EarlySign to generate the risk scores and reasons used in the studies. The authors declare no other competing interests.Peer reviewPeer review informationNature Human Behaviour thanks Mark Pezzo and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.Additional informationPublisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Extended dataExtended Data Fig. 1 Vaccination rate differences in Study 1 by actual and communicated risk levels.Differences in the Study 1 primary outcome are shown separately by actual risk level, communicated risk level, and study arm. The vertical dashed line at 0% represents the overall passive control vaccination rate across risk level of 50.8% shown in Fig. 1b. All patients who were sent messages in the top 3% of risk were told they were “in the top 3% of risk.” Patients in the top 4–10% of risk were randomized to be told they were “in the top 10% of risk” or that they were “at high risk.” Vaccination rates in the passive control group by risk level are shown for comparison. Sample sizes for the groups from top to bottom are 2,906, 2,867, 2,855, 2,863, 6,734, 3,378, 3,452, 3,377, 3,389, and 3,380. Error bars indicate 95% confidence intervals.Extended Data Fig. 2 Study 2 vaccination timing.The number of patients vaccinated by study arm for each business day relative to the intervention start date overlayed with local-linear regression lines estimated separately before (8/1/2021–9/8/2021) and on or after (9/9/2021–11/5/2021) the intervention start date. Send dates for each modality are indicated by the vertical, dotted lines. Patients in experimental arms who were eligible for SMS received either an early SMS or a late SMS (see Methods for details). Sample sizes in each arm: passive control n = 8,466, active control n = 8,522, high risk only n = 8,498, high risk medical records n = 8,470, and high risk algorithm n = 8,525.Extended Data Fig. 3 Vaccination rate differences in Study 2 by actual and communicated risk levels.Differences in the Study 2 primary outcome separately by actual risk level, communicated risk level, and study arm. The vertical dashed line at 0% represents the overall passive control vaccination rate across risk level of 31.5% shown in Fig. 2. Passive control vaccination rates by risk level are displayed for comparison with active arms. Sample sizes in each group, from top to bottom: 2,565, 2,581, 2,551, 2,555, 2,561, 5,901, 5,941, 5,947, 5,915, and 5,964. Error bars indicate 95% confidence intervals.Extended Data Fig. 4 Study 3 vaccination timing.The number of patients vaccinated by study arm for each business day relative to the intervention start date. The intervention ran from 9/13/2022 through 10/25/2022 for patients in the active control and algorithm arms and 9/14/2022 through 10/26/2022 for patients in the high risk only and high risk medical records arms. Those in the passive control arm were randomized to one of those two timeframes (see Methods for more details). The plot includes local-linear regression lines estimated separately before (8/1/2022–9/12/2022 or 8/1/2022–9/13/2022) and on or after (9/13/2022–10/25/2022 or 9/14/2022–10/26/2022) the intervention start date. Send dates for each modality are indicated by the vertical, dotted lines. Sample sizes in each arm: passive control n = 7,823, active control n = 7,840, high risk only n = 7,809, high risk medical records n = 7,797, and high risk algorithm n = 7,860.Extended Data Fig. 5 Vaccination rate differences in Study 3 by actual and communicated risk levels.Study 3 primary outcome separately by actual risk level, communicated risk level, and study arm. The vertical dashed line at 0% represents the overall passive control vaccination rate across risk level of 23.8%. All patients in the high-risk arms were told they were in the top 20% of risk. Vaccination rates in the passive control group are shown for comparison with active arms, stratified by risk level. Sample sizes in each group from top to bottom are 3,917, 3,930, 3,954, 3,848, 3,862, 3,906, 3,910, 3,855, 3,949, and 3,998. Error bars indicate 95% confidence intervals.Extended Data Fig. 6 Power sensitivity analysis.Power for a range of effect sizes for each study, given a priori assumptions. Studies were powered to detect 2.0 percentage-point minimum detectable effects (MDEs), consistent with previous nudge literature. In study 1, our preregistered power analysis revealed 92% power to detect this effect size, but 80% power is also shown for consistency with studies 2 and 3, where we had 80% power to detect a 2 percentage-point effect.Supplementary informationSupplementary Information (download PDF )Supplementary Figs. 1–18, Tables 1–41, CONSORT diagrams, baseline data, additional information about adverse events, preregistered analyses referenced in the main text, write-up and discussion of study 4, write-up and discussion of pretest and post-test surveys, and full nudge message text.Reporting Summary (download PDF )Rights and permissionsSpringer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.Reprints and permissionsAbout this article