A review of self-supervised imaging phenotypes for genome-wide association studies

Read post on nature.com

IntroductionThe availability of large-scale biobanks with imaging and genomic data has enabled studies of how genetic variation influences organ morphology and function through genome-wide association studies (GWASs). Imaging GWASs have shown that genetic variants contribute to structural and functional variation in organs such as the brain1, heart2, and eye3, thereby improving understanding of disease pathways4,5.Imaging genetics has evolved substantially over the past two decades. Early studies often focused on candidate genes, linking variants such as APOE ε4, 5-HTTLPR, and BDNF Val66Met to brain MRI phenotypes, including measures of brain structure and task-related activation6,7,8. However, many of these associations were based on small samples and showed inflated effect sizes; subsequent meta-analytic and replication studies found that some candidate-gene effects were smaller, heterogeneous, or difficult to replicate consistently9,10. This replication crisis highlighted the highly polygenic architecture of imaging traits and drove a shift toward genome-wide analyses in larger, better-powered datasets. Large collaborative efforts such as the ENIGMA and CHARGE consortia helped establish the first robustly replicated loci for imaging phenotypes by harmonizing imaging and genomic data across multiple cohorts11,12,13. More recently, population-scale biobanks with integrated imaging and genetic data have expanded imaging-genetic discovery across multiple organ systems, including cardiac, retinal, abdominal, and whole-body imaging phenotypes.Imaging-genetic resources can be broadly categorized into population-level biobanks, cohort studies, and consortia. Population-level biobanks recruit broadly from general populations, providing the sample size needed for GWAS. The UK Biobank (UKBB) is one of the largest and most prominent resources, with ~500,000 participants with genomic data and ~100,000 who have undergone multimodal imaging across brain, cardiac, abdominal, and retinal modalities14,15,16,17. Similarly, the German National Cohort (NAKO) has enrolled ~200,000 adults, including a whole-body magnetic resonance imaging (MRI) sub-cohort of ~30,000 participants18,19. With whole-genome sequencing currently underway, these resources will enable future large-scale imaging-genetics studies. The Qatar Biobank targets up to 60,000 nationals and long-term residents, combining genetic data with whole-body and brain MRI to provide a valuable resource of a Middle Eastern population20. In contrast, specialized cohorts and consortia target specific diseases. Representative examples include the Alzheimer’s Disease Neuroimaging Initiative (ADNI), which follows ~2400 participants with genomic data and longitudinal brain imaging21,22; the Adolescent Brain Cognitive Development (ABCD) Study, a pediatric neuroimaging and genomic resource with ~11,880 children23,24; the Multi-Ethnic Study of Atherosclerosis (MESA)25, which provides cardiac and vascular imaging across diverse ethnic groups along with genomic data; the Rotterdam Study with brain and retinal imaging along with genomic data26; and the Framingham Heart Study with longitudinal cardiovascular and genomic data27,28.A key component of imaging GWAS is the derivation of heritable, interpretable, and reproducible phenotypes from medical images. Traditional approaches rely on expert-defined image-derived phenotypes (IDPs), typically based on anatomical segmentations and hand-crafted measurements obtained either manually or through automated software tools1,29,30,31. While powerful, these strategies may introduce bias from human annotation and assumptions embedded in segmentation software32,33,34, require labor-intensive labeling and may be constrained by current clinical knowledge of anatomy-function relationships. As a result, they may capture only a fraction of the information encoded in high-dimensional imaging data35. These limitations highlight the need for new approaches that can scale to biobank-level datasets while uncovering novel, biologically meaningful features beyond the scope of traditional IDPs. The biological challenge is therefore to convert complex medical images into scalable, reproducible, and heritable quantitative traits that preserve meaningful inter-individual variation and can be linked to genetic variation.Self-supervised learning (SSL) models have emerged as an alternative approach for deriving phenotypes from medical images36,37,38,39,40,41. The goal of SSL is to learn a latent representation capturing the intrinsic structure of the input data modality. Instead of relying on human-labeled data in typical supervised learning, SSL supplies the prediction targets for each training example and trains itself using typical supervised losses such as input reconstruction, future prediction, or cross-encoder branch alignment42,43. In imaging genetics, SSL serves as a representation-learning framework for deriving quantitative imaging phenotypes from high-dimensional imaging data. These learned representations can then be used as imaging traits in downstream GWAS to identify genetic variants associated with imaging-derived phenotypic variation. For representation learning for medical images, SSL models enable label-free feature extraction directly from minimally processed images, without reliance on expert-defined labels42. Here, expert-defined refers to imaging phenotypes specified a priori by domain experts, such as radiologists, neurologists, cardiologists, ophthalmologists, or imaging scientists, based on established anatomical structures or clinical measurements. Examples include hippocampal volume, ventricular volume, left ventricular ejection fraction, retinal layer thickness, and vessel caliber. By learning representations via reconstruction or contrastive objectives, SSL can capture subtle, distributed anatomical patterns that may not be fully summarized by conventional phenotypes derived from predefined anatomical structures or segmentation-based measurements. Expert-defined IDPs and SSL-derived phenotypes have complementary strengths: IDPs provide direct clinical interpretability, reproducibility across processing pipelines, and established biological meaning, while SSL approaches capture broader and more distributed variation beyond predefined measurements and scale more readily to new imaging modalities, although interpretability and biological grounding remain challenges (Table 1). However, both approaches depend on image acquisition, preprocessing, quality control, technical choices, and cohort characteristics.Table 1 Expert-Defined IDPs vs. SSL-Derived PhenotypesFull size tableAlthough SSL is generally considered a form of unsupervised representation learning, it uses explicit predictive objectives whose targets are generated from the input data or naturally paired views, rather than from externally provided phenotype or trait labels. This distinguishes SSL from conventional unsupervised approaches such as clustering, dimensionality reduction, and density estimation. Although autoencoders were invented before the term SSL got popular, they are often categorized as an SSL method because the idea of deriving a compact representation (bottleneck) and using the decoder as a guide to train the encoder are very much in line with the philosophy of SSL. Our scope is defined by two inclusion criteria: (i) the method applies SSL directly to raw or minimally preprocessed medical images, rather than to pre-extracted expert-defined IDPs, and (ii) the learned representations are used as quantitative phenotypes for genome-wide association testing. This review aims to provide a methodological and biological framework for understanding how SSL-derived imaging phenotypes support genetic discovery. We searched PubMed and Google Scholar through June 2026 and screened the reference lists of included studies. Representative terms were combined: (“self-supervised learning” OR “unsupervised representation learning” OR autoencoder OR “contrastive learning”) AND (“medical imaging” OR MRI OR cardiac imaging OR retinal imaging OR fundus OR OCT) AND (“genome-wide association study” OR GWAS OR “genetic association”). We included studies that applied SSL or unsupervised representation learning directly to medical images and used the learned features as phenotypes for genome-wide association testing. We excluded studies that were based only on expert-defined IDPs, used supervised models without an SSL component, or did not include GWAS.The review is organized as follows: we first summarize methodological foundations such as data preprocessing, SSL objectives and architectures, and strategies for converting embeddings into phenotypes and assessing their heritability and suitability for GWAS. We then review organ‑specific applications in brain MRI, cardiac MRI, and retinal imaging (fundus photography and optical coherence tomography (OCT)). Finally, we provide a forward-looking roadmap and discuss future directions. While challenges remain in reproducibility, cross-cohort generalization, interpretability, and biological validation, SSL-derived phenotypes represent a promising complement to expert-defined IDPs and may scale to biobank-level analyses; however, their broader adoption will require standardized benchmarking, external replication, and validation across diverse biobank and clinical cohorts.Overall framework of self-supervised learning approaches for genetic discoverySSL-based imaging genetics frameworkTo enable genetic discovery at biobank scale, we describe the SSL imaging genetics framework in three interconnected phases (Fig. 1): phenotype derivation, genome-wide association, and model interpretation.Fig. 1: Overview of the self-supervised learning (SSL) imaging-genetics framework.Full size imageA SSL-Based Phenotype Derivation: Medical images undergo standardized medical-image preprocessing steps. Input images are processed by a trained SSL encoder to extract high-dimensional latent space embeddings, which are harvested as quantitative phenotypes. B Genome-Wide Association and Discovery: Derived phenotypes are integrated with genotype data to perform GWAS. The phase yields summary statistics for Manhattan plots, SNP-based heritability estimates, and post-GWAS biological pathway analyses. C Model Interpretation: Strategies for mapping abstract latent phenotypes back to interpretable anatomical structures using either encoder-based or decoder-based approaches. UK Biobank images in this figure are reproduced with kind permission of UK Biobank ©. Brain MRI template image is derived from the MNI152/ICBM152 T1-weighted average structural MRI template.Phase A: SSL-based phenotype derivationThe foundation of this framework, illustrated in Fig. 1A, is the derivation of robust imaging traits. Four datasets support an SSL imaging-genetics project: training and validation sets for model development and discovery and replication sets (with paired imaging and genetic data) for GWAS. Training sets in reviewed studies range from ~250040 to ~32,00036, with most studies using several thousand images. A single biobank can be split into four non-overlapping subsets to control label leakage, or separate sources can be used for model development and GWAS37.Before training SSL models, standardized preprocessing is necessary. For example, brain MRI typically requires skull stripping, registration to standard templates, bias-field correction, and intensity normalization17. Dynamic modalities such as cardiac MRI require additional temporal alignment across cardiac cycles44. The registration to a common template is especially important as it allows the model to ignore common features of all images and focus on individual variability instead.For SSL imaging-genetics, a representation of the input image is harvested as the phenotype of interest. These phenotypes could be called endophenotypes: they are often heritable and thus are informative of the underlying biology behind the images. Autoencoders, neural networks designed to learn efficient data coding by training an encoder to compress the input and a decoder to reconstruct it, are frequently used37. Usually, the bottleneck layer between the encoder and the decoder is selected as the embedding of the input. This bottleneck represents the “latent space,” a compressed, multi-dimensional abstract representation of the original image. If the dimensionality (the number of variables in the latent space) is too large, an efficient set of phenotypes can be derived through dimensionality reduction methods such as principal component analysis45,46. Other approaches, such as R-indices, learn a small set of indices, with each index capturing a distinct pattern of anatomical variation rather than maximizing statistical variance, as PCA does47.The resulting components of the latent space serve as the quantitative phenotypes (shown as P1…PN in Fig. 1A) used for downstream analysis. These SSL‑derived phenotypes are often characterized via descriptive analyses to verify their meaningfulness. This validation typically includes visualization using approaches such as UMAP48 correlations with demographics (e.g., age, sex, BMI) and traditional IDPs using correlations and predictions and multivariate CCA analyses.Phase B: Genome-wide association studies of SSL phenotypesIn the second phase, the derived latent representations are utilized as quantitative imaging phenotypes for genetic discovery. While GWAS methodology is well-established, SSL-derived phenotypes introduce specific analytical considerations. Unlike traditional IDPs, which are typically low-dimensional and clinically defined, SSL latent spaces can comprise tens to hundreds of correlated dimensions, creating challenges for multiple-testing correction and statistical power. The SSL-derived phenotypes are analyzed with standard linear or linear mixed‑model GWAS tools (e.g., BOLT‑LMM, fastGWA)49,50 (Fig. 1B). Given the high dimensionality and correlation among SSL features, multivariate tests (e.g., Multivariate Analysis of Variance (MANOVA)51) or joint/multi‑phenotype frameworks (e.g., Multivariate Omnibus Statistical Test (MOSTest)52 or Joint Analysis of multi-phenotype GWAS (JAGWAS)53) can aggregate modest effects across many traits and improve power. Post‑GWAS analyses are critical for interpreting and contextualizing association findings. Standard visualizations like Manhattan plots provide a genome-wide overview of significant loci. SNP‑based heritability estimates37 quantify the aggregate contribution of common genetic variants to phenotypic variance, indicating the extent to which the SSL-derived traits are genetically determined. Further translational steps, such as Mendelian randomization, help move beyond statistical association to infer potential causal relationships between imaging features and disease54.Phase C: Model interpretationFinally, interpreting SSL-derived features is essential because, unlike expert-defined IDPs that have built-in anatomical meaning (e.g., hippocampal volume, left ventricular mass), latent dimensions are abstract and their anatomical relevance must be inferred post hoc. For imaging models, linking phenotypes to localized image regions corresponding to specific anatomical landmarks is convenient to pinpoint the biological structures corresponding to a latent dimension (Fig. 1C). Encoder-based approaches, such as saliency maps (e.g., Grad-CAM and Integrated Gradients), help achieve this by highlighting pixel-level regions that influence specific latent dimensions55,56,57,58. Furthermore, decoder-based perturbation approaches provide a direct visualization by decoding latent vectors perturbed along dimensions associated with significant loci; researchers can observe the precise anatomical variations driven by specific genetic variants37,59. Conversely, for multivariate GWAS frameworks (e.g., MOSTest) that aggregate signals across multiple phenotypes reflecting distributed anatomical variations, mapping individual dimensions is less relevant52. For instance, MOSTest applied to regional brain morphology has revealed variants driving distributed gradients rather than localized effects. Instead, regression networks trained on embeddings can predict known traits (e.g., sulcal depth); averaging anatomical data from individuals at the prediction extremes visualizes the consensus structural variation driving the multivariate association60. Alternatively, association-based difference maps can localize morphological features by linking latent dimensions to specific conditions (e.g., disease status) and visualizing the pixel-wise differences between the average images of cases and controls40.Unlike most traditional medical imaging AI applications that are typically dealing with modest sample sizes, imaging GWAS requires large sample sizes to achieve sufficient statistical power. As a result, efficient computational strategies are required for scalable data processing, model development, and evaluation. Beyond traditional pipelines, deep learning frameworks offering GPU-accelerated model training and efficient data handling, such as NVIDIA Data Loading Library (DALI), are becoming available61,62,63.Evaluating the added value of SSL-derived phenotypes beyond conventional IDPsSSL-derived phenotypes are most likely to provide information beyond conventional IDPs when relevant imaging variation is distributed across an organ or involves complex shape, texture, spatial relationships, or temporal dynamics that cannot be summarized adequately by a small set of predefined measurements. However, added value should not be inferred solely from the number of genome-wide significant loci, because locus counts may also reflect differences in phenotype dimensionality, sample size, statistical power, or definitions of novelty. Ideally, SSL-derived phenotypes and conventional IDPs should be compared in the same participants under similar analytical conditions, with complementarity shown by reproducible genetic associations that persist after accounting for established IDPs. Biological or clinical interpretation requires additional evidence such as feature stability, independent genetic replication, functional support, or associations with meaningful clinical traits and outcomes.The following section describes the principal SSL approaches through which these complementary imaging features may be derived.Self-supervised learning approachesFor model development, we categorize SSL strategies in imaging genetics into two primary families: reconstruction-based and joint-embedding approaches (Fig. 2; Table 2). Throughout this Review, SSL encompasses reconstruction-based and joint-embedding methods in which the supervisory signal is derived from the input data itself or from naturally paired views or modalities; this includes methods historically described as unsupervised, such as autoencoders, and hybrid studies when they contain an SSL component. Despite their architectural differences, both share a common goal: learning informative, low-dimensional representations while avoiding representational collapse, the failure mode in which the encoder maps all inputs to a trivial, constant output. Each family addresses this challenge through a different mechanism.Fig. 2: Overview of self-supervised learning families used for imaging-genetics phenotyping.Full size imageA Autoencoder (reconstruction-based). An encoder compresses the input image x into a low-dimensional latent z, and a decoder reconstructs it as x̂; training minimizes reconstruction error Lrec = ‖x − x̂‖2. B Contrastive learning (e.g., SimCLR). Two augmented views of the same image (positive pair) are mapped through a shared encoder and pulled together in latent space, while views from different images (negative pair) are pushed apart, with loss Lcon = −sim(zi, zi+) + sim(zi, zi−). C Redundancy reduction (e.g., Barlow Twins). Two augmented views are encoded, and the cross-correlation matrix between their embedding dimensions is driven toward identity, decorrelating dimensions while keeping them invariant across views: Lred = Σi≠j Corr(zi, zj)2. D Multimodal alignment (e.g., ContIG). Modality-specific encoders embed paired imaging and genetic data from the same individual into a shared latent space, with alignment enforced by minimizing Lalign = ‖zimg − zgen‖2. UK Biobank images in this figure are reproduced with kind permission of UK Biobank ©. Brain MRI template image is derived from the MNI152/ICBM152 T1-weighted average structural MRI template.Table 2 GWAS-relevant comparison of major self-supervised learning approaches in imaging geneticsFull size tableReconstruction-based methods learn representations by compressing an input through a low-dimensional bottleneck and then reconstructing it (Fig. 2A). To succeed under this constraint, the model must filter out noise and redundancy and retain the most salient structural and semantic information. The resulting bottleneck representation, not the reconstruction itself, is what is used as a GWAS phenotype. A classic autoencoder maps an input x to a latent vector z = fθ(x) and decodes it back to x̂ = gϕ(z), training to minimize reconstruction error: Lrec = ‖ x − x̂ ‖22. In plain terms, the model is penalized whenever the reconstructed image differs from the original (Fig. 2A), so the encoder is forced to pack the most informative anatomical content into the latent bottleneck37,64.Autoencoders trained this way have been shown to be effective phenotypers for brain images. Variational autoencoders (VAEs) extend this objective by adding a Kullback-Leibler term that regularizes the latent distribution toward a Gaussian prior, encouraging smoother latent geometry39,65. Masked autoencoders (MAEs) instead reconstruct only randomly masked patches of the input, forcing the network to learn rich semantic context and anatomical shape priors to predict the missing tissue66.Joint-embedding methods take a different approach: rather than reconstructing the input, they learn a logical relationship among embeddings produced by parallel encoders applied to different views or modalities of the same underlying subject (Fig. 2B–D). Given encoders fθ and fθ′, paired inputs (xi, xj) are mapped to embeddings zi = fθ(xi) and zj = fθ′(xj). The choice of loss determines what relationship the embeddings should encode and divides this family into three branches: contrastive (Fig. 2B), redundancy-reduction (Fig. 2C), and multimodal alignment (Fig. 2D).Contrastive learning captures features shared across variations of the same input by maximizing similarity between positive pairs, typically two augmented views of the same image, or naturally paired data such as left-right retina, while pushing apart negative pairs drawn from different individuals36,67,68 (Fig. 2B). At its simplest, the loss attracts an anchor toward its positive partner and repels it from a negative: Lcon = −sim(zi, zi+) + sim(zi, zi−). In plain terms, this loss shrinks when the anchor embedding sits close to its positive partner and far from negative examples, thereby pulling together views of the same individual while pushing apart views from different individuals (Fig. 2B).SimCLR formalizes this idea using the InfoNCE objective, which contrasts each anchor against all negatives in the batch via a softmax over similarities; this approach requires very large batches to supply enough negative pairs for stable training68. MoCo relaxes this requirement by maintaining a memory bank of past representations as negatives, decoupling negative-sample count from batch size69.Redundancy-reduction methods avoid negative samples entirely. Barlow Twins pass two augmented views of the same image through a shared encoder and decorrelate their embeddings so that each latent dimension captures unique, non-redundant information (Fig. 2C)60,70. The objective minimizes the squared off-diagonal correlations between embedding dimensions of the two views: Lred = Σi≠j Corr(zi, zj)22. In plain terms, the loss is small when different latent dimensions carry independent, non-redundant information, driving the cross-correlation matrix between the two views toward an identity-like form (Fig. 2C).This simultaneously enforces invariance across views and decorrelation across latent dimensions. Related approaches share this redundancy-reduction philosophy: VICReg combines variance, invariance, and covariance terms to prevent collapse71. BYOL avoids collapse using an asymmetric architecture with a momentum-updated target encoder and a predictor head rather than an explicit decorrelation term72.Multimodal alignment extends joint embedding across distinct input streams (Fig. 2D). CLIP (Contrastive Language-Image Pre-Training) established the foundational framework by training paired image and text encoders with a contrastive objective73, and the same paradigm has since been adapted to imaging-genetics, for example, aligning retina images with genetics or electrocardiogram (ECG) with cardiac MRI into a shared latent space55,74. Genetic-guided approaches such as ContIG directly align an individual’s image and genetic embeddings55, in which case a separate GWAS data set is required to avoid label leakage. We classify genotype-informed methods such as ContIG and Genetic InfoMax as multimodal SSL because matched imaging and genetic data from the same individual serve as the supervisory signal without requiring manually annotated phenotype labels. When genotype data contribute to representation learning, using the same individuals for model training and subsequent GWAS can create circularity and information leakage because the tested genotypes have already influenced the phenotype definition. Representation-learning and GWAS cohorts should therefore be non-overlapping, and any sample overlap should be reported explicitly. The simplest alignment objective minimizes squared distance between paired embeddings: Lalign = ‖ zimg − zgen‖22. In plain terms, the loss is small when imaging and genetic embeddings of the same individual sit close together in the shared latent space, so the two modality-specific encoders are jointly trained to agree on a common representation per person (Fig. 2D). In practice, contrastive or auxiliary losses are often needed to preserve individual-level variation and avoid representational collapse.Reconstruction-based and joint-embedding approaches are not mutually exclusive, and several studies in this review combine them. Genetic InfoMax pairs image reconstruction with contrastive alignment between MRI and genetic embeddings45, while the cross-modal autoencoder combines modality-specific reconstruction losses with a contrastive loss across ECG and cMRI74. These objectives differ in trade-offs relevant to GWAS (Table 2), and resolving which combinations work best will require head-to-head benchmarks on shared datasets with standardized pipelines. To highlight the main conceptual and GWAS-relevant differences across these SSL families, we provide a cross-method comparison in Table 2.The choice of SSL approach should be guided by the biological question, imaging modality, available sample size, data structure, and desired level of phenotype interpretability. Fig. 3 provides a practical decision map linking common study settings to suitable SSL strategies and summarizes their principal advantages, limitations, and GWAS-specific considerations.Fig. 3: Decision map for selecting self-supervised learning approaches for imaging-phenotype GWAS.Full size imageEach question is answered in turn; a “Yes” leads to the recommended approach (gray box) with a representative implementation and its principal limitation, and a “No” proceeds to the next question. When no question applies, the default is reconstruction-based autoencoding. Superscript numbers refer to the reference list. AE autoencoder, ECG electrocardiogram, GWAS genome-wide association study, LV left ventricle, MRI magnetic resonance imaging, OCT optical coherence tomography, SNP-h² SNP-based heritability, SSL self-supervised learning, UDIPs Unsupervised Deep-Learning-Derived Imaging Phenotypes, VAE variational autoencoder, VICReg Variance-Invariance-Covariance Regularization.This decision map is intended as practical guidance rather than a strict classification, as the optimal approach will also depend on cohort characteristics, computational resources, and validation data.We next review organ‑specific applications of SSL in imaging genetics. For each organ, we summarize datasets and preprocessing, the SSL objectives and architectures used, how embeddings were converted to phenotypes and validated (including SNP‑heritability), key genetic findings and replication, interpretability analyses, and limitations.Brain imaging geneticsTraditional brain imaging-genetics studies have relied on expert-defined IDPs, ranging from volumetric measures like hippocampal volume and cortical thickness using software packages like FreeSurfer and FSL1,29 to high-dimensional voxel-wise maps derived from Tensor-Based Morphometry (TBM) and Voxel-Based Morphometry (VBM)75,76. While successful, these approaches face inherent trade-offs: segmentation algorithms are sensitive to image quality and anatomical priors limited by existing anatomical knowledge, while voxel-wise methods suffer from severe multiple-testing burdens that reduce statistical power. SSL approaches address these limitations by learning compact, data-driven phenotypes directly from the images (Table 3).Table 3 Summary of self-supervised learning approaches across brain, cardiac, and retinal imaging geneticsFull size tableEarly applications focused on reconstruction-based methods. Unsupervised Deep-Learning Derived Imaging Phenotypes (UDIPs) employed three-dimensional convolutional autoencoders to encode T1-weighted or T2-FLAIR scans into 128-dimensional vectors optimized purely for image reconstruction37. Trained on 6130 mixed-ethnicity UKBB participants, the model derived latent phenotypes for GWAS in a larger discovery (n ~ 22,000) and replication (n ~ 12,000) cohort. The analysis identified 97 independent loci, 60 of which replicated, including 26 novel loci not captured by traditional IDP GWAS1. A subsequent meta-analysis of the discovery and replication cohorts expanded the findings to 199 significant loci; notably, 145 of these were not found in traditional IDP GWAS, including 29 novel loci previously unrelated to any brain-related traits. The average SNP-heritability of UDIPs (h² ≈ 0.25) exceeded that of expert-defined traits (h² ≈ 0.18). By decoding perturbed latent vectors back into image space, the authors visualized the specific anatomical structures (e.g., ventricles, thalamus) driving genetic associations. However, perturbation of a single latent dimension can propagate globally in the decoded image, making anatomical attribution less precise than voxel-wise IDPs.A related extension applied the UDIP framework to diffusion MRI fractional anisotropy (FA) maps to capture white matter microstructure77. Using a 3D convolutional autoencoder trained on voxel-wise FA maps, the study derived 128-dimensional UDIP-FA embeddings and used them as quantitative phenotypes. Multivariate GWAS identified 420 lead SNPs across 300 loci in the discovery cohort, of which 182 lead SNPs replicated; meta-analysis of both cohorts identified 939 lead SNPs across 586 loci. Using a perturbation-based decoder approach similar to the original UDIP study, the authors perturbed individual UDIP-FA dimensions and decoded them back into image space to generate voxel-wise t-maps, which were then analyzed for enrichment across ICBM DTI-81 white-matter tracts. These findings suggest that SSL-derived phenotypes can extend beyond structural T1/T2 MRI to microstructural diffusion MRI.In contrast, Genetic InfoMax utilized a multimodal contrastive learning framework to align image and genotype data45, in addition to an image reconstruction task. Using a 3D CNN encoder for MRI and a novel transformer-based genetic encoder for SNPs, GIM was trained to maximize mutual information (InfoMax) between the MRI and genetic embeddings. Genetic InfoMax reported a heritability of h² ≈ 0.38 and improved genetic discovery with 40 loci (15 unique to this method) for the top 10 principal components (PCs) of the endophenotypes. However, unlike UDIPs, GIM did not include a dedicated interpretability pipeline or replication cohort, and the choice of using top PCs as a phenotype was meant for methodological development rather than maximizing biological discovery. Therefore, further rigorous studies at scale are warranted.Another example is TransferGWAS46 which derived phenotypes using two distinct strategies: a 3D CNN VAE trained on ADNI scans with a self-supervised image-reconstruction objective combined with supervised dementia prediction and a 2D ResNet-50 pretrained on ImageNet natural images via supervised transfer learning78. The study identified a union of 289 loci (165 from ADNI-derived phenotypes and 194 from ImageNet-derived phenotypes, with a shared overlap of 70 loci). The ADNI-derived phenotypes captured brain-specific variation (mean h² = 0.19), and ImageNet traits reflected systemic features (mean h² ≈ 0.13). This shows SSL phenotypes may capture systemic whole-body traits (e.g., bone density, BMI) rather than brain-specific mechanisms, necessitating rigorous downstream validation. The higher number of loci reported by Rakowski et al.46 is primarily driven by their significantly larger discovery sample size (N = 36,311 vs. UDIPs N = 22,800) and reduced multiple-testing burden (20 phenotypes vs. UDIP 256 phenotypes with T1/T2), combined with their capture of non-brain signals.Dufournet et al. trained a Barlow‑Twins style CNN-based model (“Champollion V0”)79 to encode local sulcal morphology within the anterior cingulate cortex (ACC) from UKBB T1 MRI60. Using BrainVISA/Morphologist80 to extract sulcal skeletons and an ACC region of interest, the model learned a 256‑dimensional latent space on 36,000 White‑British participants. This cohort served as both the training and discovery set, while a distinct non‑White‑British cohort (n = 5395) was used for replication. Principal components explaining 99.9% of variance yielded 42 PCs (left ACC) and 39 PCs (right ACC) for association testing with the MOSTest multivariate GWAS framework. In discovery, the authors reported 10 loci (left ACC) and 4 loci (right ACC) linked to specific ACC folding patterns; one locus replicated nominally in the smaller replication cohort. For interpretation, they used regression-guided visualizations by training linear models to predict specific sulcal traits (e.g., presence of the paracingulate sulcus) from the full latent space. By identifying the subjects falling at the extremes of these prediction probabilities, they computed the average of the 3D sulcal skeletons for the high-probability group versus the low-probability group and then visualized the resulting consensus anatomical configurations (e.g., double-parallel patterns). This approach provides a direct visual link between the abstract latent features and the biological structures associated with the discovered genetic loci. Unlike UDIPs, which map individual latent dimensions to brain regions, this method interprets the multivariate direction of effect consistent with the MOSTest discovery framework.Collectively, these approaches highlight key trade-offs in SSL-based imaging genetics: reconstruction-based models (e.g., UDIPs) offer broad anatomical coverage and a clear interpretation framework, whereas transmodal contrastive/InfoMax methods (e.g., GIM) can yield higher heritability. Transfer learning strategies (e.g., TransferGWAS) may reduce the amount of data required for model training, leaving more samples available for GWAS to increase discovery power, but domain shift across data sets could potentially reduce the strength of the phenotyping mode. Here, domain shift refers to differences in scanner hardware, acquisition protocols, preprocessing pipelines, and cohort characteristics across datasets. Region-focused models (e.g., Champollion) suggest that local cortical folding phenotypes may have distinct, but sometimes subtler, genetic architectures.Overall, SSL brain imaging genetics studies are still few and far between. Existing methods all demonstrated the power of identifying new loci beyond traditional IDP-based GWAS. However, these methods rely on different preprocessing pipelines (e.g., affine vs. non-linear registration), different GWAS sample sizes, different numbers of phenotypes, and different loci clumping protocols (Table 3), making direct cross-study comparison of GWAS results difficult.Cardiac imaging geneticsTraditional cardiac imaging-genetics studies rely on traditional IDPs such as left ventricular (LV) mass, end-diastolic volume, left ventricular ejection fraction, and wall thickness measurements extracted using automated analysis pipelines81,82. While clinically validated, these IDPs may fail to capture the full complexity of cardiac morphology, including regional shape variations and subtle geometric deformations. Furthermore, although cardiac MRI (cMRI) provides rich 4D (3D spatial + 1D temporal) data, standard analyses typically reduce this to static measurements at end-diastole and end-systole. SSL approaches could potentially capture these comprehensive phenotypes, though current methods remain limited by their focus on static snapshots and training cohorts of largely European ancestry (Table 3).An early application was a cross-modal autoencoder framework integrated cMRI with ECG data from 38,686 UKBB participants74. The approach used separate 1D and 2D CNN autoencoders for ECG and cMRI, respectively, with a shared 256-dimensional latent space. Training combined reconstruction losses for each modality with a contrastive loss that encouraged paired ECG-cMRI samples to have similar representations while separating unpaired samples. While cMRI data included 50 frames across the cardiac cycle, the 2D CNN architecture treats these temporal frames as separate channels rather than processing sequential temporal dynamics, limiting exploitation of true spatiotemporal cardiac motion patterns. Using MANOVA, the joint model identifies more lead SNPs (93) than the MRI-only model (33) and the ECG-only model (86), indicating it captures signals missed when using only one modality’s representation. For genetic discovery, a MANOVA analysis of the multivariate latent space identified variants affecting cardiovascular physiology that were missed by single-modality analyses. Regarding interpretation, the study did not map genetic associations to localized anatomical regions. However, the authors demonstrated that the learned representations captured relevant physiological information by predicting demographics (age, sex, BMI) and traditional phenotypes (e.g., Left Ventricular Mass [LVM] and Right Ventricular End-Diastolic Volume [RVEDV]) from single modalities74. Also, they showed it is possible to impute synthetic MRI video frames from ECG inputs, where metrics derived from the synthetic images correlated with ground truth values (e.g., LVM and RVEDV). The cross-modal integration represents a methodological advance, though the genetic discoveries largely involved known cardiovascular genes such as NOS1AP, TTN, SCN10A, and SLC35F174.A geometric deep learning approach applied graph convolutional autoencoders to LV shape analysis using 3D surface meshes from 48,651 UKBB participants (discovery) with replication in 5470 participants59. Preprocessing involved automatic segmentation to generate registered 3D LV surface meshes (~5k vertices) at end diastole. The method focused on the end-diastole phase only and used spectral Chebyshev convolutions83, a kind of graph convolutional network (GCN), to learn shape representations while preserving mesh topology. An ensemble approach pooled representations from 36 models trained with different hyperparameters, yielding 324 latent features after correlation-based filtering. GWAS identified 49 genome-wide significant loci, with 40 claimed as novel. Interpretability was achieved through visual shape effect analysis: for each significant genetic locus, they identified the most strongly associated latent variable and visualized heart meshes across quantile ranges [0–1%, 9.5–10.5%, 49.5–50.5%, 89.5–90.5%, 99–100%] to demonstrate how genetic variants affect specific 3D morphological patterns. For example, the STRN locus was associated with a change in mitral valve orientation without altering LV volume.These studies demonstrate that SSL can uncover novel loci in cardiac imaging genetics, but they represent only sparse attempts at exploration. More comprehensive studies are needed with external replication, diverse cohorts, and systematic comparisons against standard cardiac IDPs to fully unlock the potential of SSL heart imaging genetics. Also, challenges in generalizability remain. Both pipelines were trained on 1.5 T Siemens scans from the UKBB84, leaving performance on other vendors (GE, Philips) or field strengths unverified.Retinal imaging geneticsTraditional retinal imaging-genetics studies rely on expert-defined IDPs such as vessel caliber, optic disc morphology, and macular thickness extracted through specialized software3. While successful in identifying genetic associations, these approaches have limitations similar to other imaging modalities, including reliance on segmentation quality and a priori definitions of pathological features. SSL approaches in retinal imaging have focused on leveraging both 2D fundus photography and 3D optical coherence tomography (OCT) to capture broader morphological variation (Table 3). Fundus photography captures a 2D projection of the retinal surface, while OCT provides high-resolution 3D cross-sectional maps, enabling detailed assessment of retinal layers.iGWAS (Image-based Genome-Wide Association Studies) employed a metric learning approach on retinal fundus photographs using paired left and right eye fundus images36. A 2D CNN (Inception V3) was trained on 40,000 EyePACS images (32,000 train/8000 val) using an ArcFace contrastive loss85, which minimizes the angular distance between eyes of the same individual while maintaining margins between different individuals. The trained phenotyper was then directly applied to UKBB images without fine-tuning to generate 128-dimensional embeddings for 130,329 images. GWAS on these phenotypes identified 14 independent loci associated with the embeddings directly, including 3 novel loci (DCT, PDE6G, EPHB4) linked to retinal structure and function that were missed by traditional color-trait GWAS.ContIG (Contrastive Imaging and Genetics) applied contrastive learning between retina fundus images and genetic data55. Using a ResNet-50 for images and multi-layered perceptrons (MLPs) for genetics, the model was trained to distinguish matched image-genotype pairs from random ones using a batch-wise contrastive loss. The framework integrated multiple genetic modalities, including raw SNPs, polygenic scores (PGS), and burden scores. In an analysis of 155,238 UKBB left and right eye fundus images, ContIG identified phenotypic dimensions that yielded 16–22 independent loci (depending on the genetic modality used), outperforming baseline self-supervised methods (e.g., SimCLR, BYOL) which found only 2–17 loci. Furthermore, the learned features demonstrated superior performance on downstream clinical tasks, such as diabetic retinopathy detection and cardiovascular risk prediction.Similar to the case for brain images, TransferGWAS applied transfer learning to fundus images using two distinct pretraining strategies on a ResNet-50 architecture: one pretrained on ImageNet (natural objects) and another on EyePACS (retinal pathology). Feature extraction followed by PCA yielded 20 total phenotypes (10 per model)86. The study identified 60 independent loci, including 7 novel associations. Interestingly, the phenotypes derived from EyePACS pretraining captured retina-specific pathology, whereas ImageNet-derived phenotypes captured broader morphological features like pigmentation and achieved greater overall discovery power. The interpretation approach involved training separate ResNet18 networks to predict specific genotypes from fundus images, followed by visual inspection of high/low-activating images by experienced ophthalmologists.Moving to OCT images, Sergouniotis et al. applied an autoencoder-based approach to 2D OCT macular thickness maps40. Optical coherence tomography (OCT) provides high-resolution 3D cross-sectional maps of retinal structure, enabling detailed assessment of retinal layer thickness and morphology. The study trained a 2D CNN autoencoder to compress segmented thickness maps into a 64-dimensional latent space, which was further reduced to 25 PCs. Both the original 64-dimensional latent features and the 25 PCs were employed as quantitative imaging phenotypes for genetic association analysis. GWAS on these traits identified 118 significant loci, with 41 achieving independent replication in a validation cohort. Of these, at least 20 loci were novel and not previously associated with retinal phenotypes. To interpret these latent features, the authors utilized retinal thickness difference maps. First, they linked specific latent dimensions to known medical conditions (e.g., non-insulin-dependent diabetes) via statistical association. Once an association was established, they subtracted the average thickness maps of confirmed cases from controls to visualize specific pixel-level morphological changes. This allowed them to pinpoint which anatomical regions were affected by the condition associated with the latent dimension, effectively mapping the genetic findings from that dimension to specific retinal structures. For instance, this approach revealed that diabetes-associated dimensions captured thinning in the paracentral region corresponding to major retinal vessels, confirming the model’s ability to detect subtle neurovascular pathology. The derived phenotypes were shown to improve risk stratification for glaucoma and cardiovascular disease compared to traditional clinical parameters, demonstrating that SSL can extract clinically relevant biomarkers from structural maps.Retinal SSL studies demonstrate a clear trade-off between domain specificity and discovery power. Models pretrained on domain-specific data (e.g., EyePACS in TransferGWAS) or using specialized inputs (OCT thickness maps) tend to yield clearer structural associations. In contrast, models using generic pretraining (ImageNet) often capture systemic or pigmentation-related traits with potentially broader pleiotropic relevance.Challenges and future directionsThe field of self-supervised imaging genetics is transitioning from proof-of-concept discovery to a phase of maturation. While early studies have successfully demonstrated that SSL can recover heritable and interpretable phenotypes that can identify novel loci, the next phase of research must address critical bottlenecks in population diversity, methodological rigor, and model interpretability (Box 1).Box 1 Current challenges and future directions in self-supervised imaging geneticsChallengesPopulation diversity and generalizabilityTo the best of our knowledge, every SSL imaging GWAS study identified for this Review uses UKBB, a cohort dominated by middle-aged European-ancestry participants.Domain shift across scanners, vendors, and protocols outside UKBB data can degrade model performance outside the training cohort.Methodological barriersMost SSL pipelines still depend on conventional preprocessing (e.g., skull stripping, registration), introducing pipeline-specific variability.High-dimensional latent spaces can dilute statistical power; no consensus exists on the optimal balance between dimensionality and multiple-testing burden.Training on 4D modalities (e.g., cardiac cine MRI) is computationally intensive, raising access and carbon-footprint concerns.Interpretability and biological meaningSSL-GWAS interpretation is challenging and largely done post hoc: because latent dimensions are learned without anatomical supervision, biological meaning must be inferred after the fact rather than built into the phenotype definition.Perturbation-based decoding can smear effects across anatomical boundaries, and saliency maps can saturate.Whether learned features reflect reproducible biology versus dataset-specific artifacts remains open.Experimental follow-up of SSL-GWAS loci is essentially absent from the literature.Future DirectionsDiverse cohorts: Leverage emerging biobanks for cross-ancestry training and validation, including under-represented populations.Benchmarking and accessibility: Develop standardized benchmarks reporting heritability per latent dimension, replication rates, and compute costs; release pretrained models and pipelines.Emerging architectures: Vision Transformers for non-local patterns, video transformers and diffusion models for spatiotemporal dynamics, and CLIP-style multimodal foundation models.Newer modalities: Extend SSL imaging genetics to abdominal MRI, DXA and lung CT.Population diversity and generalizabilityA primary limitation in imaging genetics is the lack of diversity in data, which undermines the equity of these methods. To the best of our knowledge, every SSL imaging GWAS study identified for genome-wide discovery uses UKBB, a resource dominated by middle-aged participants of European ancestry. This bias limits the generalizability of SSL-derived findings to pediatric, non-European, or under-represented populations. Consequently, replication remains a bottleneck. The scarcity of external cohorts with comparable imaging and genetic scale makes it difficult to validate discoveries. To ensure generalizability and avoid exacerbating health disparities, expansion beyond the UKBB is urgently needed. Promisingly, other large-scale resources are emerging that may help address this gap. For instance, the Qatar Biobank20 is combining genetic data with whole-body and brain MRI in a Middle Eastern population, and the German National Cohort (NAKO)18,19 has enrolled a large MRI sub-cohort. Leveraging these diverse biobanks for cross-ancestry training and validation will be essential for the next generation of SSL models.Methodological and interpretability barriersFor SSL-specific challenges, domain shift poses a particularly acute barrier for self-supervised models, which are often sensitive to acquisition artifacts. Models trained on uniformly collected and processed UKBB data may fail when applied to diverse and heterogeneous clinical scans. A related concern is that SSL models may inadvertently encode non-biological variation such as scanner differences, acquisition artifacts, or ancestry-related morphological differences as latent features, potentially confounding downstream GWAS. Current studies mitigate this by including standard covariates (e.g., age, sex, and genetic principal components) in association models, but systematic evaluation of residual bias in learned representations remains limited. Furthermore, despite the self-supervised label, most SSL pipelines still rely on conventional preprocessing such as skull stripping and registration, which introduces its own variability. Computational burden is another significant constraint. Training models on high-dimensional modalities (e.g., 4D cardiac cine MRI) is resource-intensive. This creates a barrier for smaller and less-well-resourced research groups and raises concerns about the carbon footprint of model development. Simultaneously, the high dimensionality of latent spaces can severely dilute statistical power. While dimensionality reduction (e.g., PCA) helps, there is no consensus on the optimal balance between retaining morphological detail and maintaining statistical efficiency.Interpretability and biological validation are not a single step. They involve five separate questions, and each needs its own evidence. First, image localization asks which image regions contribute to a latent feature. Saliency maps, occlusion analyses, and decoder-based perturbations answer this, but knowing where a feature comes from does not tell us what it means37,56. Second, feature stability asks whether the same feature appears again when the model is retrained or when preprocessing, scanners, or cohorts change. This is difficult because latent dimensions are not ordered in a fixed way, so a dimension from one model does not correspond to the same dimension in another87,88. Third, genetic replication asks whether an association holds in an independent sample. It shows the association is real, but not what the phenotype measures. Replication in a held-out split of the same cohort is also weaker than replication in a separate cohort89. Fourth, biological mechanisms require further evidence, such as fine-mapping, colocalization with molecular quantitative-trait loci, pathway analyses, or experimental follow-up90,91. Fifth, clinical relevance requires association with meaningful traits or outcomes, added value beyond established IDPs, and validation in independent clinical cohorts92,93,94. Most current SSL imaging-genetics studies address only some of them. A latent feature is therefore better described by the specific evidence available for it than by a general label such as “biologically validated.”Interpretability remains a persistent difficulty even at the first of these levels. While perturbation-based decoding and saliency mapping provide partial insights, they have concrete limitations: decoder perturbations often smear effects across anatomical boundaries rather than being localized, and saliency maps can saturate, obscuring the magnitude of specific features37,95. Feature stability is rarely tested directly. Ensembling across many independently trained models, as in the Unsupervised Phenotype Ensemble, reduces the effect of single-model instability but does not measure it59. Clinical translation from SSL-discovered loci to therapeutic targets also remains largely theoretical, with experimental follow-up of SSL-GWAS loci essentially absent from the literature.Benchmarking and standardizationTo usher the field into maturation, we need the development of standardized benchmarks and open evaluation frameworks, which would allow consistent comparison of SSL models across datasets and ensure reproducibility. These benchmarks should explicitly track metrics such as heritability per latent dimension, average replication p values for top hits, and compute/energy costs to allow for transparent comparison of SSL architectures. Benchmarks should also test incremental value directly, by computing SSL phenotypes and expert-defined IDPs in the same individuals under matched analytic conditions and reporting loci unique to each and shared between them. Equally important is the release of pretrained models and standardized preprocessing pipelines, which would lower barriers to entry for research groups without extensive computational resources and enable direct comparison of downstream GWAS results across studies. Recent work has emphasized systematic benchmarks; for example, EmbedGEM evaluates whether embeddings are both genetically informative and clinically meaningful96.Emerging architecturesThe next frontier of SSL involves moving beyond static CNNs to architectures that better capture global context and temporal dynamics. Vision Transformers (ViTs)97,98 are emerging as a powerful alternative to traditional convolutional approaches, leveraging attention mechanisms to capture non-local patterns (such as hemispheric symmetry in the brain) that local receptive fields may miss. Foundation model approaches inspired by vision-language pretraining (e.g., CLIP-style contrastive frameworks)73 are also being adapted to imaging-genetics, though current implementations such as COMICAL operate on pre-extracted IDPs and pre-selected variants rather than learning representations from raw images99. Simultaneously, the scope of SSL is expanding beyond structural T1-weighted imaging to include microstructural modalities, such as modeling white matter integrity via diffusion MRI.In cardiac imaging, the focus is shifting toward spatiotemporal modeling. Newer architectures, including diffusion models100 and video transformers101, are being designed to explicitly encode temporal dynamics rather than treating frames as static independent channels. These methods aim to capture complex physiological phenotypes, such as rhythmic motion and myocardial strain. Rigorous validation of these advanced architectures in independent, diverse cohorts remains a priority to confirm their clinical utility over established CNN-based baselines.Beyond architectural advances, extending SSL imaging genetics to currently underexplored modalities such as abdominal MRI102,103, DXA104,105, lung CT106,107, and histopathology108 where supervised methods have already demonstrated substantial heritable signal represents a near-term opportunity to expand the scope of genetic discovery. For example, HistoGWAS recently demonstrated that pretrained histology encoders can be used to derive quantitative tissue embeddings for GWAS, showing histopathology as an emerging direction for image-based GWAS beyond conventional radiological imaging108.ConclusionSelf-supervised deep learning approaches allow scalable derivation of complex, high-dimensional, and heritable imaging phenotypes by enabling the data itself to define rich and reproducible features. Applied across brain, cardiac, retinal, and physiological modalities, these methods have uncovered genetic associations missed by traditional expert-defined traits and in some cases demonstrated greater heritability. Important challenges remain, including limited interpretability, lack of standardized pipelines, high computational demands, and restricted reproducibility and generalizability beyond largely European ancestry cohorts. SSL-derived phenotypes are therefore best regarded at present as promising complementary tools rather than established, clinically actionable biomarkers. With careful validation and broader application, SSL-derived phenotypes have the potential to advance genetic discovery and translate into meaningful improvements in human health.Data availabilityNo datasets were generated or analysed. All studies discussed are published and are cited in the reference list.ReferencesElliott, L. T. et al. Genome-wide association studies of brain imaging phenotypes in UK Biobank. Nature 562, 210–216 (2018).Article  PubMed  PubMed Central  Google Scholar Pirruccello, J. P. et al. Genetic analysis of right heart structure and function in 40,000 people. Nat. Genet. 54, 792–803 (2022).Article  PubMed  PubMed Central  Google Scholar Currant, H. et al. Genetic variation affects morphological retinal phenotypes extracted from UK Biobank optical coherence tomography images. PLoS Genet. 17, e1009497 (2021).Article  PubMed  PubMed Central  Google Scholar Ali, M. et al. Large multi-ethnic genetic analyses of amyloid imaging identify new genes for Alzheimer disease. Acta Neuropathol. Commun. 11, 1–20 (2023).Article  Google Scholar Elsheikh, S. S. M., Chimusa, E. R., Mulder, N. J. & Crimi, A. Genome-wide association study of brain connectivity changes for Alzheimer’s disease. Sci. Rep. 10, 1–16 (2020).Article  Google Scholar Bookheimer, S. Y. et al. Patterns of brain activation in people at risk for Alzheimer’s disease. N. Engl. J. Med. 343, 450–456 (2000).Article  PubMed  PubMed Central  Google Scholar Hariri, A. R. et al. Serotonin transporter genetic variation and the response of the human amygdala. Science 297, 400–403 (2002).Article  PubMed  Google Scholar Egan, M. F. et al. The BDNF val66met polymorphism affects activity-dependent secretion of BDNF and human memory and hippocampal function. Cell 112, 257–269 (2003).Article  PubMed  Google Scholar Murphy, S. E. et al. The effect of the serotonin transporter polymorphism (5-HTTLPR) on amygdala function: a meta-analysis. Mol. Psychiatry 18, 512–520 (2013).Article  PubMed  Google Scholar Hart, A. B., de Wit, H. & Palmer, A. A. Candidate gene studies of a promising intermediate phenotype: failure to replicate. Neuropsychopharmacology 38, 802–816 (2013).Article  PubMed  Google Scholar Stein, J. L. et al. Identification of common variants associated with human hippocampal and intracranial volumes. Nat. Genet. 44, 552–561 (2012).Article  PubMed  PubMed Central  Google Scholar Bis, J. C. et al. Common variants at 12q14 and 12q24 are associated with hippocampal volume. Nat. Genet. 44, 545–551 (2012).Article  PubMed  PubMed Central  Google Scholar Ikram, M. A. et al. Common variants at 6q22 and 17q21 are associated with intracranial volume. Nat. Genet. 44, 539–544 (2012).Article  PubMed  PubMed Central  Google Scholar Sudlow, C. et al. UK Biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLOS Med. 12, e1001779 (2015).Article  PubMed  PubMed Central  Google Scholar Littlejohns, T. J. et al. The UK Biobank imaging enhancement of 100,000 participants: rationale, data collection, management and future directions. Nat. Commun. 11, 2624 (2020).Article  PubMed  PubMed Central  Google Scholar Miller, K. L. et al. Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nat. Neurosci. 19, 1523–1536 (2016).Article  PubMed  PubMed Central  Google Scholar Alfaro-Almagro, F. et al. Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank. Neuroimage 166, 400–424 (2018).Article  PubMed  Google Scholar Peters, A. et al. Framework and baseline examination of the German National Cohort (NAKO). Eur. J. Epidemiol. 37, 1107–1124 (2022).Article  PubMed  PubMed Central  Google Scholar Bamberg, F. et al. Baseline MRI examination in the NAKO health study—findings on feasibility, participation and dropout rates, comfort, and image quality. Dtsch Arztebl Int 121, 587–593 (2024).PubMed  PubMed Central  Google Scholar Al Kuwari, H. et al. The Qatar Biobank: background and methods. BMC Public Health 15, 1–9 (2015).Article  Google Scholar Weiner, M. W. ADNI: two decades of impact and the path forward. Alzheimers. Dement. 20, e083392 (2024).Aisen, P. S. et al. The Alzheimer’s disease neuroimaging initiative clinical core. Alzheimers Dement. 20, 7361–7368 (2024).Article  PubMed  PubMed Central  Google Scholar Wilson, S., Fan, C. C. & Hewitt, J. ABCD behavior genetics: twin, family, and genomic studies using the adolescent brain cognitive development (ABCD) study dataset. Behav. Genet. 53, 155–158 (2023).Article  PubMed  PubMed Central  Google Scholar Karcher, N. R. & Barch, D. M. The ABCD study: understanding the development of risk for mental and physical health outcomes. Neuropsychopharmacology 46, 131–142 (2020).Article  PubMed  PubMed Central  Google Scholar Beussink-Nelson, L. et al. Multi-ethnic study of atherosclerosis early heart failure study: rationale, design, and baseline characteristics. Circ. Heart Fail. https://doi.org/10.1161/CIRCHEARTFAILURE.122.010289 (2024).Article  PubMed  PubMed Central  Google Scholar Ikram, M. A. et al. Objectives, design and main findings until 2020 from the Rotterdam Study. Eur. J. Epidemiol. 35, 483–517 (2020).Article  PubMed  PubMed Central  Google Scholar Andersson, C., Johnson, A. D., Benjamin, E. J., Levy, D. & Vasan, R. S. 70-year legacy of the Framingham Heart Study. Nat. Rev. Cardiol. 16, 687–698 (2019).Article  PubMed  Google Scholar Tsao, C. W. & Vasan, R. S. Cohort Profile: The Framingham Heart Study (FHS): overview of milestones in cardiovascular epidemiology. Int. J. Epidemiol. 44, 1800–1813 (2015).Article  PubMed  PubMed Central  Google Scholar Smith, S. M. et al. An expanded set of genome-wide association studies of brain imaging phenotypes in UK Biobank. Nat. Neurosci. 24, 737–745 (2021).Article  PubMed  PubMed Central  Google Scholar Zekavat, S. M. et al. Phenome- and genome-wide analyses of retinal optical coherence tomography images identify links between ocular and systemic health. Sci. Transl. Med. 16, eadg4517 (2024).Article  PubMed  Google Scholar Tomasoni, M. et al. Genome-wide association studies of retinal vessel tortuosity identify numerous novel loci revealing genes and pathways associated with ocular and cardiometabolic diseases. Ophthalmol. Sci. 3, 100288 (2023).Article  PubMed  PubMed Central  Google Scholar Schoemaker, D. et al. Hippocampus and Amygdala Volumes from Magnetic Resonance Images in Children: Assessing Accuracy of FreeSurfer and FSL Against Manual Segmentation. Neuroimage 129, 1–14 (2016).Han, X. et al. Patient-specific registration of pre-operative and post-recurrence brain tumor MRI scans. Brainlesion 11383, 105–114 (2019).PubMed  PubMed Central  Google Scholar Maier-Hein, L. et al. Metrics reloaded: recommendations for image analysis validation. Nat. Methods 21, 195–212 (2024).Article  PubMed  PubMed Central  Google Scholar Poplin, R. et al. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2, 158–164 (2018).Article  PubMed  Google Scholar Xie, Z. et al. iGWAS: image-based genome-wide association of self-supervised deep phenotyping of retina fundus images. PLoS Genet. 20, e1011273 (2024).Article  PubMed  PubMed Central  Google Scholar Patel, K. et al. Unsupervised deep representation learning enables phenotype discovery for genetic association studies of brain imaging. Commun. Biol. 7, 414 (2024).Article  PubMed  PubMed Central  Google Scholar Zhou, Y. et al. Applying multimodal AI to physiological waveforms improves genetic prediction of cardiovascular traits. Am. J. Hum. Genet. 112, 1562–1579 (2025).Article  PubMed  PubMed Central  Google Scholar Yun, T. et al. Unsupervised representation learning on high-dimensional clinical data improves genomic discovery and prediction. Nat. Genet. 56, 1604–1613 (2024).Article  PubMed  PubMed Central  Google Scholar Sergouniotis, P. I. et al. Autoencoder-based phenotyping of ophthalmic images highlights genetic loci influencing retinal morphology and provides informative biomarkers. Bioinformatics 41, btae732 (2024).Krishnan, R., Rajpurkar, P. & Topol, E. J. Self-supervised learning in medicine and healthcare. Nat. Biomed. Eng. 6, 1346–1352 (2022).Article  PubMed  Google Scholar Gui, J. et al. A survey on self-supervised learning: algorithms, applications, and future trends. IEEE Trans. Pattern Anal. Mach. Intell. 46, 9052–9071 (2024).Article  PubMed  Google Scholar Liu, X. et al. Self-supervised learning: generative or contrastive. IEEE Trans. Knowl. Data Eng. 35, 857–876 (2023).Google Scholar Qiao, M. et al. Temporally coherent cardiac motion tracking from cine MRI: traditional registration method and modern CNN method. Med. Phys. 47, 4189–4198 (2020).Article  PubMed  PubMed Central  Google Scholar Xie, Y. et al. Genetic InfoMax: exploring mutual information maximization in high-dimensional imaging genetics studies. Trans. Mach. Learn. Res. https://openreview.net/forum?id=9UgUMFW67X 2835–8856 (ISSN, 2024).Rakowski, A., Monti, R. & Lippert, C. TransferGWAS of T1-weighted brain MRI data from UK Biobank. PLoS Genet. 20, e1011332 (2024).Article  PubMed  PubMed Central  Google Scholar Yang, Z. et al. Brain aging patterns in a large and diverse cohort of 49,482 individuals. Nat. Med. 30, 3015–3026 (2024).Article  PubMed  PubMed Central  Google Scholar McInnes, L., Healy, J., Saul, N. & Großberger, L. UMAP: uniform manifold approximation and projection. J. Open Sour. Softw. 3, 861 (2018).Article  Google Scholar Loh, P.-R. et al. Efficient Bayesian mixed-model analysis increases association power in large cohorts. Nat. Genet. 47, 284–290 (2015).Article  PubMed  PubMed Central  Google Scholar Jiang, L. et al. A resource-efficient tool for mixed model association analysis of large-scale data. Nat. Genet. 51, 1749–1755 (2019).Article  PubMed  Google Scholar Porter, H. F. & O’Reilly, P. F. Multivariate simulation framework reveals performance of multi-trait GWAS methods. Sci. Rep. 7, 38837 (2017).Article  PubMed  PubMed Central  Google Scholar van der Meer, D. et al. Understanding the genetic determinants of the brain with MOSTest. Nat. Commun. 11, 3512 (2020).Article  PubMed  PubMed Central  Google Scholar Guo, B. et al. Efficient multi-phenotype genome-wide analysis identifies genetic associations for unsupervised deep-learning-derived high-dimensional brain imaging phenotypes. Preprint at https://doi.org/10.1101/2024.12.06.24318618 (2024).Davey Smith, G. & Hemani, G. Mendelian randomization: genetic anchors for causal inference in epidemiological studies. Hum. Mol. Genet 23, R89–R98 (2014).Article  PubMed  PubMed Central  Google Scholar Taleb, A., Kirchler, M., Monti, R. & Lippert, C. ContIG: self-supervised multimodal contrastive learning for medical imaging with genetics. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52688.2022.02024. (IEEE, 2022).Wang, A. Q. et al. A framework for interpretability in machine learning for medical imaging. IEEE Access 12, 53277–53292 (2024).Article  PubMed  PubMed Central  Google Scholar Selvaraju, R. R. et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV) 618–626 https://doi.org/10.1109/iccv.2017.74. (IEEE, 2017).Sundararajan, M., Taly, A. & Yan, Q. Axiomatic Attribution for Deep Networks. In Proc. 34th International Conference on Machine Learning vol. 70 3319–3328 (PMLR, 2017).Bonazzola, R. et al. Unsupervised ensemble-based phenotyping enhances discoverability of genes related to left-ventricular morphology. Nat. Mach. Intell. 6, 291–306 (2024).Article  PubMed  PubMed Central  Google Scholar Dufournet, A. et al. A self-supervised learning framework for discovering cortical folding patterns under genetic influence: application to the Anterior Cingulate Cortex. Imaging Neurosci. 3, IMAG.a.987 (2025).Article  Google Scholar Cardoso, M. J. et al. MONAI: an open-source framework for deep learning in healthcare. Preprint at https://doi.org/10.48550/arXiv.2211.02701 (2022).Henschel, L. et al. FastSurfer - A fast and accurate deep learning based neuroimaging pipeline. Neuroimage 219, 117012 (2020).Article  PubMed  PubMed Central  Google Scholar Tustison, N. J. et al. ANTsX neuroimaging-derived structural phenotypes of UK Biobank. Sci. Rep. 14, 8848 (2024).Article  PubMed  PubMed Central  Google Scholar Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006).Article  PubMed  Google Scholar Kingma, D. P. & Welling, M. An introduction to variational autoencoders. Found. Trends Mach. Learn. 12, 307–392 https://doi.org/10.1561/2200000056 (2019).Hondru, V., Croitoru, F. A., Minaee, S., Ionescu, R. T. & Sebe, N. Masked image modeling: a survey. Int. J. Comput. Vis. 133, 7154–7200 (2025).Article  Google Scholar Ye, K. et al. Bidirectional mapping with contrastive learning on multimodal neuroimaging data. Med. Image Comput. Comput. Assist. Interv. 14222, 138–148 (2023).PubMed  PubMed Central  Google Scholar Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In Proc. 37th International Conference on Machine Learning vol. 119, 1597–1607 (PMLR, 2020).He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr42600.2020.00975. (IEEE, 2020).Zbontar, J., Jing, L., Misra, I., LeCun, Y. & Deny, S. Barlow Twins: self-supervised learning via redundancy reduction. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) PMLR https://proceedings.mlr.press/v139/zbontar21a.html139, 12310–12320 (2021).Bardes, A., Ponce, J. & LeCun, Y. VICReg: variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations https://openreview.net/forum?id=xm6YD62D1Ub (ICLR, 2022).Grill, J.-B. et al. Bootstrap your own latent: a new approach to self-supervised learning. Adv. Neural Inf. Process. Syst. 33, 21271–21284 (2020).Google Scholar Radford, A. et al. Learning transferable visual models from natural language supervision. In Proc. 38th International Conference on Machine Learning vol. 139, 8748–8763 (PMLR, 2021).Radhakrishnan, A. et al. Cross-modal autoencoder framework learns holistic representations of cardiovascular state. Nat. Commun. 14, 1–12 (2023).Article  Google Scholar Zhou, X. et al. Choice of voxel-based morphometry processing pipeline drives variability in the location of neuroanatomical brain markers. Commun. Biol. 5, 913 (2022).Article  PubMed  PubMed Central  Google Scholar Ashburner, J. & Friston, K. J. Voxel-based morphometry-the methods. Neuroimage 11, 805–821 (2000).Article  PubMed  Google Scholar Zhao, X. et al. Genetic architecture of white matter microstructure captured by unsupervised deep representation learning of fractional anisotropy maps. Nat. Commun. https://doi.org/10.1038/s41467-026-73996-z (2026).Article  PubMed  PubMed Central  Google Scholar Deng, J. et al. ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/cvpr.2009.5206848 (IEEE, 2009).Laval, J. et al. Towards a foundation model for cortical folding. Mach. Learn. Clin. Neuroimaging https://doi.org/10.1007/978-3-031-78761-4_8 (2025).Article  Google Scholar Rivière, D., Geffroy, D., Denghien, I., Souedet, N. & Cointepas, Y. BrainVISA: an extensible software environment for sharing multimodal neuroimaging data and processing tools. NeuroImage 47, S163 (2009).Article  Google Scholar Pirruccello, J. P. et al. Analysis of cardiac magnetic resonance imaging in 36,000 individuals yields genetic insights into dilated cardiomyopathy. Nat. Commun. 11, 2254 (2020).Article  PubMed  PubMed Central  Google Scholar Fox, E. R. et al. Genome-wide association study of cardiac structure and systolic function in African Americans: the Candidate Gene Association Resource (CARe) study. Circ. Cardiovasc Genet. 6, 37–46 (2013).Article  PubMed  Google Scholar Defferrard, M., Bresson, X. & Vandergheynst, P. Convolutional neural networks on graphs with fast localized spectral filtering. Adv. Neural Inform. Process. Syst. (eds Lee, D. et al.) 29, 3844–3852 https://proceedings.neurips.cc/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html. (2016).Tarroni, G. et al. Large-scale quality control of cardiac imaging in population studies: application to UK Biobank. Sci. Rep. 10, 2408 (2020).Article  PubMed  PubMed Central  Google Scholar Deng, J. et al. ArcFace: additive angular margin loss for deep face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 44, 5962–5979 (2022).Article  PubMed  Google Scholar Kirchler, M. et al. transferGWAS: GWAS of images using deep transfer learning. Bioinformatics 38, 3621–3628 (2022).Article  PubMed  Google Scholar Kornblith, S., Norouzi, M., Lee, H. & Hinton, G. Similarity of neural network representations revisited. In Proc. 36th International Conference on Machine Learning 97, 3519–3529 (PMLR, 2019).Roeder, G., Metz, L. & Kingma, D. On Linear identifiability of learned representations. In Proc. 38th International Conference on Machine Learning (eds. Meila, M. & Zhang, T.) 139, 9030–9039 (PMLR, 2021).Kraft, P., Zeggini, E. & Ioannidis, J. P. A. Replication in genome-wide association studies. Stat. Sci. 24, 561–573 (2009).Article  PubMed  PubMed Central  Google Scholar Gallagher, M. D. & Chen-Plotkin, A. S. The Post-GWAS Era: from association to function. Am. J. Hum. Genet 102, 717–730 (2018).Article  PubMed  PubMed Central  Google Scholar Cano-Gamez, E. & Trynka, G. From GWAS to function: using functional genomics to identify the mechanisms underlying complex diseases. Front. Genet. 11, 424 (2020).Article  PubMed  PubMed Central  Google Scholar Collins, G. S. et al. TRIPOD + AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 385, e078378 (2024).Article  PubMed  PubMed Central  Google Scholar Bush, W. S., Oetjens, M. T. & Crawford, D. C. Unravelling the human genome-phenome relationship using phenome-wide association studies. Nat. Rev. Genet. 17, 129–145 (2016).Article  PubMed  Google Scholar Manolio, T. A. Bringing genome-wide association findings into clinical use. Nat. Rev. Genet. 14, 549–558 (2013).Article  PubMed  Google Scholar Arun, N. et al. Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging. Radio. Artif. Intell. 3, e200267 (2021).Article  Google Scholar Mukherjee, S. et al. EmbedGEM: a framework to evaluate the utility of embeddings for genetic discovery. Bioinform. Adv. 4, vbae135 (2024).Article  PubMed  PubMed Central  Google Scholar Islam, S. R., Xia, T., He, W., Xie, Z. & Zhi, D. Vision transformer autoencoders captures local and non-local features in brain imaging to reveal novel genetic associations. Commun. Biol. https://doi.org/10.1038/s42003-026-10430-6 (2026).Article  PubMed  PubMed Central  Google Scholar Dosovitskiy, A. et al. An image is worth 16×16 words: transformers for image recognition at scale. In International Conference on Learning Representations https://openreview.net/forum?id=YicbFdNTTy (ICLR, 2021).Machado Reyes, D., Burch, M., Parida, L. & Bose, A. A foundation model for learning genetic associations from brain imaging phenotypes. Bioinform. Adv. 5, vbaf196 (2025).Article  PubMed  PubMed Central  Google Scholar Chen, M., Mei, S., Fan, J. & Wang, M. Opportunities and challenges of diffusion models for generative AI. Natl. Sci. Rev. 11, nwae348 (2024).Article  PubMed  PubMed Central  Google Scholar Arnab, A. et al. ViViT: a video vision transformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv48922.2021.00676. (IEEE, 2021).Liu, Y. et al. Genetic architecture of 11 organ traits derived from abdominal MRI using deep learning. Elife 10, e65554 (2021).Agrawal, S. et al. Inherited basis of visceral, abdominal subcutaneous and gluteofemoral fat depots. Nat. Commun. 13, 3771 (2022).Article  PubMed  PubMed Central  Google Scholar Tobias, J. H. et al. Femoral neck width genetic risk score is a novel independent risk factor for hip fractures. J. Bone Min. Res. 39, 241–251 (2024).Article  Google Scholar Baird, D. A. et al. Identification of novel loci associated with hip shape: a meta-analysis of Genomewide Association Studies. J. Bone Min. Res. 34, 241–251 (2019).Article  Google Scholar Manichaikul, A. et al. Genome-wide study of percent emphysema on computed tomography in the general population. The multi-ethnic study of atherosclerosis lung/SNP Health Association Resource Study. Am. J. Respir. Crit. Care Med. 189, 408–418 (2014).Article  PubMed  PubMed Central  Google Scholar van der Molen, M. C. et al. Determinants of lung fissure completeness. Am. J. Respir. Crit. Care Med. 204, 807–816 (2021).Article  PubMed  PubMed Central  Google Scholar Chaudhary, S. et al. HistoGWAS: an AI-enabled framework for automated genetic analysis of tissue phenotypes in histology cohorts. Genome Biol. https://doi.org/10.1186/s13059-026-04031-z27,122 (2026).Download referencesAcknowledgementsThis work is supported by grants from 5R01AG081398, 5R01EY032768, 5U01AG070112.Author informationAuthors and AffiliationsJ.C. Walter Jr. Transplant Center, Department of Surgery, Houston Methodist Hospital, Houston, TX, USAKhush PatelMcWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USAXingzhong Zhao, Ziqian Xie & Degui ZhiDepartment of Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, CT, USAXingzhong Zhao, Ziqian Xie & Degui ZhiAuthorsKhush PatelView author publicationsSearch author on:PubMed Google ScholarXingzhong ZhaoView author publicationsSearch author on:PubMed Google ScholarZiqian XieView author publicationsSearch author on:PubMed Google ScholarDegui ZhiView author publicationsSearch author on:PubMed Google ScholarContributionsK.P., Z.X., and D.Z. conceptualized and designed the study. K.P. and Z.X. conducted the literature search. X.Z. and K.P. designed and created the figures. X.Z., Z.X., and D.Z. provided critical technical review and verified the details of the summarized studies. K.P. developed the first draft. D.Z. supervised the project and performed the final review and editing. All authors wrote and approved the final manuscript.Corresponding authorCorrespondence to Degui Zhi.Ethics declarationsCompeting interestsThe authors declare no competing interests.Peer reviewPeer review informationCommunications Biology thanks all reviewer(s) for their contribution to the peer review of this work. Primary Handling Editors: George Inglis. A peer review file is available.Additional informationPublisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Supplementary informationRights and permissionsOpen Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.Reprints and permissionsAbout this article