From Clinical Notes to MRI Scans: How Multimodal AI Creates Better Research Insights

From Clinical Notes to MRI Scans: How Multimodal AI Creates Better Research Insights

Published: Oct 29th, 2026

A radiologist asked to read an MRI without the patient's history, medications, or prior reports is working with less than the full picture, and the evidence says it shows. A systematic review of imaging and EHR fusion notes that a lack of access to clinical and laboratory data during image interpretation has repeatedly been shown to reduce performance and clinical utility for the referring provider (Huang et al., npj Digital Medicine, 2020). The same logic applies to a model. An algorithm that sees only pixels is missing context that a clinician would never ignore.

That is the position much of today's deployed imaging AI is in. Radiology accounts for roughly three-quarters of the devices on the FDA's list of AI-enabled medical devices, a share that has stayed between 75 and 80 percent in published analyses (Lee and Youn, JMIR, 2026), and the list has passed 1,400 authorisations (MedTech Dive, 2026). These tools are valuable, but they are built around analysing images. Multimodal AI healthcare programmes aim to close the gap by reading clinical notes, imaging, laboratory values, and other sources together.

For research teams, the question is practical: does combining these sources produce insights that a single source cannot, and what does it take to do so reliably? This article covers how models read notes and scans, how the streams are combined, where the evidence is strong and where it is thin, and what a clinical intelligence platform needs to provide for the results to be usable and defensible.

What multimodal AI in healthcare means in practice

Multimodal AI healthcare refers to models that learn from more than one type of data at once. Where a data platform brings these sources together and makes them consistent, the AI layer extracts meaning from each and relates them to one another. Two of the most common inputs, clinical notes and MRI scans, show how different the underlying work is.

Research activity reflects how quickly the approach is maturing. A scoping review of multimodal AI in medicine counted three qualifying studies in 2018 and 150 in 2024, and the same review is reported to have found multimodal imaging models outperforming single-modality counterparts by roughly six percentage points of area under the curve (Cureus Blog, 2026, summarising the review). Practitioner overviews describe the goal in similar terms: to mirror the way clinicians already combine visual findings with history, laboratory results, and context (Gleecus, 2026).

How a model reads clinical notes

Clinical notes are unstructured text written for people, not databases. Natural language processing identifies the clinical entities in them, such as diagnoses, medications, procedures, and symptoms, along with qualifiers like severity, negation, and timing, then maps them to standard terminologies such as SNOMED CT, RxNorm, and LOINC. The value is in what structured fields omit: the clinician's assessment, the reason a treatment was changed, and how a condition has progressed between visits.

How a model reads an MRI scan

An MRI is a high-dimensional volume rather than a single image. Deep learning models, typically convolutional or transformer-based, convert it into numerical representations that capture features such as lesion size, tissue characteristics, or patterns of atrophy. Those features can be tracked quantitatively across scans, which makes imaging a source of measurable biomarkers rather than only a visual impression. On its own, though, that output has no knowledge of the patient's symptoms, history, or treatment.

How the streams are combined

The central design decision in multimodal AI healthcare is when to combine the modalities. A scoping review of studies that fused EHR and imaging data found early fusion to be the most widely used approach, appearing in 22 of 34 studies (Mohsen et al., Scientific Reports, 2022), while a 2026 survey identifies intermediate fusion as the most effective strategy overall (Vavekanand et al., 2026). A 2025 review in Information likewise treats early, intermediate, and late fusion as the three main strategies in medical diagnostics, and adds attention-based and architecture-search designs that learn how to combine modalities rather than fixing the design in advance (Jandoubi and Akhloufi, 2025). The trade-offs are summarised below.

There is no universally best choice. The right strategy depends on how complete the data is, how well the modalities line up in time, and how much training data is available, which is why the platform around the model matters as much as the model itself.

Where combined evidence produces better research insights

Multimodal approaches are not valuable simply because they use more data. They are valuable where the question genuinely depends on evidence that lives in different places. Four research applications illustrate this.

Disease characterisation and progression

In neurological and oncology research, where imaging findings and clinical course have to be read together, multimodal models can relate changes on an MRI to the symptoms, treatments, and assessments recorded in notes. That supports a more granular view of progression than either source provides alone, and it helps separate patients who look similar on imaging but behave differently clinically. In dementia research, datasets such as ADNI and NACC pair structural MRI with cognitive assessments, biomarkers, and clinical follow-up, and the review literature describes models that fuse MRI-derived features with cognitive and demographic variables for dementia staging (Jandoubi and Akhloufi, 2025). Interpreting serial scans alongside treatment history and clinical markers also supports response monitoring in oncology (Gleecus, 2026).

Cohort building and trial eligibility

Eligibility criteria often combine an imaging-defined finding, such as lesion burden, with a clinical-history requirement documented only in narrative text, such as prior lines of therapy. NLP applied to notes, together with imaging-derived features, lets research teams identify candidate cohorts that queries on coded data alone would miss, and estimate feasibility before a protocol is finalised.

Risk prediction

The evidence for combining modalities in prediction is encouraging but nuanced. A medRxiv preprint that combined vital signs, laboratory tests, chest X-ray, and radiology notes found that multimodal models outperformed unimodal models across several risk prediction tasks, while also finding that the time-series signals carried more information than the images or notes in all three tasks (medRxiv, 2023). Modalities do not contribute equally. Ablation testing, which measures what each source adds, is needed to show that the added complexity is earning its place. 

Studies summarised in a 2025 review in Information show a similar pattern: a model combining hip fracture radiographs with structured EHR data predicted 30-day mortality with an AUROC of 84 percent, against 80 percent for images alone and 78 percent for EHR data alone, and a paediatric appendicitis model combining ultrasound with EHR variables reached 86 percent, against 76 percent and 79 percent for the single-modality versions (Jandoubi and Akhloufi, 2025). These are individual studies reported in a review, so they illustrate the pattern rather than prove it will hold in every setting.

Biomarker and endpoint discovery

Imaging-derived measures are increasingly used as candidate biomarkers and exploratory endpoints. Linking them to clinical notes, laboratory trends, and, where available, genomic data helps test whether an imaging signal tracks something clinically meaningful or merely something measurable. It is the same discipline that determines whether a biomarker generalises beyond the dataset on which it was found.

Where multimodal AI still falls short

The honest summary is that the evidence cuts both ways. Enthusiasm outpaces reproducible clinical impact, and it is worth setting out why. A systematic review of 69 studies identified five recurring methodological challenges in multimodal medical modelling: missing modalities, small datasets, imbalance in dimensionality, interpretability, and the selection of an appropriate fusion strategy. The review noted that these challenges interact rather than occurring in isolation (Farhadizadeh et al., 2025).

Missing modalities deserve particular attention. In real clinical data every patient has notes, but only some have an MRI, and imaging is usually ordered for a reason. A scoping review of multimodal AI in medicine found that 69 percent of the included papers handled missing data by excluding any entry with a missing modality, which reduces sample size and introduces selection bias that can compromise validity and generalisability (arXiv, 2024). A model validated only on complete-case patients may behave very differently on the broader population it will actually be used on.

A 2026 survey of multimodal machine learning in predictive healthcare similarly points to modality misalignment, missing data, and limited external validation as persistent gaps, and to regulatory constraints and interoperability limits as barriers to deployment (Vavekanand et al., 2026). For research teams, the implication is practical: results should be tested on external data, reported with and without each modality, and accompanied by explanations that a clinical reviewer can assess.

Practical constraints compound these methodological ones. A Cureus Blog overview of recent research lists scarce labelled multimodal datasets, with privacy rules limiting how much clinical data institutions can share, misalignment between modalities that operate on different scales and timelines, the absence of standardised benchmarks, limited interpretability, and the cost of training or fine-tuning large models (Cureus Blog, 2026). The 2025 Information review names missing modalities, generalising across institutions, and achieving regulatory approval as challenges that remain open (Jandoubi and Akhloufi, 2025). The Cureus overview adds that most published work is still at the research and validation stage rather than in routine clinical deployment.

What a clinical intelligence platform adds

A model is only one component. A clinical intelligence platform is the layer that makes multimodal outputs usable: it connects harmonised data to the models that interpret it, and connects the models' outputs to the people who must act on them. This is where healthcare AI analytics and healthcare data intelligence move from experiments to repeatable capability. Four functions matter most.

Cross-modal orchestration. NLP on notes, imaging pipelines on scans, and fusion models on the combined output run as a managed, versioned workflow, rather than as separate scripts maintained by separate teams.

Traceability from insight to source. Every output links back to the note passage, scan series, and data transformation behind it, so a reviewer can verify why the model reached its conclusion.

Explainability and human review. Outputs are presented in a form clinicians and researchers can interrogate, with a defined point at which a person reviews or overrides the result.

Monitoring and change control. Performance is tracked as data, scanners, and documentation practice change, with defined triggers for retraining and a record of which model version produced each analysis.

None of this is visible in a demonstration, and all of it determines whether a result can be reproduced six months later. It is also what turns a model trained on research data into one that can run inside a pharmaceutical or health system's production environment. The harmonisation burden is substantial in trial settings, where study data follow CDISC standards, EHR data are coded in ICD and SNOMED CT, genomic data use their own ontologies, and wearable data arrive as continuous time series. Without a normalisation layer, multimodal outputs risk being computationally sophisticated but disconnected from the standards that regulatory submissions must meet (Pienomial, 2026).

Governance and regulatory considerations

Reading more kinds of data raises the governance bar in specific ways, and the regulatory position depends on how the output will be used.

Privacy. Clinical notes routinely contain names, dates, and locations in free text, and MRI files carry identifiers in their DICOM metadata and sometimes in the image itself. De-identification under HIPAA, through Safe Harbor or Expert Determination, has to account for both, and automated de-identification of text should be validated before it is relied on.

FDA. The January 2025 draft guidance, “Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products” [FDA-2024-D-4689], sets out a risk-based credibility framework built around a defined context of use, with seven steps for planning, gathering, and documenting evidence of model credibility. It applies where an AI model generates information intended to support a regulatory decision on safety, effectiveness, or quality, and it excludes AI used in drug discovery and for operational efficiency. Separately, a model intended for clinical use may be regulated as a medical device, in which case the FDA's final guidance on predetermined change control plans, issued in December 2024, describes how planned model updates can be specified in advance.

EMA. The “Reflection Paper on the Use of Artificial Intelligence in the Medicinal Product Lifecycle”, adopted on 30 September 2024, covers AI across the full product lifecycle, recommends early regulatory engagement where a system may influence a product's benefit-risk balance, and stresses continuous monitoring for model drift.

Validation. Where a model supports a GxP-regulated activity, GAMP 5 Second Edition (2022) and 21 CFR Part 11 audit trail and record integrity controls still apply. A model that keeps updating itself raises change control questions that belong in validation planning, not after go-live.

Why integration decides whether multimodal AI delivers

The research insight does not come from the model alone. It comes from the combination of complete, linked data, models validated against clinical reality, and a platform that keeps the whole chain traceable. Building a capable multimodal model on a curated dataset is increasingly achievable. Running it on real-world data with missing modalities, shifting documentation practice, and regulatory scrutiny, and earning the trust of the clinicians and scientists who rely on it, is the harder and less standardised part.

That combination of clinical fluency and engineering depth is where teams tend to need support. Reveal HealthTech's life sciences practice is one example, pairing clinical and product expertise with hands-on AI and cloud engineering on HIPAA-compliant infrastructure, with case work spanning biomarker modelling on time-series clinical data and MLOps deployment for pharmaceutical clients running several production models at once.

Frequently Asked Questions