How well can a frozen CT foundation model diagnose disease from a 3D scan?

Summary

Together with the Netherlands Cancer Institute, kaiko establishes a public-private AI research lab: the Foundation Models for Oncology (fomofo) lab.

In our mission to develop novel foundation models specifically adapted to oncological applications and eventually more precise cancer treatment, fomofo reached a significant milestone. Our submission in the "CVPR 2026 Foundation Models for General CT Image Diagnosis" challenge won first place in the task focused on linear probing (Task 1, track 1) among 139 participants and 209 submissions to the open validation board.

Our submission topped the leaderboard on both primary metrics: balanced accuracy and AUROC, in this article we dive deeper into the solutions, results, and impact of this win.

The challenge

The CVPR 2026 challenge asked a deceptively simple question:

The task focuses on linear probing: the FM is only evaluated through training a lightweight linear classifier on top of frozen FM features. The open validation benchmark spanned 17 binary classification tasks across three datasets, covering conditions as varied as colorectal cancer, splenomegaly, COVID-19, kidney stones, and lung-nodule malignancy.

What makes this genuinely hard is the nature of CT diagnosis itself. Many tasks are a needle-in-the-haystack problem: the decisive abnormality may occupy only a handful of voxels in an entire volume, yet the model must reach a holistic verdict on the whole scan. Of the 17 tasks, 11 provide a foreground ROI mask to help localize the target condition, while the remaining 6 require reasoning over the full field of view with no such hint.

This split creates a fundamental tension. Some foundation models excel at local detail (fine-grained structures, small lesions), while others excel at global context (scan-level reasoning).

The solution

Our distillation framework fuses TAP-CT's local detail with SPECTRE's global, vision–language-aligned context, without retraining either backbone.

Our key insight: instead of choosing between a local-specialist and a global-specialist model, combine them, without retraining either one.

Given that many diagnostic tasks/labels are radiology report derived, it gives a competitive edge to have some form of vision–language alignment (VLA).

Therefore, we built a distillation framework around two prominent, publicly available CT foundation models

• TAP-CT-B-3D, our prior work, which operates on a small context window with fine patches, making it strong at local detail.
• SPECTRE, which uses larger patches and a two-stage design (a crop-level stage plus a scan-level VLA stage with Qwen 3), making it strong at global reasoning.

We distill SPECTRE's knowledge into two intermediate stages sitting between the frozen TAP-CT extractor and the linear probe. Two lightweight CNN-plus-linear heads are trained to align pooled TAP-CT features with SPECTRE's representations:

• A crop-level head aligns local TAP-CT features with SPECTRE's crop-level [CLS] token.
• A global head aligns pooled global TAP-CT features with SPECTRE's scan-level VLA [CLS] token.

Both heads are trained purely with a cosine-similarity objective while both backbones stay completely frozen. The result is an enriched intermediate representation that carries both local detail and distilled global, language-aligned context.

At probing time, we apply task-aware pooling:

• ROI tasks combine masked pooling (mean/max/std) over the foreground region, multi-scale context cubes around the mask, a surrounding "context ring," HU intensity statistics, and the distilled crop- and global-level features.
• Non-ROI tasks pool over all patch tokens, add depth-binned and top-K poolings, HU statistics, and the distilled global representation.

A single linear probe is then trained on these frozen, aggregated features. The whole pipeline leverages models that are already public, with no backbone retraining required.

The results

The tricky part about this challenge is that the participants did not know what tasks their FM was going to be evaluated on. That is, the final test set determining the leaderboard consisted of a performance evaluation on undisclosed classification tasks, where team fomofo ranked #1, leading on both primary metrics.

On the open validation set, the distillation approach reached an average AUROC of 0.815 and balanced accuracy of 0.804 across the 17 tasks, beating both backbones on their own (SPECTRE: 0.788 AUROC, TAP-CT: 0.789 AUROC).

Crucially, the per-condition results confirm our core hypothesis. TAP-CT and SPECTRE are complementary: TAP-CT is stronger on conditions like splenomegaly and adrenal hyperplasia, while SPECTRE is stronger on ascites. Combining information of both models through distillation improves the overall performance, ie the sum of the two is greater than the max performance of either. A few representative validation AUROCs:

The one trade-off: our runtime is higher than some competitors, since inference requires a sliding-window pass through TAP-CT plus passes through both distilled intermediate stages. With 9 seconds per case, it was well below the maximum runtime of 40 seconds.

The impact

The core takeaway is that we can combine models into a single unit that can capture both local detail and global context for clinical CT. We fuse the complementary strengths of two publicly available foundation models into one pipeline that handles both ROI-localized tasks and whole-scan, needle-in-the-haystack classification.

For oncology specifically, this matters: cancer findings are frequently confined to a small number of voxels, demanding both fine localization and holistic interpretation of the full scan. A single pipeline that does both moves us closer to practical, general-purpose CT evaluation.

What’s next

We're excited to keep pushing on this direction: a vision-only pretrained CT model that natively represents both fine-grained structures and the global conclusions drawn from them.

This challenge inspired us to pursue a new research project for CT FMs, with more work coming soon.

Work by Tim Veenboer, Razvan-Andrei Matisan, George Yiasemis, Jonas Teuwen, and Kevin Groot Lipman. Foundation Models for Oncology (fomofo), Netherlands Cancer Institute (NKI), Amsterdam.

With thanks to the data owners, the CodaLab challenge organizers, and the Research High Performance Computing (RHPC) team at NKI.

The models and methods described are research prototypes and have not been approved or cleared as medical devices. They are not intended for clinical diagnosis or patient care.