Aug 2026

Testing an Open Relational Foundation Model on My Laptop

Since I posted twice about relational foundation models (RFMs), quite a few friends have reached out with questions. So I've decided to write about this space more regularly — starting with experiments, not just papers.

The benchmark

I built Relagentship, a small open-source benchmark comparing four approaches:

XGBoost with relational features
Task-trained RelGT
Zero-shot Relational Transformer (RT)
RT continued-pretrained on other real databases

How RT works

RT, introduced in Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data (Ranjan et al., 2025), is an ambitious idea: instead of engineering features or training a new model for every prediction task, it samples a neighbourhood around a target row by following database relationships, converts the connected cells into tokens, and predicts using examples found inside that context. The published RT-PluRel checkpoint can therefore attempt new tasks without updating its weights.

Relational Transformer architecture: database schema and task schema, a sampled context window of cell tokens with in-context task labels, and transformer blocks with column, feature, neighbour and full attention feeding a per-datatype decoder. — How RT works: a sampled context window of cells from related tables is tokenised, then processed by relational attention layers that predict the masked target cell. Figure from Ranjan et al., "Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data" (arXiv:2510.06377).

Results

I tested it on two RelBench tasks. Both ran locally on a 48 GB Mac in roughly 18 hours.

On Formula 1 driver-top3, continued-pretrained RT was competitive: 0.844 AUROC, compared with 0.855 for XGBoost and 0.804 for RelGT.

But on the larger clinical-trial study-outcome task, the result changed sharply. Zero-shot RT achieved 0.515 AUROC — almost random ranking — and continued pretraining improved it only to 0.590. XGBoost and RelGT reached 0.718 and 0.699 respectively.

More strikingly, at its validation-selected threshold, zero-shot RT predicted the positive class for every test row. Its apparently respectable F1 score therefore concealed a serious failure mode.

Caveats

This does not prove that RT is fundamentally broken. The comparison is not apples-to-apples: XGBoost and RelGT receive task-specific training, while RT uses published checkpoints. These are also fixed-seed local experiments.

Why this rhymes with OpenRFM

However, the result rhymes closely with the diagnosis in the recent OpenRFM paper. The authors argue that RT depends on labels encountered during its sampled relational walk. When too few label-bearing rows appear, its in-context support becomes sparse and prediction can collapse into something resembling underdetermined kernel regression. They also find that synthetic-only pretraining may remain in a "lazy" regime rather than learning genuinely useful relational features.

Interestingly, study-outcome is one of the failure cases highlighted in that paper.

Why this matters

A model used to choose actions cannot merely work on average — we need to understand when its relational context contains enough evidence, when its pretraining assumptions transfer, and when a confident-looking metric hides collapse.

Next, I want to examine these failure modes directly: label coverage, neighbourhood composition, calibration, and whether OpenRFM's proposed fixes change the result.

Looking forward

If RFMs eventually solve the prediction layer of enterprise ML the way LLMs solved much of NLP, I wonder whether policy decisioning becomes the next bottleneck for us MLEs to solve.

All notebooks, predictions, configurations, and reproducibility checks are in the repo.