Shuffle the Test Set and Protein AI's Best Scores Collapse

A score of 0.482, and then almost nothing#

A team at the University of Delaware set a general-purpose machine learning probe the task of predicting the optimal pH of an enzyme, the acidity at which it works best. Using standard practice, the probe reached an R-squared of 0.482, edging past EpHod, a specialist model for this exact task that had been selected from 11,550 training runs. Then they re-split the same data so that proteins in the test set were not close evolutionary relatives of proteins in the training set. Every pooled-embedding probe collapsed to near-zero R-squared. Only one approach, a transformer probe reading the full per-residue output, recovered appreciable signal, at 0.248 (Rafailidis et al., 2026, preprint, not peer-reviewed).

Same data. Same models. Roughly half the performance in the best case, and nothing at all in the common case.

If that holds up, it has implications well beyond one enzyme property, because the way those two splits differ is the way most protein language model results are produced.

Background: what these models are, and how we grade them#

A protein language model is trained the way a text model is, except the sentences are proteins. A protein is a chain built from 20 amino acids, writable as a string of letters. Hide some letters, train a model to predict them across hundreds of millions of natural sequences, and it absorbs statistical regularities that turn out to encode real biophysics. ESM-2, often written ESM2, is the best-known example in open use.

Three more terms, because the argument depends on them.

An embedding is the vector of numbers a model produces to represent a sequence or a residue. Pooled embeddings average those per-residue vectors into a single vector for the whole protein, which is convenient and by far the most common way people use these models downstream. A probe is a small model trained on top of frozen embeddings to predict some property, which tests how much useful information the embeddings already contain.

Finally, R-squared measures how much of the variation in the true values a prediction captures. One is perfect. Zero means you would have done just as well guessing the average every time.

Now the thing that matters. Homology means shared ancestry. Proteins in the databases are not independent draws; they come in families of relatives with similar sequences and similar functions. Split such a dataset at random and you will almost certainly put close relatives on both sides of the train-test divide. The model can then score well by recognising a familiar family rather than by learning anything generalisable. That is homology leakage, and a homology-controlled split is a split designed to prevent it by keeping related sequences together on one side.

What the new benchmarking work found#

The preprint introduces Protify, an open-source platform for standardising protein language model evaluation across zero-shot, supervised and transfer-learning settings, currently supporting 35 pretrained models and 61 curated datasets (Rafailidis et al., 2026, preprint, not peer-reviewed). Two of the authors disclose that they are co-founders with an equity stake in Synthyra, which is worth knowing when reading a paper that evaluates competing models.

The enzyme pH result above is the sharpest finding. The chart shows all four numbers together.

A second finding is quieter but arguably more useful. Across 32 supervised tasks spanning enzyme classification, gene ontology annotation, protein-protein interaction, subcellular localisation and other protein function prediction problems, no single model dominated. Fourteen different models took the top score on at least one dataset, and the margins between best and second-best fell within run-to-run variability (Rafailidis et al., 2026, preprint, not peer-reviewed).

Read that twice. On most of these tasks, the difference between the winner and the runner-up was smaller than the difference you would get by re-running the same model with a different random seed. Leaderboards built on margins like that are measuring noise. It is the same failure mode that has dogged AI benchmark design in other fields, and it feeds directly into the wider reproducibility crisis in machine learning for science.

This is not one preprint's complaint#

A single unreviewed paper is a reason to pay attention, not a reason to believe. So it matters that homology leakage has an established peer-reviewed literature, and that the independent findings point the same way.

The most pointed evidence comes from drug discovery. Researchers built PDBbind CleanSplit, a training set curated by a structure-based filtering algorithm that removes train-test leakage and internal redundancy. Retraining current top-performing binding affinity models on it caused their benchmark performance to drop substantially, which the authors read as showing that the performance of existing models was largely driven by data leakage (Graber et al., 2025). That appeared in Nature Machine Intelligence, not on a preprint server.

In protein-protein interaction prediction, a systematic investigation found that overlaps between training and test sets arising from random splits, together with sequence similarity and node degree information, accounted for a great deal of reported deep learning performance (Bernett, Blumenthal & List, 2024).

And the problem has now been quantified in a protein-specific workflow. Building a leak-minimised subcellular localisation benchmark, researchers measured a systemic data leakage rate of 5.2% plus or minus 0.32 in conventional homology augmentation workflows, even when sequence similarity searches were restricted to just 10% of the training set (Ouso & Pollastri, 2026).

Three independent groups, three different sub-problems, one consistent conclusion: a meaningful share of reported performance in protein machine learning is recognition of relatives rather than generalisation.

Why pooled embeddings fail first#

The detail I find most instructive is which approach survived.

Pooled embeddings average across every residue in a protein to produce one vector. That average is an excellent summary of what family a protein belongs to, because family membership is a whole-sequence property. It is a much poorer summary of where the specific residues sit that determine, say, the pH at which an active site works.

So when relatives are present on both sides of the split, a pooled embedding looks brilliant: it has essentially been handed the answer key. Remove the relatives and the same vector has little left to offer. The transformer probe reading full per-residue embeddings retained some signal, at 0.248, which fits the interpretation that residue-level information carries genuine mechanistic content that averaging throws away (Rafailidis et al., 2026, preprint, not peer-reviewed).

That is a practical message for anyone building on these models. The convenient default is also the one most exposed to leakage.

What better evaluation looks like#

None of this means protein language models do not work. They plainly do, and the zero-shot fitness prediction results that made the field's reputation are measured against experimental assays rather than held-out database entries. ProteinGym, the standard benchmark here, is built on over 250 standardised deep mutational scanning assays covering millions of mutated sequences (Notin et al., 2023). The Protify team's own ProteinGym run reports DPLM3B as the current top self-supervised encoder (Rafailidis et al., 2026, preprint, not peer-reviewed).

What the accumulated evidence does mean is that the supervised side of the field needs different habits. Report results under a homology-controlled split alongside the random split, rather than instead of it, because the gap between the two is itself the informative number. Treat a margin smaller than seed-to-seed variance as a tie. Say which split was used, in the abstract, because at present a reader usually cannot tell.

Those are unglamorous asks. They are also the difference between a field that knows how good its tools are and one that merely hopes.

The next time a protein language model posts a state-of-the-art number, the useful question is no longer which architecture it used. It is whether anything in the test set shares an ancestor with something in training.

Frequently asked questions#

What is a protein language model? A model trained on large numbers of natural protein sequences in the same way text models are trained on sentences, learning statistical patterns that encode real biophysics. ESM-2 is a widely used open example, and such models now underpin much of computational protein engineering.

What is homology leakage? Proteins come in families of evolutionary relatives. If a dataset is split at random, close relatives end up in both the training and test sets, so a model can score well by recognising a familiar family rather than generalising. A homology-controlled split keeps relatives on one side to prevent this.

What exactly did the new work report? On enzyme optimal pH prediction, a general-purpose probe reached R-squared of 0.482 under a standard split, against 0.425 for a specialist model. Under a homology-controlled split of the same data, pooled-embedding probes fell to near zero and only a transformer probe over full residue embeddings held up, at 0.248 (Rafailidis et al., 2026, preprint, not peer-reviewed).

Is this finding peer reviewed? No. It is a bioRxiv preprint posted on 5 November 2026 and has not been certified by peer review. Two of its authors also disclose an equity stake in a company in this space. The broader phenomenon it describes, however, is documented in peer-reviewed journals.

What is the peer-reviewed evidence for data leakage in machine learning for proteins? Retraining top binding affinity models on a leak-free split caused substantial performance drops (Graber et al., 2025); much of the reported advantage in protein-protein interaction prediction traced to train-test overlap (Bernett, Blumenthal & List, 2024); and a 5.2% systemic leakage rate was measured in conventional homology augmentation (Ouso & Pollastri, 2026).

Does this mean protein language models do not work? No. It means a portion of reported supervised performance reflects evaluation design rather than model capability. Zero-shot results measured against experimental assays, such as those in ProteinGym, are a different and sturdier kind of evidence (Notin et al., 2023).

Why did pooled embeddings fail worst? Averaging across all residues produces an excellent summary of which family a protein belongs to and a poor summary of which specific residues drive a given property. With relatives removed from the test set, family recognition stops helping.

What should researchers do differently? Report performance under both a random and a homology-controlled split, treat differences smaller than run-to-run variability as ties, and state the splitting strategy explicitly. Benchmarks such as ProteinGym and platforms that make splits replayable help, but only if the splits are reported.

References#

1. Bernett, J., Blumenthal, D. B., & List, M. (2024). Cracking the black box of deep sequence-based protein-protein interaction prediction. Briefings in Bioinformatics, 25(2), bbae076. https://doi.org/10.1093/bib/bbae076
2. Graber, D., Stockinger, P., Meyer, F., Mishra, S., Horn, C., & Buller, R. (2025). Resolving data bias improves generalization in binding affinity prediction. Nature Machine Intelligence. https://doi.org/10.1038/s42256-025-01124-5
3. Notin, P., Kollasch, A. W., Ritter, D., van Niekerk, L., Paul, S., Spinner, H., Rollins, N., Shaw, A., Weitzman, R., Frazer, J., Dias, M., Franceschi, D., Orenbuch, R., Gal, Y., & Marks, D. S. (2023). ProteinGym: large-scale benchmarks for protein design and fitness prediction. (Preprint, not peer-reviewed, subsequently presented at NeurIPS 2023.) https://doi.org/10.1101/2023.12.07.570727
4. Ouso, D., & Pollastri, G. (2026). Developing SCL2205: a protein sequence-based spatial modelling dataset for the protein language model frontier. Bioinformatics. https://doi.org/10.1093/bioinformatics/btag666
5. Rafailidis, N., Hallee, L. P., Peleg, T., Horger, C., Haviland, L., & Gleghorn, J. P. (2026). Protify: a benchmarking platform revealing task dependence and homology leakage in protein language model evaluation. bioRxiv, posted 5 November 2026. (Preprint, not peer-reviewed.) Competing interests declared: two authors are co-founders with an equity stake in Synthyra, PBLLC. https://doi.org/10.64898/2026.11.04.764373