There is a genre of number in machine learning that I have started to distrust, and this is the story of how I produced one myself and nearly believed it.
The genre is the transferability score. You have a model trained on one kind of data and you want to know, before spending anything, whether it will hold up on a different kind. That is the out-of-distribution question, and it is the one that decides whether a model survives deployment. So you compute some distance between the two datasets and use it as a warning light. Several of these exist in the literature. They are attractive because they are cheap and because they promise to answer a question that is otherwise only answerable by doing the expensive thing.
I tested one on a benchmark I built. It worked. Then it stopped working for a reason that I think generalises well past my corner of engineering.
the number that looked real
The setting: twenty-two families of mechanical metamaterial, perforated sheets and strut lattices, each with a surrogate model fitted to predict its stiffness. The question is which two families you should train on so that the model survives contact with the other twenty. Some choices are hundreds of times better than others. The prize is a rule that finds the good ones in advance.
One candidate rule came from the domain-generalization literature, where it is used to choose source datasets for fault diagnosis: take the maximum mean discrepancy from any unseen family to your training pool, and pick the subset that minimises the worst one. Sensible. Cheap. Prior art, so not my idea to defend.
It scored a rank correlation of 0.54 against the achieved outcome. Then 0.54 again at a different budget, on an independently drawn set of subsets. Then 0.50 and 0.54 on a second target.
Four numbers, two of them identical to two decimal places, across samples that share no subsets. In my experience that pattern means you have found something. Noise does not usually repeat itself that politely.
the check that almost nobody runs
Everything above used a random forest. I had a second model sitting in the repository from earlier work, a convolutional network written from scratch in NumPy, and it seemed cheap to re-run 45 of the combinations through it as a robustness check.
Two things came out of that, and they point in opposite directions.
The first is reassuring. The two models substantially agree on which training sets are good: rank correlation 0.53 on one target and 0.73 on the other. Bagged trees on raw pixels and a convolutional network are about as different as two things that both do regression can be, and they rank the choices similarly. The phenomenon is a property of the data, not of the algorithm.
The second is the reason for this note.
| criterion | under the forest | under the network |
|---|---|---|
| maximum mean discrepancy | 0.54, 0.54 | 0.14 |
| sliced-Wasserstein | 0.39, 0.47 | 0.02 |
| geometric coverage | 0.28, 0.14 | -0.15 |
| directional coverage | 0.19, 0.14 | 0.42 |
The best result I had went from 0.54, stable across two budgets, to 0.14 and not statistically distinguishable from zero. The second-best went from 0.39 and 0.47 to 0.02, which is to say it became a coin.
what that actually means
Hold the two halves together, because the contrast is the whole finding.
Which training sets transfer well is stable across model classes. Which criterion predicts that is not.
So the thing I measured was not a property of the transfer problem. It was a property of how a random forest happens to carve up raw pixel space, dressed as a property of the data. Change the shape of the model and the signal is gone, even though the underlying fact it claimed to predict has not moved at all.
The directional-coverage criteria make the same point from the other side. They are weak-to-useless under the forest and become the second-strongest group under the network, 0.42 to 0.47. That is consistent with a convolutional model being more sensitive to directional structure than an axis-aligned tree ensemble, which is a perfectly reasonable thing to be true, and it means the ranking of criteria is not a fact about the world either. Which criterion looks best can depend on the model you happened to fit.
One criterion is positive under both models on both tested targets. It is also strongly negative on the third target under the forest, which was not re-tested under the network. I cannot report anything that is robust across both the target and the model at once.
why I am writing this up as its own thing
Because a single-model study cannot see this, and most studies use a single model.
If I had stopped where the result was good, I would have written a paper reporting a stable transferability criterion with a correlation of 0.54 replicated across two independent budgets. That paper would have been wrong in a way that no reviewer could have caught, because every number in it is correct. The replication is real. The stability is real. It is stable because the model was held fixed, and nobody thinks to report that as a limitation because nobody thinks of it as a choice.
There is a related trap one level down. I originally ran everything in a compressed feature space, which is the normal thing to do. The rankings of training sets survive decompression fine, correlations of 0.47 to 0.82. The criterion correlations do not: the physics-based ones fell from about 0.45 to about 0.28 and the distributional one rose from about 0.06 to about 0.20 when I took the compression out. So a criterion can also be measuring your preprocessing.
Predicting out-of-distribution generalization from a cheap statistic is a reasonable thing to want, and I still want it. But the practical suggestion I can actually support is small and boring. If you propose a metric that predicts whether a model will generalise, validate it against at least two model classes before anyone uses it. And if you are reading someone else's, check how many they used.
the general version
A measurement can be perfectly reproducible and still be about the instrument.
Reproducibility tells you the procedure is stable. It says nothing about what the procedure is pointed at. I ran mine twice, got the same answer, and the agreement was between two runs that shared the assumption that mattered, so of course they agreed.
The way you find out is by varying the thing you were not varying. That is uncomfortable to do late, because it is exactly the move that can delete your best result, and it deleted mine.
The benchmark is MetaXfam22. Data, code and every per-subset score are open, so a criterion can be scored against the full answer key without recomputation: https://doi.org/10.5281/zenodo.21601041