A held-out test set answers one question: can the model handle another example of what it was trained on. Out-of-distribution generalization is a different question: can it handle a case unlike its training data. Most validation numbers answer the first and get read as the second.
I ran into this in research on surrogate models, described in a note about a model that scored perfect. Here is the same failure, small enough to compute by hand.
The scale: R², where zero means guessing
R² compares a model's squared errors with those of a baseline that ignores the input and always answers with the average of the true values. R² = 1 is perfect. R² = 0 means the model did exactly as well as that baseline. Below zero means it did worse, so the model is adding error, not removing it.
Train a line on a curve
The true relation is y = x². The training data covers only x from 0 to 1: the points (0, 0), (0.5, 0.25) and (1, 1). Fit a straight line by least squares and you get y = x − 1/12, about x − 0.083.
A straight line is the wrong model, but inside 0 to 1 the curve is gentle and the line follows it closely.
Inside the training range
Test on new points the model never saw, still between 0 and 1: x = 0.25 and x = 0.75. The true values are 0.0625 and 0.5625. The line predicts 0.1667 and 0.6667, off by about 0.10 each time.
R² on those points is 0.83. On five new points spread from 0.1 to 0.9 it is 0.86. By the usual held-out test, this is a good model.
Outside it
Now test at x = 2 and x = 3. The true values are 4 and 9. The line predicts 1.92 and 2.92, off by 2.1 and 6.1.
R² on those points is −2.31. The same model, with the same fitted coefficients, is now far worse than ignoring the input and guessing the average. Nothing about it changed. Only the data it was asked about did.
Why the real problem is harder
The toy fails because the new inputs are bigger than anything it saw. That kind of shift is easy to spot: check the input range and refuse to answer outside it. The failure I measured was subtler, and the difference is the part worth knowing.
I trained models to predict the stiffness of metal sheets with patterns of holes, from a picture of the pattern. Stiffness depends mostly on how much metal is left, so I first forced every family of patterns into the same density band, 59 to 65 percent solid. The inputs were then in the same range by the most obvious measure. Only the shape changed: round holes, squares, ovals, crosses, honeycombs, random lattices.
Trained on one family and tested on another, 20 of the 30 combinations scored below zero. A random forest, a completely different kind of model, showed the same pattern. The models learned a family of hole-shaped things, not how stiffness works, and no range check would have caught it.
In a follow-up with 22 families, the choice of which families to train on moved the worst-case result by a factor of about 390, and none of the nine cheap rules I tested could predict which choice would be good. That is in its own note.
What to do with this
Build at least one test set from a different process than the training set, not just held-out rows from the same one. Control the obvious confounds first, so that a failure means what you think it means. And report the worst case on new conditions, not only the average on familiar ones.
The data, the simulator and the from-scratch network behind the stiffness results are open at doi.org/10.6084/m9.figshare.32993597.