A while ago I wrote about training a network to predict how stiff a metal sheet is when you punch holes in it, and finding that it fell apart the moment the holes changed shape.
The proper names, for anyone who wants to search further: the sheets are mechanical metamaterials, the network is a surrogate model standing in for a physics solve too slow to run inside a design loop, and the way it fails is out-of-distribution generalization. I will mostly not use those words again.
The natural response to that is not despair. It is procurement. If a model only works on the shapes it was fed, then feed it better shapes. Simulating a new family of designs costs a few hundred slow physics solves, so you can afford a handful of them. Which handful?
That is a real engineering question with a budget attached, and as far as I can tell nobody had measured the answer. So I did.
The short version: the choice matters enormously, far more than the choice of model. And I could not find a way to make the choice.
The setup, and why it took so long
Twenty-two families of pattern this time, not six. Circles, squares, crosses, six-pointed stars, hexagons, diamonds, ovals, slots, honeycombs, kagome lattices, chevrons, re-entrant cells, and a dozen more. Six hundred cells each, at 48 by 48.
Pick two families to train on. Test on the other twenty. Score yourself on the worst of those twenty, because in deployment you do not get to average over the case that ruins you. Then do that for all 231 possible pairs, and again for 200 four-family combinations.
Most of the work was not the experiment. It was making the pool honest enough that the experiment meant something.
Stiffness is dominated by how much metal is left, so every family is rejection-sampled into the same density band and then histogram-matched cell by cell, until the family averages sit between 0.587 and 0.664 solid. An earlier version of this pool had three families that could not physically reach the bottom of the band: a rectangular hole with sides capped at 0.75 and 0.45 has area at most 0.34, so its plate is always at least 66 percent solid. Those families showed spectacular cross-family errors, and the errors were mostly the model being asked to extrapolate density. I had to re-parameterize the families by hole area and aspect ratio independently to kill it.
Every cell also has to actually connect to itself in both directions. Without that filter, 89 percent of the random-lattice cells are floating debris whose stiffness is the stiffness of the void. Their C11 over C22 ratio is one numerical zero divided by another. Unfiltered, that family looks strongly directional. Filtered, its median ratio is 1.012, which is to say perfectly symmetric. Any statistic computed on the unfiltered pool is an artefact of the disconnected cells.
That kind of thing is most of what building a benchmark is.
The spread
Worst-case score on the held-out families, by which pair you trained on:
| target | best choice | median | worst choice |
|---|---|---|---|
| C11, two families | -7.86 | -25.07 | -78.77 |
| C22, two families | -7.80 | -80.62 | -275.76 |
| C33, four families | -3.28 | -99.51 | -1273.24 |
Zero on that scale means you did exactly as well as ignoring the input and always answering with the average. Everything above is worse than that.
The bottom row is the headline. Same model, same amount of data, same everything, and the only difference is which two-or-four families you happened to buy. Best to worst is a factor of about 390.
That is a bigger lever than any architecture change I have seen reported in this literature. It is sitting in the part of the pipeline that most papers describe in one sentence of the methods section.
It is not a quirk of one model
First thing I checked, because a result this large is usually a bug.
I re-ran 45 of the same combinations through a completely different kind of model, a convolutional network I wrote from scratch in NumPy. Not a re-tuned forest. A different family of algorithm with a different set of biases.
The two agree on which training sets are good ones: rank correlation 0.53 on one target and 0.73 on another. Whatever makes a training set transfer well is a property of the training set, not of the thing you fit to it.
So the lever is real. Which makes the next part worse.
Nine ways to choose, and none of them work
The whole point of measuring the spread is to then find a cheap rule that lands near the top of it. Cheap meaning: computable before you spend the simulation budget.
I tested nine. Four look only at the pictures. Five spend a pilot of twenty solves per candidate family, which is about four seconds against the three hundred solves it costs to actually populate one.
The standard heuristic in this situation is to pick the most geometrically dissimilar families you can, on the theory that diversity buys coverage. On the smaller pilot pool, where I could enumerate every subset and score the rule against the true best one, it came in at a mean regret of 13.58. Picking subsets at random scored 13.90. The heuristic everybody reaches for is worth roughly nothing.
The best performer was a distribution-distance measure borrowed from the domain-generalization literature, where it is used to pick which source datasets to train on. On two of the three stiffness components it was genuinely good and remarkably stable: correlation of 0.54, 0.54, 0.50 and 0.54 across two different budgets with independently drawn subsets. You do not often see a number repeat itself that cleanly.
On the third component it was negative. Minus 0.12 and minus 0.10. A rule that helps you on two axes and actively misleads you on the third is not a rule.
And the criterion I expected to win, the one my own previous paper pointed at, came last. I had argued that these models fail across families because the training families are symmetric and never show the model what directional stiffness looks like. That diagnosis is, I still think, correct. It is also useless as a purchasing rule: across every variant of it and every job, the correlation never got above 0.26 and was frequently indistinguishable from zero.
Every subset I tested that contains any directional family breaks that symmetry. They still differ by two orders of magnitude. Knowing why a model fails did not tell me what to buy.
the number I have to lead with
Look at the best column in that table again. Every single one of those is negative.
Even with perfect hindsight, even choosing the optimal training families out of all 231 options by cheating and looking at the answers, the resulting model still does worse than a constant on the worst family it meets. Selection buys you a factor of 390 on a problem that is not solved at either end.
I want that stated first rather than buried, because the framing where selection is a fix is available and wrong. The honest version is: this is a large, reproducible, model-independent effect on a quantity that remains bad.
what this doesn't prove
Two dimensions, linear elasticity, one resolution, one contrast ratio. Budgets of at most four families out of twenty-two, so I cannot speak to what happens at ten. The four-family results are 200 sampled subsets rather than all 7,315. Two model classes, which is one more than usual and one fewer than I would like. Worst case over held-out families is one objective; the average is in the released data and I did not analyse it.
The pilot-based criteria all use twenty solves per family. Nobody knows what forty would do, including me.
The solver underneath all of this is checked against cases with known exact answers: a uniform sheet returns its input to about sixteen decimal places, a layered material matches closed-form theory to fourteen, and refining the mesh drops the error from 5.91 percent to 0.59 percent. The network's gradients are verified against finite differences to nine decimal places. The labels are right, whatever else is wrong.
the part that isn't about metal
Strip the engineering out and what is left is a question about where the leverage in a machine-learning system actually sits.
The composition of the training set moved my worst case by 390x. It is chosen, in most projects, by whatever was available, or by whoever happened to label something first, and then it is never revisited. The model architecture, which moved things far less, gets the meetings.
The uncomfortable half is that I cannot hand you a rule. I can tell you the lever exists, that it is bigger than the ones you are pulling, and that all nine of the obvious ways to find the handle failed under one change of model or one change of target.
Everything is open. The benchmark is called MetaXfam22: the solver, all 22 families at 600 cells each, and every per-subset score, so a better criterion can be scored against the full answer key without rerunning anything: https://doi.org/10.5281/zenodo.21601041