I trained a neural network to predict how stiff a metal sheet is when you punch holes in it. It scored perfect on the held out test set. Not 98 percent. Perfect.
Then I changed the shape of the holes and it became worse than useless. Not less accurate. Worse than deleting the model and always answering with the average of everything it was ever trained on.
That happened on 20 out of 30 tests.
If you have a model in production with a good validation number on a slide somewhere, this post is about what that number is actually measuring.
The setup
You don’t need materials science for this part.
Take a metal sheet and punch a repeating pattern of holes in it. It gets springier. How much springier depends completely on the pattern, not on the metal. This is how you make something lighter without paying for a better alloy, and it’s how you get materials that behave in ways that look wrong.
Working out the stiffness of a given pattern means running a physics simulation, and simulations are slow. So the standard move is to train a neural network on a few hundred simulated patterns and ask the network instead. Show it a picture of the pattern, get a number back, instantly.
This is a large field with a lot of papers and they report excellent accuracy. I got excellent accuracy too.
The question
Almost every one of those results is made the same way. Generate a thousand patterns of one type, say circular holes at different sizes and spacings. Train on 800. Test on the 200 you held back. Report the score.
But those 200 are circular holes too. Same generator. They just weren’t used in training.
So the test answers one question, can the model handle another example of the thing it already saw. Everyone reads it as a different question, did the model learn how stiffness works.
Those aren’t the same, and in real use you need the second one. Nobody wants a stiffness predictor for circular holes. They want one for whatever gets designed next month.
So I built the test that separates them. Six completely different families of pattern: round holes, square holes, stretched ovals, cross shaped cutouts, honeycombs, random strut lattices. 250 of each. Train on one family, test on all six.
The part I nearly got wrong
My first version of this experiment was worthless and it took me two days to notice.
Stiffness is mostly about one boring thing, how much metal is left. A sheet that’s 40 percent holes is stiffer than one that’s 70 percent holes, and that effect is enormous. It drowns everything subtler.
My six families, generated the obvious way, sat at different densities. So when a model trained on circles fell apart on honeycombs, I had learned nothing. The honeycombs were simply denser than anything it had seen. That’s not the model failing to understand shape. That’s asking someone who only ever saw prices between 10 and 20 to price something at 50.
The fix is tedious and it’s the reason any of this counts. I generated patterns and threw away every single one whose density fell outside a narrow band, over and over, until all six families sat in the same range. A huge fraction of the generated data went in the bin. Every family now averages between 59 and 65 percent solid.
With density pinned, anything that still fails, fails because of shape.
That control is the actual contribution here. The result is just what fell out of it.
What happened
The network learned round holes perfectly. Show it a round hole pattern it had never seen and it was essentially exactly right.
Show it a cross shaped cutout and the predictions were about as good as guessing. Worse than guessing, measurably. Worse than ignoring the picture completely and always answering with the average.
Across all 30 train on one, test on another combinations, 20 came out below that line.
Averaged over unseen shapes, the neural network did worse than a crude rule that looks at nothing except how much metal is left. That crude rule loses badly on the network’s home turf and wins overall, because it never learned anything it could be wrong about.
I ran the same thing with a random forest, a completely different kind of model. Same story, slightly less severe. So it isn’t an artifact of one architecture.
the failure has a shape
If the answer were just “these models don’t generalise” it would be a shrug. What makes it worth your time is that the collapse follows a pattern you can nearly see.
Train on ovals, test on circles: nearly perfect. Ovals and circles are the same idea with one parameter changed and the model is fine with it. Ovals to squares: still good.
Circles to crosses: catastrophe. Squares to crosses: catastrophe. Anything to crosses: catastrophe.
Perforated plates transfer to other perforated plates. They do not transfer to shapes built from struts and thin arms. The model didn’t learn elasticity. It learned a family of hole shaped things, and a cross isn’t in that family.
Which changes the practical question. It isn’t “will my model generalise,” that has no useful answer. It’s how far the new case sits from what you trained on, and what far even means here. I can see the pattern in my results and I can’t measure it. Nobody else seems to have a way to measure it either, as far as I can find. That gap is more interesting to me than the failure.
why you should believe any of this
Fair objection at this point. I made my own training data with my own simulator. If the simulator is wrong then everything above is noise with a story attached.
So I checked it against cases where theory already knows the exact answer. Feed it a plain sheet with no holes and it should hand back exactly the stiffness you put in. It agrees to about 15 decimal places. Feed it a layered material where classical theory gives a closed form answer and it matches to about the same precision. Refine the mesh and the error drops by a factor of ten, which is what it has to do.
Then there’s the test I actually enjoyed.
There’s a family of structures called auxetics that break the rule everyone knows about materials. Stretch a rubber band and it gets thinner. Auxetics get fatter when you stretch them. It looks like a mistake. It’s real, it’s used in body armour and stents, and it comes purely from geometry. Take a honeycomb and fold the cell walls inward so they point back on themselves, and the impossible behaviour appears.
My simulator has no concept of auxetics anywhere in it. Nowhere in the code is that idea represented. But set the wall angle positive and it reports normal behaviour. Set it negative and the sign flips, and the material starts getting wider when you pull it, close to what theory predicts for that geometry.
The physics was in the shape the whole time. The program just did arithmetic and the strangeness fell out.
That’s what a validated instrument looks like, and it’s why I trust the failure results. The labels those models trained on are right.
One more thing that should change how you read the network result. The machine I was working on had no internet access, so no PyTorch and no way to install it. I wrote the network from scratch in NumPy, convolutions and training loop and all the gradient math, then checked the calculus against brute force before trusting a single output. It’s a small honest network, not a tuned one. A better model might transfer further than mine did. That’s a real limitation and I’m not being modest about it.
what this doesn’t prove
250 examples per family. Coarse 32 by 32 images. One stiffness number. Two dimensions, simple elasticity. One network design, one training run each. The random strut family is genuinely hard even at home, so treat its numbers gently. A larger study could move these magnitudes.
I looked hard for someone who had already run this comparison and found nobody. Not finding something isn’t proof it doesn’t exist and I’m not claiming to be first at anything.
the part that isn’t about metal
Take the engineering out and this is a statement about how nearly all machine learning gets validated.
A test set carved out of the same pipeline that made your training set measures one thing: can the model do again what it already saw. That’s worth knowing. It is not what most people think they’re being shown when that number goes on a slide, and more decimal places won’t convert one into the other.
The gap only appears if someone deliberately builds a test from a different process and controls the obvious confounds. Almost nobody does that. It’s expensive, it’s boring, and it makes your results look worse.
My model scored perfect. What it was really telling me was how good my pattern generator is at making circles.
If you have a model in production, the question isn’t what it scored. It’s what changed between the data it was tested on and the data it sees now, and whether anyone checked.
Everything is open, simulator, validation tests, dataset, the from scratch network: https://doi.org/10.6084/m9.figshare.32993597
Also published on Medium · Substack. This page at davidmashiah.com is the canonical version.