There was a random number hiding inside my measurement

This is a short note about a result I reported, believed, and had to withdraw, and about why the safeguard I had in place did not fire. The engineering is a detail. The failure mode is not.

The quantity in question describes how a small patterned tile of metal moves when it vibrates: what proportion of the motion is rotating, what proportion is shearing, what proportion is stretching. Three numbers. They are useful because a machine-learning model given them transfers to shapes it has never seen, where the same model given a picture of the tile does not.

Computing them needs an eigenvalue solve, which costs about as much as the thing it is supposed to be a cheap substitute for. The obvious move is to shrink the tile first, solve on a coarser grid, and see how much of the signal survives.

The negative result

I ran it at four resolutions and watched the value fall away.

At full resolution the three numbers were worth +0.388 on the transfer task. Coarsened to a third of that resolution, which is a tenth of the work, they were worth +0.051. That is below the free control of simply telling the model how much metal is in the tile. Useful at full price, worthless at a discount.

That is a perfectly publishable shape of finding. It is monotone, it has a plausible mechanism, and it is the kind of limitation you are supposed to report rather than bury. I wrote it up as one.

where the random number came from

Most of the tile families in this benchmark have four-fold symmetry. A square has it, a circular hole in a square frame has it, a cross has it. When a shape has that symmetry, its second and third vibration modes come out at the same frequency.

Two modes at one frequency is not a coincidence to be cleaned up. It means the solver has been asked a question with no single answer. Any mixture of the two is equally a valid mode, and which particular mixture comes back depends on where the iterative solver started, which was a random vector.

My three numbers were computed from one of those modes, which is to say from a single eigenvector. They are quadratic in it, so they are not invariant to the mixture: rotate the pair against each other and the numbers change. I had built a descriptor whose value was part physics and part the seed of a random number generator, and I had not noticed, because at full resolution the physics part is much the larger of the two.

Coarsening is what exposed it. A coarser grid pushes the two frequencies closer together, the mixture becomes less determined, and the random component grows. So the descriptor degraded smoothly as I made it cheaper.

That is the part worth sitting with. The artefact did not look like an artefact. It looked like physics. A quantity that decays as you spend less on it is the most ordinary result in numerical work, and it is the one shape of curve that would never have made me suspicious.

the check that passed

I was not careless about the coarsening. I had a fidelity check for exactly this: compute the frequencies on the coarse grid and on the fine one, and correlate them across the whole pool of tiles. If the coarse grid still ranks tiles the way the fine grid does, the coarsening is faithful.

It scored 0.87 to 0.93. It gave no warning at any resolution.

And it was telling the truth. The coarsening was faithful in the sense the check measured. The ranking was preserved. What had been destroyed was the ability to tell apart tiles that sit close together in that ranking, and a correlation cannot see that, because a correlation is a statement about order and the loss was inside the order.

My check verified a property. It was not the property I depended on.

what it looks like fixed

The repair is not subtle once the diagnosis is right. Do not compute the three numbers from one mode. Compute them from every mode at that frequency and average, which is an operation that does not care how the solver split the pair, then start the solver from a fixed vector so the run is reproducible. Separately, coarsen by averaging the metal in each block rather than by thresholding it, because thresholding erases thin struts and thin struts are most of what these tiles are.

The invariance is now checked directly: rotate the mixture at random inside the degenerate pair and the answer moves in the fourteenth decimal place.

Under the old definition the three numbers fell from +0.388 at full resolution to +0.051 at a third of it, below the free control. Under the corrected definition, taken with that free control alongside them, the curve does not fall off at all:

The conclusion inverts. The finding is not that the physics is unaffordable. It is that the physics is cheap, and I had been measuring my own solver's starting vector.

The superseded version of the extractor ships in the deposit alongside the corrected one, so anyone can rerun the comparison and watch the bad curve appear.

the general version

Two things I would take out of this into work that has nothing to do with metal.

The first is about ties. If a quantity you compute is defined only up to a choice, and some library makes that choice arbitrarily on your behalf, then part of what you have measured is the library. Degenerate eigenvalues are one instance. So is an argmax over equal scores, a sort that is not stable, a cluster label, an unseeded initialisation. The tell is that the ambiguity is usually invisible while the signal is strong, and only surfaces as noise when you make the problem harder, which is the same moment you are most willing to believe you have found a real limit.

The second is about checks. A validity check is a claim about one property, and it is easy to pick one that is cheap to compute rather than the one your conclusion rests on. Mine compared rankings when what I needed was discrimination. It passed honestly and told me nothing, which is worse than having no check at all, because a check that passes buys you the confidence to stop looking.

The three numbers, the corrected extractor, the superseded one, and every result file are here — now at version 2, recomputed on duplicate-free data: https://doi.org/10.5281/zenodo.22137052