The competition was ROGII Wellbore Geology Prediction, an international machine-learning competition. Each horizontal oil well comes with its first 700 feet or so labelled, and the task is to predict a depth-like quantity, true vertical thickness, along the remaining 4,900 feet. The score is the root mean squared error in feet, pooled over every row of every hidden well.
I finished 463rd of about 5,900 teams, a bronze medal. During the competition I kept a running record of every experiment, and the part worth publishing is what that record measured about the numbers I was using to make decisions. Each of these traps is general, and each has a direct counterpart in forecasting for operations.
Half the score was already answered
I built on a public notebook. It contained a routine that resolved any hidden test well whose ID also appeared in the training data, to about 0.01 ft. That is a leak in the competition data, and every fork of that notebook inherited it, mine included.
From the notebook's own validation score and its first leaderboard score, about 53.7 percent of graded rows were being answered by that leak. So the leaderboard only measured my model on the other half, and the two were related by a fixed factor: leaderboard ≈ 0.68 × validation. It also meant that a change could look small on the leaderboard and large in validation, or the other way round, depending on which wells it touched.
Validation ranked five builds backwards
Late in the competition I submitted five builds on the same day. They differed in one number: the share of the final blend given to a learned stacking model, from 0.30 to 0.50. Submitting all five on one day removed any drift between days from the comparison.
| Ridge fraction | Validation change (lower is better) | Leaderboard RMSE (lower is better) |
|---|---|---|
| 0.30 | +0.056 | 6.243 |
| 0.35 | −0.015 | 6.368 |
| 0.40 | −0.069 | 6.321 |
| 0.45 | −0.106 | 6.496 |
| 0.50 | −0.122 | 6.949 |
The correlation between validation and leaderboard over the five is −0.745. The build validation liked most, 0.50, scored worst, 6.949. The one it liked least, 0.30, scored best, 6.243. Only the middle two swap places. This was not noise around a true relationship. On this particular setting, validation was pointing the wrong way.
The reason is the kind of number being tuned. The weight decided how much to trust a model learned across all wells against a model fitted to each well on its own. Out-of-fold validation still lets the learned model benefit from wells that resemble the held-out one, so validation keeps rewarding it. My best explanation is that the hidden wells did not offer that benefit to the same degree, though I never proved it. My standing rule after this was never to let validation choose that weight.
Identical code is not identical
The model was a particle filter, run with many random seeds. I submitted the exact same code and version three times. It scored 6.241 and 6.259, and then 6.368.
After two runs the spread looked like 0.018, and for a day I judged every comparison against that. The third run showed a standard deviation of about 0.07. Two runs cannot estimate a spread. Several decisions I had made that day treated differences of 0.02 to 0.05 as real, and they were inside the noise.
With limited submissions left, it is tempting to re-run the best configuration and keep the best result, on the reasoning that one run produces the predictions for both the public and the hidden part of the test set, so a good run is good on both. That is only partly true. Picking the best of several noisy runs by its public score also picks the part of the luck that belongs to the public split, and that part does not carry over to the final ranking.
Confidently wrong
The error was not spread evenly. The worst 10 percent of wells held 60.7 percent of the total squared error, and in the worst of them about 90 percent of the error was a constant offset: the model had the right shape at the wrong depth.
One well scored an RMSE of 39.85 ft while its 128 random seeds agreed with each other to within 1.16 ft. The seeds were unanimous and wrong. I tried four separate ways to detect the bad wells in advance, including that seed disagreement, and none of them worked well enough to use. Agreement between runs of the same model is not evidence that the model is right.
Validation that leaks through neighbours
Some of my additions predicted a well from nearby wells in the same geological group. In leave-one-well-out validation, the held-out well's neighbours stay in the training data, so those additions always had good neighbours to borrow from. When I split the training wells by how many same-group neighbours they had, the neighbour-based models were 1.8 times worse on wells with none, against 1.25 times for the model that used no neighbours.
If the hidden wells came from whole groups held out together, those models would face the neighbour-poor case on every test well. That would explain why they transferred so badly. It is a hypothesis: the test that would confirm it, validating with whole groups held out, I did not finish before the deadline.
What it means outside geology
Swap wells for stores, products or delivery routes and each of these has a direct counterpart in forecasting for operations.
- Validate the way the model will be used. If new stores or products arrive with no history, hold out whole groups in validation, not single rows. Otherwise related items leak into training and validation flatters the model. The cross-validation note shows the same effect on six points.
- Measure the noise floor first. Run the identical setup at least three times before comparing anything. A difference smaller than that spread is not a result. One test split can move a score 27 times.
- Do not select on a noisy score. Choosing the best of several runs, seeds or configurations by one number chooses its luck too.
- Look at the worst entities, not the average. A tenth of the wells held most of the error. The same check, per product or per store, belongs next to every average error a forecast reports.
- Check when validation and reality disagree, and believe reality. The −0.745 was only visible because I submitted five builds on one day instead of trusting validation on all five.
This is the same question my research asks of machine-learning models in engineering: not how well a model scores where it was measured, but whether that score survives where it will be used.
I have since packaged these checks as forelint, an open-source Python library: leakage-aware splits, the noise floor, model comparison tests across many series and a per-item audit of bias and error. It installs with pip install forelint and is archived with a DOI, 10.5281/zenodo.24044927.