A model has to be scored on data it was not trained on, or the score only measures memory. Holding back one test set works, but with little data the score then depends heavily on which rows landed in it. Cross-validation fixes that by testing on every row exactly once.
It also has a failure that does not show in the number. I ran into it in my research, which I wrote about in a note on a model that scored perfectly. This note works both parts through by hand.
How k-fold cross-validation works
Split the data into k equal parts, called folds. Train on k − 1 of them and test on the one left out. Repeat k times, so each fold is the test set once, and combine the k test results into one score.
Every row is used for testing once and for training k − 1 times. Five and ten are the usual choices for k. With k equal to the number of rows it is called leave-one-out.
Six points, three folds
The true relationship is y = x², and there are six measurements in two groups. Group A: x = 0, 0.5, 1. Group B: x = 2, 2.5, 3. The model is a straight line fitted by least squares, and the score is R², where 1 is perfect and 0 means no better than always answering with the average.
Ordinary 3-fold cross-validation, with the rows dealt into folds without regard to group: fold 1 holds x = 0 and 2.5, fold 2 holds 0.5 and 3, fold 3 holds 1 and 2. Every fold mixes one point from each group.
| Test fold | Line fitted on the other four | Predictions (true) |
|---|---|---|
| x = 0, 2.5 | y = 3.517x − 2.153 | −2.15 (0), 6.64 (6.25) |
| x = 0.5, 3 | y = 2.483x − 0.602 | 0.64 (0.25), 6.85 (9) |
| x = 1, 2 | y = 3.000x − 0.625 | 2.38 (1), 5.38 (4) |
Pooling all six test predictions, the mean squared error is 2.23 and R² is 0.80. (How those two scores relate, and when to use RMSE or MAE instead, is in a note on error metrics.) A straight line through a curve, and cross-validation calls it a decent model.
The same model, split by group
Now split by group instead: train on group A and test on group B, then the reverse. This is group k-fold, and with one group per fold it is called leave-one-group-out.
Trained on group B, the line is y = 5x − 6.083. On group A it predicts −6.08, −3.58 and −1.08, against true values of 0, 0.25 and 1. Trained on group A, the line is y = x − 1/12. On group B it predicts 1.92, 2.42 and 2.92, against 4, 6.25 and 9.
The mean squared error is now 18.68 and R² is −0.67: worse than ignoring x altogether. Same six points, same model. The only change is which points were allowed to sit together in a fold.
Why the random folds flattered it
In the random folds, every test point had neighbours from its own group in the training set. The model never had to reach anything new. It only had to fill gaps between points it had already seen, and a straight line can fill small gaps in a curve quite well.
That is the general rule. Cross-validation measures performance on data like the test folds. If the folds mix groups that will never be mixed in real use, the score answers a question nobody is asking.
What this looked like in my research
I trained models to predict the stiffness of metal plates with patterns of holes. A network trained on round holes scored essentially perfectly on held-out round-hole cells. Trained on one family and tested on another, 20 of the 30 combinations scored below zero.
Splitting by row would have kept the near-perfect score. Splitting by family, the equivalent of the group split above, showed the model had learned the families it saw rather than how stiffness works. The worked version of that collapse is in a note on out-of-distribution failure.
Choosing the split
Ask what a new case will look like in use, and make the folds match. Several measurements from the same patient, machine, customer or design family belong in the same fold, which is what GroupKFold and LeaveOneGroupOut do in scikit-learn. Data in time order needs folds that train on the past and test on the future, as TimeSeriesSplit does, because a random split lets the model see tomorrow.
When the grouping is unclear, run both splits. A large gap between them is not noise. It is the size of what the model has memorised.