Most retail items do not sell every day. For those items the usual advice is to stop using ordinary forecasting methods and use one built for intermittent demand, starting with Croston's method from 1972. I tested that advice on real data, with the models as they ship in statsforecast, and scored them with forelint, the open-source library I wrote for auditing forecast backtests.
The earlier note on value added by demand class did the same for a machine-learning model. This one asks the classical question: which of the standard methods should a planner reach for.
The setup
The data is the public M5 set of daily Walmart unit sales: store CA_1, department FOODS_3, 823 items, the last 730 days. Each forecast covers 28 days, and StatsForecast.cross_validation repeats it at ten forecast origins 28 days apart, so the forecast windows do not overlap and every comparison rests on ten separate results.
Eight models ran: AutoETS with a weekly season, the seasonal naive, Croston's method (optimized), ADIDA, IMAPA, TSB, the naive forecast, and a 28-day moving average, which is the baseline. Each item is classed by the Syntetos-Boylan rule on its history before the first origin: smooth, erratic, intermittent or lumpy. The script is in the forelint repository.
Overall
On absolute error across all items:
| Model | MAE | Value added vs moving average | Items where it is worse | Bias |
|---|---|---|---|---|
| AutoETS | 1.649 | +1.6% | 45% | −0.2% |
| ADIDA | 1.676 | 0.0% | 56% | +4.3% |
| 28-day moving average | 1.676 | baseline | baseline | −0.7% |
| IMAPA | 1.681 | −0.3% | 58% | +4.9% |
| TSB | 1.702 | −1.5% | 64% | +6.9% |
| Seasonal naive | 1.927 | −14.9% | 86% | +0.7% |
| Croston (optimized) | 1.960 | −16.9% | 84% | +17.2% |
| Naive | 2.398 | −43.0% | 86% | +35.0% |
Only AutoETS beat the moving average with a margin a test can see: 1.6 percent, with a Diebold-Mariano p = 0.02 across the ten origins. ADIDA tied it exactly (p = 0.97). TSB was worse (p = 0.03), and Croston's method was worse by 17 percent (p < 0.001). A model confidence set at the 10 percent level kept AutoETS alone; the moving average, ADIDA and IMAPA fell just outside it, at p = 0.09.
By demand class
Value added over the moving average, on absolute error, per class:
| Demand class | Items | AutoETS | ADIDA | TSB | Croston |
|---|---|---|---|---|---|
| Smooth | 145 | +5.2% | −1.3% | −3.3% | −6.2% |
| Erratic | 36 | +1.1% | −0.4% | −0.2% | −6.1% |
| Intermittent | 463 | 0.0% | +0.6% | −0.9% | −24.6% |
| Lumpy | 175 | −1.1% | +1.5% | 0.0% | −24.8% |
On the 463 intermittent items, the class Croston's method exists for, it was 24.6 percent worse than the moving average, and worse on 83 percent of them. On the lumpy items it was 24.8 percent worse. ADIDA, also built for intermittent demand, did slightly better than the moving average on both classes. AutoETS gained 5.2 percent on the smooth items and nothing on the intermittent ones.
Why Croston lost here
Its forecasts ran high. Across the store they were 17.2 percent above actual sales, against under 1 percent for the moving average and AutoETS, and the bias was significant on 8 percent of items. Croston's method is known to over-forecast, which is why Syntetos and Boylan proposed a correction for it in 2001. The size of that bias on real data is the finding, not its direction.
One caution cuts the other way. On items that sell on few days, absolute error rewards a forecast that leans toward zero, so MAE can be unfair to a method that forecasts the average rate. The squared-error score says the same thing here, though: Croston's RMSSE was 0.857 against 0.810 for the moving average. It lost on both measures.
What I would take from it
- A method named for a demand class is a candidate, not a result. Test it against a moving average on that class.
- Check bias next to the error score. Croston's problem here showed in the bias column first.
- Test differences across origins whose windows do not overlap. Without that, a 1.6 percent gain and a tie look the same.
- This is one store and one department, scored on sales rather than demand. Stockout days count as low demand, as in the censored demand note. Run it on your own data before you trust it.
forelint reads the cross_validation output as statsforecast returns it, so the whole audit above is one call after the backtest. It installs with pip install forelint.
References
- Croston, J. D. Forecasting and stock control for intermittent demands. Operational Research Quarterly 23(3), 1972.
- Syntetos, A. A. and Boylan, J. E. On the bias of intermittent demand estimates. International Journal of Production Economics 71(1-3), 2001.
- Teunter, R. H., Syntetos, A. A. and Babai, M. Z. Intermittent demand: Linking forecasting to inventory obsolescence. European Journal of Operational Research 214(3), 2011.
- Nikolopoulos, K., Syntetos, A. A., Boylan, J. E., Petropoulos, F. and Assimakopoulos, V. An aggregate-disaggregation intermittent demand approach (ADIDA) to forecasting. Journal of the Operational Research Society 62(3), 2011.