The order sheet is the most trusted document in a distribution business, and it is a record of what you managed to sell.
Those are not the same thing, and the difference is not small. It is also not random, which is the part that does real damage.
If you have a demand forecast fitted on your own sales history, this is about what that history is a record of.
Sales are not demand
Every reporting system I have seen counts sales. Sales is what left the warehouse. Demand is what people wanted. The two are joined by one line:
sales = min(demand, what you had)
On any day you did not run out, those are the same number and the distinction is academic. On any day you did run out, sales stops measuring your market and starts measuring your own warehouse. The figure that lands in the system is a fact about your inventory, not about anybody's appetite.
Statistics has a name for an observation truncated at a boundary: it is censored. Right-censored, specifically, because the true value is somewhere above what you recorded and you do not know how far above.
Nothing in the record marks it. The row for a day you sold out and the row for a day you happened to sell exactly that much are identical.
The error points one way
If the gap were random noise you could ignore it and let it average out. It is not random, and it does not average out.
Censoring happens only when you run out. You run out only when demand is high. So the days your data understates demand are precisely the days demand was highest.
Your dataset is missing its own upper tail, and it is missing it by construction rather than by accident.
Anything fitted to that history learns a smaller and calmer market than the real one. Not a noisier one: a smaller one. The mean is pulled down and the spread is pulled in, because the excursions were clipped before anybody wrote them down. A model can only be as ambitious as its training data, and the training data has had its best days trimmed off.
And then it feeds itself
The loop is short enough to write out.
- Forecast from the censored history. The forecast comes out low.
- Order to the forecast. You hold less.
- Holding less, you run out sooner, and on more days than last season.
- More stockout days means more censoring, so recorded sales fall again.
- Next season's forecast, fitted to that, is lower still.
Every step is defensible on its own. Each one is a reasonable decision taken on the best numbers available. The problem is that the numbers available are the output of the previous step's decision, so the whole thing is a system reading its own exhaust and treating it as weather.
There is no bug anywhere in that loop. Nothing is broken, nothing throws, no alert fires. It is a chain of correct arithmetic converging on the wrong answer, and it will do it slowly enough that no single season looks wrong.
Perishables make it structural
In a chain where nothing spoils, this is a nuisance you can buy your way out of. Understocking costs you a sale, overstocking costs you a bit of working capital and some warehouse space, and since overstocking is much the cheaper mistake you simply hold more. The buffer both protects service and, incidentally, keeps your data honest, because you rarely censor anything.
Flowers do not let you do that. The unsold unit is not carried into next week at a small cost. It is destroyed, and you paid for it.
The classical result here is the newsvendor problem, and it is worth stating because the arithmetic is unforgiving. Write Cu for the cost of being one unit short, which is the margin you did not earn, and Co for the cost of one unit left over. The stocking level that maximises expected profit is the one where the probability of meeting demand equals Cu / (Cu + Co).
For a perishable, Co is close to the entire landed cost of the unit, while Cu is only the margin on it. That puts the ratio well below one half. The mathematics is telling you, correctly, to plan on running out more often than not.
Follow that advice, which you should, and you have committed to censoring your own demand data on the majority of days. The measurement problem is not a failure of discipline. It is a side effect of the right operating decision, and it gets worse the better your stocking policy is.
That is the part I find genuinely interesting, and it is why I do not think this is solved by trying harder. Two objectives that both look like good practice are pulling against each other: run the chain efficiently, and keep a record of what the market actually wanted. You cannot have both from the same number.
What I saw, and what I am inferring
I should be exact about which parts of this I watched and which parts I am reasoning about, because they are not the same and the difference matters.
What I watched: a year of selling flowers to shops, business to business, on the order-taking side. I was not running anyone's forecasting and I did not see the inside of their planning system.
What I watched happen, repeatedly, is a shop asking for something I did not have. What they did next was take a substitute, or take less, or say never mind and call somebody else. Almost none of that reached a system anywhere. There was no field for it and no transaction to hang it on. The order that got written down was the one that could be filled, and the shop's actual request left no trace at all.
The step from there to "the forecast learns a smaller market" is inference about how those systems work rather than a measurement I made inside that company. Treat it as a mechanism to go and check in your own data, not as a finding about the flower trade.
The check is cheap. Count the days you hit zero on a line. If your forecast was fitted over a period with a meaningful number of them, and nobody accounted for that, then it was fitted to a clipped distribution and it is confidently telling you about a market that does not exist.
What would actually fix it
Two things, and only the first one is hard.
Write down the request you could not fill. Date, line, quantity asked for. That is the whole schema. It is unglamorous and there is nothing to invoice against it, which is exactly why almost nobody captures it: every other row in the system corresponds to money moving, and this one corresponds to money not moving. Somebody has to be persuaded to record a sale that did not happen.
Then stop treating those days as ordinary observations. This part is a solved problem borrowed from somewhere else. Survival analysis has handled right-censored data for decades, because a patient still alive at the end of a study is the same shape of problem as a line that sold out at four in the afternoon: you know the true value is at least what you observed, and you know nothing more. The Tobit family of models does the same job for censored regression. The methods are old, well understood and sitting in every statistics library.
So the expensive part is not the mathematics. The mathematics has been waiting since the 1950s. The expensive part is the clipboard.
The part that is not about flowers
I have written before about a model of mine that scored perfectly on its test set and had learned almost nothing. The failure there was in the evaluation: the test measured the wrong thing, and the fix was to build a harder test.
This one is worse, and no amount of evaluation catches it. Here the training data itself is a biased record, so a model can be validated impeccably, hold up beautifully on held-out days, and still be fitted to a market that was never observed. Every honest test you run will agree with it, because every test set has the same bias baked in as the training set.
There is no scoring metric that detects this. It is not visible from inside the data. The only way to see it is to know how the data was generated and to notice that the generating process had a lid on it.
So the question is not whether your forecast is accurate against your history. It probably is. The question is whether your history is a record of your market or a record of your warehouse.