David Mashiah

Note · Machine learning · Supply chain

MAPE picked the forecast that RMSE called worse

MAPE turns forecast errors into percentages, which is why managers like it. On the same five days of demand it ranks two forecasts the other way round from RMSE, and on a day with zero demand it has no answer at all. The formula, worked, and what to report instead.

MAPE, the mean absolute percentage error, is the error metric people outside the data team understand. "The forecast is off by 8 percent" needs no explanation. That readability has a price, and it shows on the same five days used in the RMSE versus MAE note.

True demand was 12, 9, 15, 10, 14. Forecast A missed by one unit every day. Forecast B was perfect for four days and missed by 5 on the last.

The formula

For each day, take the size of the miss as a share of what actually happened, then average:

MAPE = (100% / n) × Σ |ŷ − y| / |y|

Here y is the actual value and ŷ the forecast. For Forecast A the five shares are 1/12, 1/9, 1/15, 1/10 and 1/14, which average to 8.65 percent. For Forecast B only the last day counts: 5/14 = 35.7 percent, divided over five days, 7.14 percent.

MAPE prefers the forecast RMSE calls worse

MetricForecast AForecast B
Errors+1, −1, +1, −1, +10, 0, 0, 0, −5
MAE1.001.00
RMSE1.002.24
MAPE8.65%7.14%
Weighted MAPE8.33%8.33%

By MAE the two are equal. By RMSE, B is more than twice as bad, because one big miss dominates its squared error. By MAPE, B is better. The reason is the denominator. A percentage error weighs a miss by the size of the day it happened on. B made its one miss on a day of 14. A spread its misses over every day, including a day of 9, where one unit is 11.1 percent.

So MAPE does not measure units of stock. It measures misses relative to the day, and quiet days count for more. On items with a wide range between slow and busy days, that changes which forecast wins.

MAPE calculator

Paste your own actual values and predictions. It gives MAPE and weighted MAPE next to MAE and RMSE, so the disagreement is visible on your own data. It opens on Forecast B.

RMSE, MAE, MSE and MAPE calculator

Everything is computed in your browser. The defaults are Forecast B from the example above.

Where MAPE breaks

A day with zero demand divides by zero, so MAPE is undefined for any item that ever sells nothing, which in retail is most of them. Near zero it does not fail, it explodes: a forecast of 3 on a day that sold 1 is a 200 percent error, and one such day can outweigh a month of good ones.

It is also lopsided. Forecast 15 when 10 sold, and the error is 50 percent. Forecast 10 when 15 sold, the same 5-unit miss the other way, and it is 33.3 percent. An under-forecast can never miss by more than 100 percent; an over-forecast has no limit. A forecast tuned to minimise MAPE is pushed low, which in stock means running out.

What to report instead

Weighted MAPE divides the total miss by the total demand: Σ |ŷ − y| / Σ |y|. It is the MAE as a share of average demand, 1 / 12 = 8.33 percent for both forecasts. It only breaks if total demand is zero, and it weighs items by their volume, which is usually what a business means by "how far off are we".

sMAPE divides each miss by the average of forecast and actual. The 10-versus-15 miss scores 40 percent in both directions, and the score is capped at 200 percent. Here it gives 8.73 and 8.70 percent: close to a tie. It still misbehaves when both values are near zero.

MASE, the mean absolute scaled error, divides the MAE by the MAE of the naive forecast that repeats yesterday, measured on the history. The day-to-day changes here average 4.5 units, so both forecasts score 1.00 / 4.5 = 0.22. Below 1 means better than repeating yesterday. It has no division by an actual value, so zeros are no problem, and it compares fairly across items of different sizes, which is why Hyndman and Koehler recommend it.

Neither number shows direction. A forecast that is consistently high and one that misses both ways can score the same, which is what forecast bias is for.

MAPE, wMAPE, sMAPE and MASE in Python

import numpy as np

actual = np.array([12, 9, 15, 10, 14])
forecast_a = np.array([13, 8, 16, 9, 15])
forecast_b = np.array([12, 9, 15, 10, 9])


def scores(f, y):
    e = f - y
    mape = np.mean(np.abs(e) / np.abs(y))  # undefined if any actual is 0
    wmape = np.sum(np.abs(e)) / np.sum(np.abs(y))
    smape = np.mean(2 * np.abs(e) / (np.abs(y) + np.abs(f)))
    mase = np.mean(np.abs(e)) / np.mean(np.abs(np.diff(y)))  # scaled by the naive forecast
    return mape, wmape, smape, mase


for name, f in [("A", forecast_a), ("B", forecast_b)]:
    mape, wmape, smape, mase = scores(f, actual)
    print(name, f"MAPE {mape:.2%}  wMAPE {wmape:.2%}  sMAPE {smape:.2%}  MASE {mase:.2f}")

# A MAPE 8.65%  wMAPE 8.33%  sMAPE 8.73%  MASE 0.22
# B MAPE 7.14%  wMAPE 8.33%  sMAPE 8.70%  MASE 0.22

References

  1. Hyndman, R. J. and Koehler, A. B. Another look at measures of forecast accuracy. International Journal of Forecasting 22(4), 2006.
  2. Hyndman, R. J. and Athanasopoulos, G. Forecasting: Principles and Practice, 3rd edition, OTexts, 2021, section 5.8, Evaluating point forecast accuracy.