Forecasting, and Why Better Models Rarely Fix It

Supply chain describes a set of activities with almost nothing in common except that goods move through them. Separating them determines where tooling helps.

Planning. Forecasting demand, setting inventory targets, deciding what to make or buy and when. Quantitative, model-heavy, and the subject of this lesson.

Sourcing. Finding suppliers, negotiating, qualifying, managing supplier relationships and risk. Relational and contractual.

Execution. Purchase orders, production scheduling, warehousing, transport booking. High volume, transactional, heavily systematised already.

Documentation and compliance. Commercial invoices, packing lists, bills of lading, certificates of origin, customs declarations. Enormous document volume with legal consequences, and the subject of lesson two.

Exception handling. The shipment that is late, the container held at customs, the supplier who cannot deliver, the quality failure. This is where planners actually spend their days, and it is lesson three.

The honest summary of where AI lands. Planning has used statistical models for decades and gains less from machine learning than vendors suggest, for reasons this lesson explains. Documentation gains enormously, because it is structured extraction from unstructured documents. Exception handling gains from triage and drafting while the decisions stay human.

So the biggest wins in this field are not in the part everyone talks about.

Before improving a forecast it is worth knowing what limits it, because the limit is usually not the model.

Forecast error has three distinct sources and they respond to completely different interventions.

Irreducible uncertainty. Some demand genuinely cannot be predicted. A customer's decision, a competitor's promotion, weather, a viral moment. No model removes this, and the correct response is not a better forecast but a supply chain that tolerates being wrong: buffer stock, flexible capacity, shorter lead times.

Model error. The forecasting method fails to capture structure that is genuinely present: seasonality, trend, promotional lift, cannibalisation between products. This is the part machine learning can actually improve, and it is usually the smallest of the three.

Process error. The forecast is adjusted by people and processes in ways that make it worse. Sales inflates numbers to secure supply. Finance adjusts toward the budget. Someone overrides the statistical output based on a conversation with one customer.

That third category is routinely the largest, and it is invisible in any discussion framed as which algorithm should we use.

So the diagnostic question before adopting anything is which of these dominates your error. If it is irreducible, buy flexibility rather than accuracy. If it is process, fix the process. Only if it is model error does a better model help, and you can measure which you have, as the next step shows.

There is a simple diagnostic that most organisations have never run, and it frequently produces an uncomfortable answer.

Forecast value added, popularised by Michael Gilliland in his work on the business forecasting process, compares each step of a forecasting process against a naive benchmark. The usual benchmark is the random walk: next period equals this period. It requires no model, no data science and no meetings.

You then measure the accuracy of the statistical forecast against that naive baseline, and the accuracy of the final forecast, after human adjustment, against the statistical one.

What organisations repeatedly discover when they do this is that some steps subtract value. The elaborate statistical model beats the naive forecast, sometimes only slightly. And then the human override makes it worse, consuming meeting time to reduce accuracy.

The reason that finding matters here is direct. If your process already adds negative value at the adjustment stage, replacing the statistical engine with a machine learning model does not fix anything. You have improved an input that a later step degrades.

And it gives the correct order of operations. Measure against the naive benchmark first. Remove the steps that subtract value. Then, if model error is what remains, consider a better model.

That sequence is unglamorous and it is why many forecasting improvement projects deliver nothing. They begin at step three.

The most important structural fact in supply chain, and it is not a forecasting failure.

A retailer sees fairly stable consumer demand. They order from a distributor, and because they batch orders and hold safety stock, their order pattern is more variable than the demand they observed. The distributor sees that amplified pattern, applies its own batching and safety stock, and orders from the manufacturer with more variability still. By the time the signal reaches raw material suppliers, the swings are enormous.

Jay Forrester described this dynamic in Industrial Dynamics in 1961, and Lee, Padmanabhan and Whang analysed its causes in Management Science in 1997, identifying demand signal processing, order batching, price fluctuation and rationing behaviour as the drivers.

The crucial point is what causes it. Each participant is behaving rationally given what they can see. The amplification comes from the structure of the chain, not from anybody forecasting badly.

Which means a better forecasting model at any single node cannot fix it. What fixes it is changing the structure: sharing actual point-of-sale demand upstream, reducing batch sizes, stabilising prices, and shortening lead times.

This is the clearest case in the field of a problem that looks quantitative and is actually organisational.

Having lowered expectations, it is worth being precise about where modern methods do add value, because there are real cases.

Many related series. Classical methods forecast each item independently. Machine learning approaches can learn patterns across thousands of items at once, which helps most where individual histories are short: new products, long tails, and items whose behaviour resembles other items more than it resembles their own past.

External variables. Weather, competitor pricing, local events, macroeconomic indicators. Classical methods struggle to incorporate many of these; tree-based models handle them naturally. This helps most where demand genuinely responds to observable outside factors, which is truer of ice cream than of industrial fasteners.

Intermittent demand. Items that sell zero units most periods and then several at once. This is a genuinely hard case where specialised methods matter, and it describes a large share of spare parts inventories.

Hierarchical reconciliation. Making forecasts agree across levels, so item forecasts sum to category forecasts sum to total. Real, unglamorous, and valuable.

What does not help. Applying a complex model to a small number of stable, high-volume items with long clean histories, which is exactly where classical methods already perform well and where the pilot usually gets run because the data is good.

The practical filter. If you cannot describe what structure the new method captures that the old one missed, you are unlikely to see an improvement that survives contact with production.

A framing error that causes real damage: treating forecast accuracy as the goal rather than as an input to a decision.

The forecast exists to support decisions about how much to buy, make, hold and position. Those decisions have asymmetric costs, and the asymmetry is what should drive them.

Consider two errors of identical size. Forecasting too low on a fast-moving item means a stockout: lost sale, possibly a lost customer. Forecasting too high means holding stock: capital tied up, storage cost, possible markdown. For most consumer goods those costs are wildly different, and for perishable or fashion goods the asymmetry reverses depending on the item.

Which means the right forecast to act on is frequently not the most likely outcome. It is a quantile chosen from the cost of being wrong in each direction. This is the newsvendor problem, one of the oldest results in operations research, and it says the optimal stocking level depends on the ratio of understocking to overstocking cost.

Why this matters for tooling. A model optimised for average accuracy, minimising something like mean absolute error, is optimising the wrong thing. It treats an expensive stockout and a cheap overstock as equivalent errors.

So the useful question about any forecasting system is not how accurate is it. It is what does it produce, and does the downstream decision use the uncertainty at all. A system producing a single number discards exactly the information the decision needs.

A forecast with an honest distribution beats a more accurate point estimate, because the distribution is what the stocking decision consumes.

Everything so far has been about statistical and machine learning forecasting, which is a different technology from the language models the rest of this catalogue discusses. It is worth separating them, because conflating them is common and produces bad decisions.

What a language model is not good for. Producing a demand number. It has no access to your sales history, no numerical method, and it will produce a plausible figure with no basis. This is the calculator rule from elsewhere in the catalogue, and it applies with full force to anything that will drive a purchase order.

What it is genuinely good for in a planning function.

Explaining the forecast. Turning a statistical output and its drivers into a narrative a commercial team will actually engage with, which is a real bottleneck in sales and operations planning meetings.

Interrogating your own data. Turning a planner's question into a query, so that why did this item spike in March does not require a data analyst.

Capturing the reasons behind overrides. When a planner adjusts a forecast, the reason usually lives in their head or in a meeting. Capturing it in structured form is what later makes forecast value added measurable by reason code.

And summarising the qualitative signals: supplier communications, market commentary, customer notes, which currently reach planners as unread email.

The division holds throughout the field. Numerical methods produce numbers. Language models handle the language around them, which is where a surprising amount of planning time goes.

An order that follows the diagnosis rather than the vendor pitch.

First, measure forecast value added. Compare your statistical forecast to a naive benchmark, and your final forecast to the statistical one. This costs a spreadsheet and it tells you whether you have a model problem or a process problem. Almost nobody does this, and it changes what you should buy.

Second, fix what subtracts value. If human adjustment degrades accuracy on average, the intervention is process design: requiring a reason code, limiting who can override, or removing the step for item classes where it never helps.

Third, check whether your decisions use uncertainty at all. If the planning system consumes a single number, no forecasting improvement will reach the decision. Fixing that is often worth more than any model change.

Fourth, consider modern methods for the cases where they actually help: many related series, short histories, intermittent demand, real external drivers.

And separately from all of it, apply language tooling to the language work: explanation, querying, capturing override reasons, digesting supplier signals. That runs on a different track and does not wait for the forecasting question to be settled.

The general principle for the field, and the reason this lesson spent most of its time on structure rather than algorithms. Most supply chain problems presented as forecasting problems are information problems, incentive problems or structural problems. A better model applied to those is an expensive way to change nothing.