Before using an AI forecast backtest to plan staffing, production or support capacity, ask the team to reproduce the inputs available at each historical decision time. A table containing only past observation dates is insufficient. A past value might have been corrected later; a future date might legitimately contain a schedule published in advance.
The practical question is whether the test recreates the information the business could have used. That comes before deciding whether its error score justifies a pilot. This guide develops an input reconstruction example, rather than assessing a particular forecasting product.
The distinction matters when evaluating models with extra predictors. In its 15 October 2026 Timestral announcement, Mistral describes support for both past-only and known-future covariates: additional variables supplied alongside the series being forecast. Its examples include production plans and weather forecasts. That is a vendor description of model inputs, not evidence that a customer’s historical test supplies those inputs correctly. We have not independently tested Timestral.
A past observation can contain later knowledge
Suppose a service team predicts Tuesday’s workload every Monday morning. Its spreadsheet includes Sunday’s backlog. In the current export, a row labeled Sunday looks safely historical. But the value might include a Wednesday correction to incorrectly closed cases.
Replaying Monday with that corrected value gives the forecasting process information it did not have then. Moving the test window forward through calendar time does not, by itself, remove this problem.
The underlying research predates today’s AI. Croushore and Stark’s 2001 Journal of Econometrics paper constructed historical data snapshots, called vintages, from information available at particular dates. Its forecasting exercise showed that revised and contemporaneously available data could produce different forecasts in selected periods. However, its long-period forecast-error statistics were not sensitive to that distinction. This is a macroeconomic data study, not a measurement of AI forecasting or a prediction of how large your team’s error will be.
Our recommendation is therefore to inspect input history, not assume that every revision makes reported accuracy better. Depending on the data and model, results may improve, worsen or change very little.
Reconstruct one Monday before evaluating a year
Consider this invented example. The decision time is 5 October 2026 at 09:00 UTC, and the forecast concerns workload on 6 October. All values below are fictional. The rain signal is a simple zero-or-one illustration, not a probability or a calibrated weather model.
| Input and period it describes | Value | Available to the forecasting process, UTC | Eligible at the decision time? |
|---|---|---|---|
| Backlog, 4 October, initial snapshot | 24 cases | 5 October, 08:00 | Yes |
| Backlog, 4 October, corrected snapshot | 29 cases | 7 October, 12:00 | No: later revision |
| Rain signal for 6 October, issued forecast | 0 | 5 October, 06:00 | Yes |
| Rain signal for 6 October, observed weather | 1 | 7 October, 00:00 | No: later observation |
| Closure planned for 6 October | 1 | 20 September, 10:00 | Yes |
The eligible inputs are 24, 0 and 1. A latest-value export that merges forecasts and observations into the same rain-signal column could instead supply 29, 1 and 1. The rows would still describe the right business periods. Their information history would be wrong for Monday’s decision.
We executed a small Python record selector against these fictional records. It retained the latest eligible version for each input and period, rejected unknown availability, and produced the values above. Reversing record order did not change the result. Moving the cutoff to just before 08:00 made the backlog snapshot unavailable. This checks record selection only: no forecasts were generated and no AI model or production dataset was tested.
The proposed selection rule is:
- Specify the decision timestamp and the required input periods.
- Exclude versions that became available after that timestamp.
- For each required input and period, choose the most recent eligible version, retaining its type and provenance.
- Flag missing or ambiguous versions instead of quietly taking today’s value.
For a real implementation, define what happens when timestamps tie, records are withdrawn or definitions change. A correction should create a new version rather than erase the version that an earlier decision used.
Known future means available ahead of time
The textbook Forecasting: Principles and Practice distinguishes forecasts using information available in advance from calculations using predictor values observed later. The latter can help diagnose a model, but do not represent a forecast that could actually have been issued. It also explains why calendar variables can be known ahead of time. See its discussion of ex-ante and ex-post forecasts.
Apply that distinction separately to each predictor. A public holiday can be known before the forecast period. An announced shutdown may also be available, but its schedule can change. The correct replay needs the plan that existed then. An archived weather forecast is different from the weather eventually observed, even if both have the same target date.
Ask for these fields in the input history:
| Field | What the reviewer needs to establish |
|---|---|
| Input identity and business period | Which quantity, location and period this version describes |
| Version and value | Which snapshot was supplied, including units and definition |
| Availability timestamp | When the forecasting process could actually retrieve this version |
| Input type and source | Observation, issued forecast or schedule; its origin and retained artifact |
Publication time alone may be too early if your pipeline retrieves a source hours later. Conversely, a database row’s latest modification time may overwrite the history you need. Choose an availability definition that matches the system being evaluated and document the ingestion delay. Use explicit timezones throughout.
Give the baseline the same historical information
Once one reconstructed decision works, repeat it over multiple forecast origins: the times at which forecasts would have been issued. Match the horizon to the actual decision. A next-day staffing test and a four-week procurement test answer different questions.
Hyndman’s 2016 practitioner explanation of time-series cross-validation describes evaluation that moves the forecast origin through time and can assess multiple steps ahead. It also distinguishes errors on fitted historical data from errors on forecasts of held-out observations. That rolling evaluation approach is useful here, but reconstructing historical predictor versions is an additional requirement.
For each origin, our proposed comparison uses the same reconstructed inputs for the AI system and the relevant simple baseline. If one method receives an archived weather forecast while the other receives actual future weather, the comparison includes an information advantage. Keep transformations and any local model fitting within the information available at that origin as well.
This does not certify a foundation model’s pretraining history. Historical input reconstruction addresses the records your evaluation supplies. Whether a pretrained model has encountered benchmark outcomes is a separate question that this example cannot resolve.
Retain the forecast, input versions, horizon and model configuration together. Define the outcome version used for scoring too: provisional and corrected workload totals may differ. Fix that scoring rule in advance and apply it consistently. Later information can establish the eventual outcome without becoming an eligible historical predictor.
For forecasts expressed as probabilities of business events, our guide to making predictions resolvable covers outcome definitions and scoring. The issue here is the earlier step: whether numerical forecast inputs reproduce the past information set.
If historical versions are missing, change the claim
A small team may have one overwritten spreadsheet; a large enterprise may have many systems with incompatible revision histories. Neither can recover an authentic historical snapshot merely by asking an assistant to infer it.
Use the latest data for a clearly labeled retrospective analysis if that is all you have. It may help explore relationships or diagnose sensitivity to predictors. Avoid presenting it as a replay of decisions the team could actually have made.
Then begin a prospective pilot that retains input snapshots, issued forecasts and subsequent outcomes. Compare methods at the real decision time before using the results to change operational capacity. A short pilot will still have limits, especially if it misses seasonal peaks or unusual disruptions; preserving its records at least makes those limits inspectable.
For the next review, request one concrete artifact: the complete input bundle for one historical forecast origin, with evidence of when each version became available. If the team cannot produce it, that is the next evaluation task. A more impressive model score does not reconstruct missing history.