A better model should not mean starting over

The model changes. The organisation should not forget.

Most organisations treat AI strategy as a series of model decisions: which one to buy, which one to switch to, which one to trial next. Judged only on model choice, an organisation starts roughly level with any competitor who can read the same leaderboard. What actually compounds is something else: the accumulated record of what was tried against real work, what an evaluation showed against cases the organisation trusts, which corrections were needed and why, and which promising approaches turned out not to hold up. Owning that record, not just picking a model, is the strategy that keeps paying off as the underlying technology keeps changing.

Without it, every model swap starts the clock again. The evaluation work that established what “good” looks like for a workflow gets redone from scratch. Mistakes get repeated because nobody wrote down why the last approach was rejected. Decisions get re-litigated in the next planning meeting because the reasoning behind the first one lived only in the memory of whoever made it.

What the record holds

DecisionsWhat was chosen and why

Each material AI decision, which model, which architecture, which workflow to automate or leave alone, kept with the reasoning behind it and the evidence available at the time, not just the outcome.

EvaluationsWhat was measured, against what

Results against cases the organisation's own experts trust, kept alongside the criteria used and when the check was last run, so a claim about performance can be checked rather than taken on faith.

CorrectionsWhat was wrong, and what triggered the fix

Every material correction to a system's behaviour, tied to the case that exposed the problem, so a pattern of similar failures becomes visible over time instead of being fixed anew each time it resurfaces.

Rejected approaches belong in the record as much as adopted ones. An architecture tried and abandoned, a model that failed a workflow's evaluation, a workflow judged not ready for AI at all: all of it is evidence the organisation paid for, and discarding it means paying for the same lesson twice.

Kept by the organisation, not the vendor

The natural place for this record to accumulate is inside whichever tool is currently fashionable: a chat history, a vendor's dashboard, a specific model's context window. All three disappear or reset on the organisation's behalf, on the vendor's schedule rather than its own, exactly the risk a dependency register is built to manage for the models themselves. The record has to live in infrastructure the organisation controls, addressable independently of which model or which vendor's tools happen to be in use this quarter.

This is a different asset from eliciting one specialist’s tacit judgement for a single system, though the two connect: a specialist’s corrections are exactly the kind of entry this record needs to keep, attributed and dated, rather than absorbed anonymously into a model’s next fine-tuning pass.

Staying usable as the estate changes

A record nobody can find is not much better than no record. Entries are versioned, so a later decision that supersedes an earlier one is visible as a supersession rather than a silent overwrite, and linked to the specific workflow or system each entry concerns, so a question about one part of the estate surfaces the decisions and evaluations that actually bear on it rather than the whole organisation’s history. It is built to be queried by the people doing the next piece of work, not archived for an audit that may never come.

What it costs to not have this

The absence is easy to underestimate because it never shows up as a single failure. It shows up as a team re-running an evaluation whose result already existed somewhere, as a new hire proposing an architecture that was tried and abandoned eighteen months earlier for reasons nobody can now explain, and as a leadership team debating a decision the organisation has effectively already made and unmade twice. None of it looks like waste in the moment. All of it is.