All notes

What changes when a pilot goes live

A successful pilot can leave out the hardest parts of daily work. Check how the system handles incomplete records, exceptions and limited review time before it goes live.

6 min

A pilot usually proves that a model can do a task on the data you chose. A rollout asks it to do that task on the data that arrives when nobody senior is watching. Those are different tests. The first is allowed to skip the ugly cases. The second is made of them.

When a rollout stalls, the model is often blamed. In the engagements we see, the model is rarely the part that changed between the demo and the queue. The surrounding system changed. Here are the four gaps we now look for in the first conversation.

The sample was kind

Someone exported the rows that were complete. Production data has missing fields, last year’s template, scanned pages, and a free-text box that customers use as a complaint line. A model tuned on the kind sample will look careless on the real one, because nobody showed it the mess.

The fix is unglamorous. Take twenty items from last month that a supervisor still remembers, including the ones the team groaned at. If the system cannot be shown those, it is not ready to be judged.

The loop was a person with time

During the pilot, a senior person checked every answer. They enjoyed it. It was their idea. On Monday that person has their own job, and the check becomes a glance, then a skip. If the design assumes their attention, the design is the pilot.

Write down the check as a step with an owner and a duration. If the duration is not available, the product has to change: fewer items routed to a human, or a narrower job for the model, or a decision that this queue is not the one.

The exception list was never written

Teams apologise for “edge cases” that they already handle by habit. Those habits are the specification. A new colleague learns them by sitting nearby for a month. A model does not. If the habits stay unspoken, every ordinary exception looks like a model failure, and the staff lose trust for a reason.

A rollout that starts everywhere starts nowhere.

Nobody owns the week after

The vendor leaves. The champion is proud. The queue supervisor was not in the room when “done” was declared. The supervisor decides, quietly, to go back to the old way, because the old way has a person to shout at. That decision rarely appears in a status report. It appears in a queue that is full again.

Name the owner before go-live. Give them the right to pull the system. A system that cannot be pulled will be worked around, which is a pull with extra steps.

What we do instead

Pick one queue. Write twenty real exceptions. Agree what a good answer is, in the words the supervisor already uses. Ship to that queue only. Read every failure for two weeks. Then, and only then, talk about the next team.

The company rarely needs a smaller model at this point. It needs a narrower door. Width can come later, bought with the evidence from the first door.

If this is the situation you are in

Tell us which systems you use, which step takes the most time and what data you can share for testing.

Discuss a project