Launch when AI mostly works, then improve it in production

Our approach is to get a useful AI system into real work as soon as it mostly works, with people reviewing consequential results, then improve it rapidly from production feedback. Waiting for near-perfect development performance delays the learning that makes the system reliable.

This is part of the new playbook. Software is getting easier to build and change, especially when an LLM is the reasoning engine. The central challenge is agreeing on what the AI should do and making sure it does it correctly. Production is part of solving that challenge.

What we mean by 80% and 95%

We describe the approach as launching at roughly 80% accuracy and moving toward 95% during the first month live. These numbers are a mental model: the system mostly works, still has problems, and improves quickly through real use. They are not universal release thresholds or promised results.

The important question is how the team learns and improves. A debate about the exact percentage misses the point. Trying to reach 95% entirely in development can consume time without exposing the problems that actual employees, documents and operating situations will reveal immediately.

Production reveals what development cannot

LLM behavior is non-deterministic. Development examples cannot anticipate every way people will ask questions, every combination of source information or every judgment the business will need to make. Production experience is never exactly what the team expected.

A human reviewing a live result may immediately recognize something seriously wrong that nobody anticipated in development. That observation is valuable: it gives the team a real failure to investigate and a concrete behavior to improve.

Keeping the system in development until it appears nearly perfect does not remove those unknowns. It postpones finding them.

Launch with people in the loop

Start with a defined scope and an initial group. The workflow should already perform useful work, and humans review critical decisions before those results reach customers or operations. Mostly works does not mean leaving consequential errors unchecked.

That review does two jobs: it protects the work being done now and supplies feedback to improve the system. Capture the input, the AI's result, the expert's correction and why the correction matters. Use those examples to clarify expected behavior and guide engineering.

As live examples accumulate, build and improve LLM monitoring and auditing around the experts' judgment. The team can find recurring problems and inspect more work without relying only on user complaints.

Improve in days and weeks

The team has many ways to address a production failure: update prompts, improve search and retrieval, add a tool, make another LLM call, improve document processing or clarify the operating guidance. The fix does not necessarily require months of conventional software development.

In our approach, the first month live is an intensive period of learning and improvement. The cycle runs in days and weeks: observe real work, understand the failure, make a change, evaluate it and put the improvement into use. New failures become evaluation cases so later changes preserve what the team has learned.

That is why we want production feedback early. It directs engineering toward the problems that actually matter, using a system that is already contributing to the business.

Reliability grows through operation

Building reliability from the start includes preparing to learn from production. Development establishes the initial workflow, expert agreement and evaluations. Operation exposes new situations and turns them into better guidance, tools and behavior.

The management question is whether the team has that learning loop and can act on it quickly. The percentages help explain the direction; the operating method is what makes the progress possible.

Performance-Driven Development explains the engineering and production-review loop. Why AI needs a new playbook explains the shift in risk, and working with Prolego describes the engagement.