Putting an LLM Inside a Product That Already Works: What a Senior Team Does in the First 30 Days

The hard LLM project is not the greenfield one. Greenfield gives you a blank repository, no users to break and no bill to compare against. The hard project is the product that already works — paying customers, a data model chosen three years ago, an on-call rota, a margin — into which somebody now has to put a language model without breaking any of it.

I have run that project several times on both sides of the table, and I have watched the other version of it: the two-week hackathon that produced a chat box, a demo that impressed the board, and a feature quietly turned off four months later because nobody could say whether it was correct or what it cost. The difference between the two outcomes is decided in the first thirty days. Not by the model, and not by the framework. By the order in which a senior team does things.

This is that order, week by week. It assumes a product with real traffic and a team that has to keep shipping everything else while this happens.

Week one: pick one workflow, measure it, and build the eval set from real traffic

The first mistake is choosing «AI» as the scope. The scope is one workflow with a measurable outcome: the support reply that takes an agent eleven minutes, the invoice that a human classifies, the search that returns nothing useful for a third of queries. A workflow that already has a number attached, because the LLM's only job is to move that number.

Then, before a single prompt is written, the eval set. Not synthetic examples — fifty to two hundred real cases pulled from production logs, with the answer a competent human gave, anonymised where the data requires it. This is the artefact the whole project stands on: it is how you will know in week four whether the thing works, and it is how you will know in month six whether the model provider's silent upgrade broke it. Teams that skip this step do not skip evaluation; they defer it to their users.

Week one also ends with an integration map: where in the existing codebase the call will live, what data it will see, what it must never see, and which existing service owns the result. On a three-year-old system that map usually reveals the real first task — a piece of context that lives in two places and disagrees.

Week two: a thin slice behind a flag, with retrieval before generation

The second week produces the smallest possible end-to-end path, in the real codebase, behind a feature flag nobody outside the team can see. Input from the real system, a call to a model, output written back where the product expects it. Ugly is fine. What matters is that it runs through your CI, your auth and your logging, because that is where the integration problems live, and they are cheaper to find in week two than in week seven.

For most product workflows, the model needs your data more than it needs cleverness, which is why retrieval comes before generation. The thin slice therefore includes the retrieval path — what gets fetched, from where, with which permissions — even if the retrieval is a simple query in week two and becomes a vector index in week five. The permission part is not optional: a model that can be asked about another customer's records is a data breach with a friendly interface.

By the end of the week, the eval set runs against the slice. The score will be bad. That is the point of having the number.

Week three: evals become the gate, and cost gets a ceiling before it gets a graph

Week three is where a senior team separates itself from a hackathon, because week three is boring. The eval run moves into CI: every change to a prompt, a retrieval query or a model version runs the set and reports the score, and a drop below the threshold blocks the merge. This is the single practice that keeps an LLM feature alive past its first model upgrade. A prompt without an eval is an opinion; a prompt with an eval is a test.

The same week, cost. Before there is a graph of spend, there is a ceiling on it: a per-request token budget, a per-tenant daily cap, and a circuit breaker that degrades to the old behaviour when the cap is hit. Model prices differ by two orders of magnitude between a frontier model and a small one, and the right answer for most product workflows is the cheapest model that passes the eval — a decision you can only make if the eval exists. I have written about what happens to teams that treat tokens as a metric instead of a cost; the short version is that the bill arrives before the value does unless the ceiling is there first.

Week three also settles the failure modes, enumerated and tested: the model times out, the model returns something the parser cannot read, retrieval returns nothing, the request is a prompt injection through a customer field. Each one has a defined behaviour — fall back, retry once, refuse, log — and each one is in the eval set as a case.

Week four: five per cent of traffic, with observability, and a decision written down

The rollout is small on purpose. Five per cent of traffic, or one customer segment, or internal users only — whichever the risk profile allows — with three things watched: the eval score on live samples, the cost per request against the ceiling, and the human metric from week one. A support reply that takes eleven minutes should now take fewer; if it does not, the feature does not expand, whatever the demo looked like.

Observability for an LLM feature is not the same as for a service. You need the prompt, the retrieved context, the model version and the output stored together per request, with retention that respects your data policy, because the only way to debug «the answer was wrong» is to see exactly what the model saw. Teams that log only the output spend month two guessing.

The month ends with a written decision, one page: what was built, what the eval says, what it costs per thousand requests, what the human metric did, and one of three verdicts — expand, iterate, or stop. A stop verdict after thirty days and one workflow is a success, not a failure; it is the cheapest possible way to learn that this workflow was not the one. The failure is the feature that ships to everyone because nobody wrote the page.

The objection: thirty days is slow, and a vendor did it in two weeks

The strongest counter-argument is that all of this is slower than the demo, and the demo is what the board wants. It is true, and I will give the objection its due: if the goal is a demo, do the demo, in a week, in a sandbox, and never connect it to production. Demos are cheap and useful and I build them.

The thirty-day plan is not for the demo. It is for the feature that will still be running in a year, billing tokens on real traffic, after two model upgrades and one engineer's departure. That feature needs an eval set, a cost ceiling, permissioned retrieval and per-request observability, and none of those can be bolted on later at the price they cost in weeks one to three. The two-week version is not faster; it moves the thirty days to after launch, when they are paid for by users.

What this needs from the team

None of the above requires an AI research team. It requires senior engineers who know your codebase's seams, one of whom has shipped an LLM feature to production before and carries the scars — the injection through the address field, the retrieval that leaked across tenants, the upgrade that changed the answer format on a Tuesday. That combination is rarer than either half alone, which is why the engagement we run most often is one or two senior engineers embedded in the client's team for the thirty days, not a separate AI project delivered from outside.

If you have a product that works and a workflow with a number on it, the first conversation is thirty minutes, and it starts with the eval set.