Every serious engineering team eventually asks the same question about large language models: the demo was easy, so why is production so hard? Over the past year we have shipped LLM-powered features inside loyalty and CRM platforms that serve millions of members, and the gap between a working prototype and a dependable product turned out to be where all the real engineering lives.
Start with the failure modes, not the happy path
A prototype only has to succeed once. A production system has to fail gracefully, thousands of times a day. Before we write a single prompt, we now write down what should happen when the model times out, returns malformed output, hallucinates a field, or simply refuses. That single exercise shapes more of the architecture than any model choice does.
Treat the model as an unreliable collaborator with excellent ideas — design every integration so that its worst day is survivable.
Structured output is non-negotiable
Free-text responses are where production systems go to die. Everything we ship validates model output against a schema before it touches business logic:
- Constrain the model with typed tool calls or JSON schemas, never prose parsing.
- Validate server-side and retry with the validation error appended to the prompt.
- After two failed retries, fall back to a deterministic path — a template, a rule, or a human queue.
Evaluation is a product feature
The teams that succeed with LLMs are the ones that can answer "did yesterday's change make things better or worse?" with a number. We maintain a golden dataset for every feature — real, anonymised cases with agreed-correct outputs — and every prompt or model change runs against it in CI. It is the same discipline as a test suite, and it pays for itself the first time a model upgrade silently changes behaviour.
Latency budgets change the design
A chat interface can stream tokens and feel alive at three seconds. A checkout flow cannot. We tag every LLM call with a latency budget up front: interactive calls get small fast models with aggressive timeouts, while batch enrichment runs on larger models overnight where nobody is waiting. Choosing the model per call site, rather than one model everywhere, halved our p95 latency without hurting quality where it mattered.
The boring parts are the product
Rate limiting, idempotent retries, cost tracking per feature, prompt versioning, audit logs of every generation — none of this is glamorous, and all of it is what separates a feature customers trust from a demo that impressed a stakeholder once. The model is maybe a fifth of the system. The other four fifths are the same disciplined software engineering that made the rest of your platform dependable.
If you are taking an LLM feature to production and want a second pair of eyes on the architecture, talk to us — this is exactly the kind of system we build.
