Adding an LLM feature to an existing product without it becoming a liability

We've put machine-learning models into production before, sentence embeddings for categorising a large catalogue among them, and we've integrated a great many third-party APIs: payments, identity, marketplaces, suppliers. A large language model sits at the intersection of those two. It's an API call. But the output isn't deterministic, the cost scales with how much text goes in and comes back out, and it can be wrong while sounding completely certain. Most of the engineering work is in containing those three properties.

Decide what happens when it's wrong, first

Before wiring anything up, answer one question. If the model returns nonsense, an empty answer, or something subtly incorrect, what does the user see and what does the system do? If the honest answer is "something bad and hard to undo", the feature isn't ready, however good the demo looked. The features that fit early are the ones where a wrong answer is cheap: a draft a human edits, a suggestion they can reject, a summary shown next to its source.

Constrain the output to something you can check

• Ask for structured output. JSON against a schema beats prose you then have to parse, and you can validate it on the way out, rejecting or retrying on a miss.
• Give it a closed set of options wherever you can: "classify into one of these five" instead of "describe the category".
• Keep a deterministic fallback. If validation fails twice, fall back to a rule, a default, or "unavailable". Never ship the unvalidated blob.

Treat the prompt as an untrusted-input boundary

If any part of the prompt is built from user content, a web page, or a document you didn't write, assume it contains instructions aimed at the model. Prompt injection is the SQL injection of this area. Don't give the model tools or data in that context that you wouldn't hand the user directly, keep system instructions separate from user content, and don't let model output trigger a privileged action without a check in between.

Cost and latency are product constraints now

• Price a realistic request, input plus output tokens, and multiply by expected volume before you commit to a design. Building something that works and costs more than it earns is easy.
• Cache aggressively. Identical or near-identical inputs shouldn't pay twice, and a hash of the normalised prompt is usually a good enough cache key.
• Calls take hundreds of milliseconds to seconds, so run them asynchronously and show a pending state. Never block a checkout or a save on one.
• Set a token ceiling per request. One pathological input shouldn't be able to run up a large bill.

It's a dependency, so give it the same treatment as any other

Rate limits, retries with backoff, timeouts, a circuit breaker, a plan for when the provider has an outage. Every bit of discipline you would apply to a payment gateway applies here too. Log the prompt, the response, the model version and the token counts, so that when output quality shifts you can tell whether it was your change or theirs.

The model is the easy part. The retry logic, the validation, the cost ceiling, and the eval set are the feature.

Evaluate it like code, not by vibes

Keep a fixed set of real inputs with known-good outputs, then run the feature against them whenever you change the prompt, the model, or the surrounding logic. Without a regression set, "improving the prompt" is guesswork, and you find out it got worse from a user.

The JSON Formatter helps while you're shaping and validating structured model output. The Hash Generator gives you a stable cache key from a normalised prompt, and the Word Counter is a quick stand-in for gauging how large an input is before you meter tokens properly.

Pin the model version, and treat an upgrade as a deploy

Providers update the model behind an endpoint without asking you first. That's normal in this space, and it can shift behaviour on inputs that used to work fine. A prompt that reliably returned clean JSON with one model version starts including a stray explanatory sentence with the next. Pin an explicit model version wherever the provider offers one, and when you do decide to move to a newer one, re-run the regression set on purpose instead of learning that behaviour changed from an incoming support ticket.

Decide what the model is allowed to see before you decide what it says

Whatever goes into the prompt is now sitting with a third party, subject to whatever data-handling terms are in your agreement with that provider, not your own. Strip or mask anything you wouldn't be comfortable putting in a support ticket to an external vendor: full account numbers, unrelated customers' data dragged in by an overly broad query, anything your own data-processing agreement with customers doesn't clearly cover yet. Make that call before the feature ships. A redaction pass added after a customer asks where their data went is already late.

Give users a way to flag a wrong answer, and feed it back

A regression set catches the problems you already know to check for. It can't catch a failure mode nobody has seen yet. A visible, low-friction way for a user to flag a wrong output, attached to the output itself and not to a generic feedback form elsewhere in the product, turns real usage into the next addition to that set. Without it, a recurring bad output sits unnoticed because no single user's complaint ever reached whoever owns the prompt.

Adversarial and edge-case inputs need their own test pass

Empty input. An input sitting right at the token ceiling. A non-English input, if the product is meant to be multilingual. A prompt written by someone obviously trying to break things. All of these behave differently enough from typical inputs that a regression set built only from realistic examples will miss the problems specific to them. We test the boundaries on purpose, at zero input and at the maximum the UI allows, instead of assuming the well-behaved middle of the input space represents the whole range a real user population will send.

What should you consider before adding an LLM to your product?

Decide what happens when the output is wrong, and only proceed if that path is cheap and reversible. Constrain the model to structured, checkable output with a deterministic fallback. Treat any user-supplied prompt content as untrusted. Budget token cost and latency as real constraints, and keep a fixed evaluation set so prompt changes don't quietly regress.