Adding an LLM feature to an existing product without it becoming a liability

We've put machine-learning models into production before - sentence embeddings for categorising a large catalogue - and we've integrated a great many third-party APIs: payments, identity, marketplaces, suppliers. A large language model sits at the intersection of those two. It's an API call, but one whose output isn't deterministic, whose cost scales with how much text goes in and out, and which can be wrong while sounding certain. Most of the engineering is in containing those three properties.

Decide what happens when it's wrong, first

Before wiring anything up, answer this: if the model returns nonsense, an empty answer, or something subtly incorrect, what does the user see and what does the system do? If the honest answer is "something bad and hard to undo", the feature isn't ready, however good the demo looked. The features that fit early are the ones where a wrong answer is cheap - a draft a human edits, a suggestion they can reject, a summary shown next to its source.

Constrain the output to something you can check

• Ask for structured output - JSON against a schema - not prose you then have to parse. Validate it on the way out, and reject or retry on a miss.
• Give it a closed set of options wherever you can: "classify into one of these five", not "describe the category".
• Keep a deterministic fallback. If validation fails twice, fall back to a rule, a default, or "unavailable" - never ship the unvalidated blob.

Treat the prompt as an untrusted-input boundary

If any part of the prompt is built from user content, a web page, or a document you didn't write, assume it contains instructions aimed at the model. Prompt injection is the SQL injection of this area. Don't give the model tools or data in that context that you wouldn't hand the user directly, keep system instructions separate from user content, and don't let model output trigger a privileged action without a check in between.

Cost and latency are product constraints now

• Price a realistic request - input plus output tokens - and multiply by expected volume before committing to a design. It's easy to build something that works and costs more than it earns.
• Cache aggressively: identical or near-identical inputs shouldn't pay twice. A hash of the normalised prompt is usually a good enough cache key.
• Calls take hundreds of milliseconds to seconds. Run them asynchronously, show a pending state, and never block a checkout or a save on one.
• Set a token ceiling per request so one pathological input can't run up a large bill.

It's a dependency - give it the same treatment as any other

Rate limits, retries with backoff, timeouts, a circuit breaker, and a plan for when the provider has an outage. Every bit of discipline you'd apply to a payment gateway applies here. Log the prompt, the response, the model version, and the token counts, so that when output quality shifts you can tell whether it was your change or theirs.

Evaluate it like code, not by vibes

Keep a fixed set of real inputs with known-good outputs, and run the feature against them whenever you change the prompt, the model, or the surrounding logic. Without a regression set, "improving the prompt" is guesswork, and you find out it got worse from a user.

The JSON Formatter helps while you're shaping and validating structured model output; the Hash Generator gives you a stable cache key from a normalised prompt; and the Word Counter is a quick stand-in for gauging how large an input is before you meter tokens properly.

The model is the easy part. The retry logic, the validation, the cost ceiling, and the eval set are the feature.

What should you consider before adding an LLM to your product?

Decide what happens when the output is wrong, and only proceed if that path is cheap and reversible. Constrain the model to structured, checkable output with a deterministic fallback. Treat any user-supplied prompt content as untrusted. Budget token cost and latency as real constraints, and keep a fixed evaluation set so prompt changes don't quietly regress.