Building AI
Pipelines That
Actually Work
In Production

Production AI is not about intelligence — it's about reliability. A system that's right 95% of the time and silently wrong 5% of the time will eventually destroy something you care about. This is what I've learned building AI pipelines for clients over the past year.

The Gap Between Demo and Production

A ChatGPT demo hides almost everything that can go wrong. It assumes a fast, reliable internet connection. It assumes the model API will respond in under two seconds. It assumes you'll never hit a rate limit, a context window, or a token budget. It assumes the user's input will be clean, unambiguous, and well-structured.

Production assumes none of that. In production, your pipeline will receive malformed input, stall on slow API responses, hit rate limits at 3am on a Sunday, and encounter edge cases your prompts never anticipated. The question isn't whether these things will happen — it's whether your code will degrade gracefully or fail catastrophically when they do.

The most dangerous failures in AI pipelines aren't the obvious ones — those get caught. The dangerous ones are the silent hallucinations, the confidently wrong extractions, the responses that look fine but contain subtly incorrect data that propagates downstream for weeks before anyone notices.

Choose Your Abstraction Level

LangChain is the React of AI frameworks — extremely popular, occasionally the right tool, and frequently the source of problems it was supposed to prevent. Before reaching for any orchestration framework, ask yourself whether your pipeline is genuinely complex enough to warrant it.

For most use cases — document Q&A, content extraction, classification, simple chat — you can get away with raw API calls wrapped in a few utility functions. This is usually better. You understand exactly what's happening, debugging is straightforward, and you're not fighting an abstraction layer when something breaks at 2am.

Reach for LangChain, LlamaIndex, or similar frameworks when you genuinely need:

• Complex multi-step agent reasoning with dynamic tool selection
• Retrieval augmented generation with a managed vector store
• Multi-model orchestration with conditional routing logic
• Built-in memory and conversation management at scale

If you don't need at least two of those, you probably don't need the framework.

Design for Failure From Day One

This is the part that separates the demo builders from the production engineers. Every call to an LLM API is a network call to a third-party service that can be slow, expensive, unavailable, or rate-limited. Your code needs to account for this from the first line you write.

Retry with exponential backoff

The most basic — and most commonly omitted — protection. Every LLM API call should be wrapped in retry logic with exponential backoff and jitter. Here's a minimal implementation I use in almost every project:

async function callWithRetry<T>
  fn: () => Promise<T>,
  options = { maxRetries: 3, baseDelay: 1000 }
): Promise<T> {
  let attempt = 0;

  while (attempt < options.maxRetries) {
    try {
      return await fn();
    } catch (err) {
      const isRetryable = isRateLimitError(err) || isTransientError(err);
      if (!isRetryable || ++attempt >= options.maxRetries) throw err;

      // Exponential backoff with jitter
      const delay = options.baseDelay * 2 ** attempt
        + Math.random() * 500;
      await sleep(delay);
    }
  }
}

Note the jitter on the delay. When multiple instances of your pipeline hit a rate limit simultaneously, deterministic backoff means they'll all retry at exactly the same time — making the problem worse. Jitter spreads the retries out and prevents thundering herds.

Monitoring Is Not Optional

You cannot improve what you cannot measure. For AI pipelines specifically, this means tracking four things: latency per step, cost per request, error rate by error type, and — most importantly — output quality over time.

That last one is the hard part. Latency and cost are easy to measure. Output quality requires either human review, automated evaluation against a test set, or both. Most teams skip this entirely until a user reports a serious quality regression that's been happening for weeks.

At minimum, log every LLM call with: the full prompt, the raw response, the parsed/extracted output, the latency, the token count, and a unique trace ID that links all the calls in a single pipeline run. Store this somewhere searchable. You will need it.

Hard-Won Rules

After shipping a dozen production AI systems, these are the rules I've stopped negotiating with myself about:

1. Never trust LLM output without validation. Zod schema validation on every extraction. If the model returns something that doesn't match, treat it as a failure and retry.
2. Build a test set before you ship, not after. 50 representative inputs with expected outputs. Run it before every prompt change.
3. Set hard cost budgets per request. Calculate the maximum token cost of a successful pipeline run and set a ceiling 20% above it. Alert when you hit 80%.
4. Stream by default when talking to users. A fast first token feels faster than a slow complete response. Use streaming everywhere the UX allows it.
5. Version your prompts like code. Prompt changes are deployments. Treat them that way — with review, testing, and rollback capability.

Building reliable AI pipelines is a discipline, not a capability. The fundamentals — error handling, observability, testing — are the same ones that make any software system robust. The LLM is the novel part; the engineering around it should be as boring and battle-tested as possible.

If you're building something and want to talk through the architecture, I'm always up for a conversation.