When enterprise AI applications scale, they inevitably hit a wall. For many engineering teams, this wall is initially diagnosed as a billing issue, a monthly API invoice that has grown out of control. However, viewing token consumption purely as a financial metric fundamentally misunderstands how LLMs operate in production. Token optimization, unlike its other optimization cousins, is not an accounting exercise; it’s a distributed systems and hardware utilization challenge.

“Token optimization, unlike its other optimization cousins, is not an accounting exercise; it’s a distributed systems and hardware utilization challenge.”

In this guide, we explore, through the lens of Concierge (a latency-sensitive, synchronous customer support agent) and Pathfinder (an asynchronous, multi-step autonomous CI debugging agent), how these systems fell victim to autoregressive bottlenecks as they grew, and how we fixed these issues.

What you’re actually paying for

A token is not a word: treating it like one will break your budgeting “models.” Every major LLM provider tokenizes text using byte-pair encoding (BPE), breaking words into subword units. While common words stay intact, rarer words or punctuation split into fragments. As a rule of thumb, 1 token = 4 characters, or 0.75 words in standard English prose.

When budgeting for production, you must account for the structural pricing spread; providers bill input tokens and output tokens at different rates. Output tokens are typically 4-5X more expensive than input tokens.

As a baseline, assume a mid-tier frontier model runs roughly $3 per million input tokens and $15 per million output tokens.

The quadratic history tax

LLM provider APIs are completely stateless; to make an LLM behave as if it remembers past events, you must resend the entire history of the session and input with every single API call.

This means that a model’s own previous outputs are continuously re-billed to you as inputs on subsequent steps. This triggers a compounding cost that impacts both Concierge and Pathfinder, though their curves scale differently.

Let S be the static system context (instructions and schemas), u be the incoming data per step, and r be the model’s response payload. The input cost for every given turn k is

When you sum this across a complete execution run of N steps, the total input token volume compounds quadratically. 

This O(N^2) accumulation of history is the exact mechanism that causes the explosion in cost and latency.

When we calculate the total input tokens consumed by a single session using the quadratic formula

• Concierge: Consumed 45,300 tokens per 10-turn ticket
• Pathfinder: Consumed 150,000 tokens per 15-step turn

Because Pathfinder’s step increment was 4X larger than Concierge, its token cost curve was drastically steeper. If Pathfinder were to get stuck in an infinite tool-use loop and hit 30 steps, a single run could consume 570,000 tokens.

The solution

Fixing the individual call

Prompt hygiene: Hardcoding static reference documentation into the system prompt means you pay to parse identical text on every turn. So we stripped static text from the prompt and switched to dynamic injection. 

For Concierge, we implemented a RAG step to fetch only the 2-3 policy snippets relevant to the ticket. The prompt dropped from 3,100 tokens to 380. A 60% reduction for a 10-turn thread.

For Pathfinder, we applied automated prompt compression using LLMLingua-2 to compress verbose CI log files before sending them to the model. By filtering out non-essential log lines, we reduced the size of incoming tool observations by 3X without sacrificing debugging accuracy.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
use_llmlingua2=True
)

try:
compressed_result = compressor.compress_prompt(
raw_ci_log_text,
rate=0.33,
force_tokens=["Error", "Exception", "Failed", "Traceback", "FATAL"]
)
# Pass high-density payload to the frontier model
compact_prompt = compressed_result["compressed_prompt"]
except Exception as e:
print(f"Compression failed, falling back to raw log text: {e}")
# Graceful degradation: pass the raw (or truncated) log if compression fails
compact_prompt = raw_ci_log_text

Eliminating the retry: Relying on open-ended prose instructions “return JSON” caused malformation. When parsing failed, the system initiated a synchronous retry, sending the entire accumulated context as if it were a new attempt. We replaced the entire natural language formatting request with strict structural contracts via forced schema validation.

In both Concierge and Pathfinder, we converted the output format to a strict pydantic schema for tool-calling mode and tool-execution payloads. Malformed outputs across both systems dropped to under 0.5%, eliminating tail latency spikes caused by cascading queues.

# Unified Schema Enforcement for Concierge Responses & Pathfinder Tool Execution
from pydantic import BaseModel
from typing import Literal

class TicketResponse(BaseModel):
    reply: str
    category: Literal["shipping", "returns", "billing", "product", "other"]
    escalate: bool
    confidence: float

# The API is structurally locked into emitting validated JSON matching the schema
response = client.messages.create(
    model="claude-opus-4",
    system=SYSTEM_PROMPT,
    messages=messages,
    tools=[
    {
    "name": "respond_to_ticket",
    "description": "Formulate a response and classify the support ticket.",
    "input_schema": TicketResponse.model_json_schema()
    }
    ],
    tool_choice={"type": "tool", "name": "respond_to_ticket"},
)

Output token bounding: Models naturally generate verbose reasoning chains and conversational filler, inflating expensive output tokens. Where the LLM provider exposes logit bias, you can directly suppress every token outside the valid set at decode time; where it doesn’t, constrained decoding libraries (Outlines, Guidance) or a forced tool call with an enum-typed schema will get you the same guarantee.

class ClassifyOnly(BaseModel):
    category: Literal["shipping", "returns", "billing", "product", "other"]
    priority: Literal["low", "medium", "high", "urgent"]

State management 

The stateless nature of the models meant we had to parse the static prompt prefix and historical steps on every turn. We introduced explicit cache breakpoints to allow the inference engine to reuse the states of static blocks. We altered both Concierge and Pathfinder to flag stable, historical segments for caching. Under standard vendor pricing, cache reads are discounted by 90%. It is important to check with your vendor on whether caching is enabled. 

# Caching the stable history prefix for a multi-turn session
response = client.messages.create(
    model="claude-sonnet-4",
    max_tokens=4096,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"} # Cache hits drop prefix costs by 90%
    }],
    tools=TOOL_SCHEMAS,
    messages=session_history + [{"role": "user", "content": current_step_input}],
)

For a 10-turn Concierge chat, this dropped input costs by ~70%. For a 15-step Pathfinder trajectory, it resulted in a 76% cost reduction.

Semantic caching 

Duplicate queries across separate sessions were triggering redundant frontier model invocations. We implemented a vector similarity cache layer upstream of the LLM using Redis. Our Concierge service analysis showed that 34% of customer support tickets were semantic duplicates of common FAQs.

Intercepting these requests reduced latency to sub-50ms for hits. Because of the nature of CI pipeline logs, we have not yet found a suitable cache for Pathfinder’s inputs. 

import os
import json
import redis
from redis.commands.search.query import Query

# Configure connection via environment variable for environment portability
redis_url = os.environ.get("REDIS_URL", "redis://localhost:6379")
r = redis.Redis.from_url(redis_url)

def get_cached_response(tenant_id, query_text, threshold=0.92):
    try:
results = r.ft(f"cache_idx:{tenant_id}").search( # scoped by tenant -- see below
Query("*=>[KNN 1 @vector $vec AS score]").sort_by("score").dialect(2),
query_params={"vec": query_vec.tobytes()},
)
except redis.RedisError as e:
print(f"Redis cache error: {e}")
return None # Fail-open: gracefully fall back to a cache miss)
  query_vec = embed(query_text) # small, fast bi-encoder -- not the frontier model

try:
results = r.ft(f"cache_idx:{tenant_id}").search( # scoped by tenant -- see below
Query("*=>[KNN 1 @vector $vec AS score]").sort_by("score").dialect(2),
query_params={"vec": query_vec.tobytes()},
)
except redis.RedisError as e:
print(f"Redis cache error: {e}")
return None # Fail-open: gracefully fall back to a cache miss

Semantic caching could be a security problem, because if you choose a global cache, Customer A’s account-specific answer could get served to Customer B because their phrasing embeddings are close enough. To mitigate this, we split the cache into two tiers: a global cache for tenant-agnostic content, and a per-tenant, per-user namespace keyed with the tenant ID baked into the prefix itself for anything touching account state.

Cache poisoning is another risk: we only write to the cache from responses that passed schema validation and the injection-pattern classifier, we stamp every cache entry with its source traceId, and we encourage routine purging of unknown caches.

Context compaction

Uncapped conversation or agent trajectories allowed N to grow continuously, expanding the cost curve and causing latency degradation. We capped N by implementing a sliding window that summarizes historical context via a small, ultra-cheap model. For Concierge, we kept the last 3 turns verbatim while condensing older turns into a rolling metadata block.

For Pathfinder, when the debugging steps exceeded 4 runs, we trimmed and summarized the oldest tool execution outputs into a compact chronological timeline, transforming the open-ended quadratic cost explosion into a predictable, bounded window.

def compact_session_history(history_steps: List[Dict[str, Any]], keep_recent: int = 3) ->     List[Dict[str, Any]];
    """Flattens older history into a cheap summary block, preserving recent context."""
    if len(history_steps) <= keep_recent:
        return history_steps
    old_steps = history_steps[:-keep_recent]
    recent_steps = history_steps[-keep_recent:]
    
# Compress the old history using a fast, low-cost utility model
try:
historical_summary = summarize_with_utility_model(old_steps)
# Note: Anthropic prohibits 'system' roles in the messages array.
# Using 'assistant' ensures cross-provider compatibility.
return [{"role": "assistant", "content": f"[System Context: Summary of prior steps: {historical_summary}]"}] + recent_steps
except Exception as e:
print(f"History compression failed: {e}")
# Fallback: Return the uncompressed history to gracefully degrade
return history_steps

Model cascading

Directing every single operation to an expensive frontier model represents massive overprovisioning for mundane tasks. We integrated LiteLLM as an internal routing gateway to implement model cascading, routing every request to the lowest-cost model capable of completing the task.

“Directing every single operation to an expensive frontier model represents massive overprovisioning for mundane tasks.”

# litellm_config.yaml
model_list:
  - model_name: fast-path
    litellm_params:
      model: openai/mistral-support-ft
      api_base: http://vllm-internal:8000/v1
  - model_name: frontier-path
    litellm_params:
      model: anthropic/claude-opus-4

Simple, repetitive tasks are routed to a lower model, which offloads 70% of Concierge chats from the frontier model. For Pathfinder, we broke the agent loop down into separate sub-tasks: high-level planning, tool selection, and code-patch synthesis remained with the frontier model, while mechanical, text-heavy operations, such as log parsing, regex extraction, and error-string formatting, were offloaded to the lower models. This hybrid orchestration reduced Pathfinder’s token costs by more than 50%.

What’s next

The transformation of Concierge and Pathfinder proves a fundamental truth about production AI. You cannot achieve scale by simply relying on the natural language capabilities of a frontier model. You must engineer the system around it. By shifting your focus from naive token reduction to maximizing system resource utilization, we reclaimed absolute control over the infrastructure.

“Efficiency in the era of gen AI is not defined by how cheaply you can operate but by how densely you can pack information.”

Efficiency in the era of gen AI is not defined by how cheaply you can operate but by how densely you can pack information, how quickly you can serve it, and how reliably you can parse the output. The architectural decisions detailed here represent more than just a token optimization strategy; they are a required foundation for building high-throughput, battle-tested, and resilient AI systems at scale.

The post The systems guide to production token optimization appeared first on The New Stack.

Every engineering team deploying AI agents eventually discovers an uncomfortable truth: the model isn’t the biggest expense. The hidden cost is everything around it: repeated retrievals, duplicate prompts, unnecessary tool calls, oversized context windows, multiple agents reasoning over the same information. Individually, these architectural decisions seem harmless. At production scale, they become a severe tax on latency, infrastructure, and cloud spend.

A proof-of-concept agent that answers 50 questions a day can tolerate inefficiencies. An enterprise platform coordinating thousands of requests per minute cannot.

This article explores practical techniques for engineering token-efficient AI systems without sacrificing output quality. Rather than focusing solely on prompt compression, we will optimize the entire workflow from routing and retrieval to caching and model selection.

Why token optimization is a systems problem

Most discussions around token optimization begin and end with prompt engineering. In practice, architecture drives token consumption.

Consider a typical multi-agent workflow:

User 
  ↓
Intent Agent
  ↓
Retriever
  ↓
Research Agent
  ↓
Planning Agent
  ↓
Writer Agent
  ↓
Reviewer Agent
  ↓
Final Response

At each stage, the system might retrieve the same documents, repeat identical instructions, call the same model, and resend the entire conversation history. By the time a response reaches the user, the architecture has processed tens of thousands of unnecessary tokens.

“Improving efficiency requires redesigning the workflow, not just shortening the prompts.”

Improving efficiency requires redesigning the workflow, not just shortening the prompts.

Architecture overview

A production-ready, token-efficient architecture introduces optimization before every expensive model invocation.

User Request
       │ 
       ▼
Intent Router 
       │ 
       ▼
Semantic Cache ───────► Cached Response
       │ 
       ▼
Context Budget Manager
       │ 
       ▼
Adaptive Retriever
       │ 
       ▼
Model Router
       │ 
       ▼
LLM
       │ 
       ▼
Validated Response

“The large language model is no longer the first component. It is the final, most expensive operation.”

Notice the critical shift: the large language model is no longer the first component. It is the final, most expensive operation.

Step 1: Install modern dependencies

Use the latest package structure to avoid deprecated imports and align with the current LangChain ecosystem.

Bash
pip install \
   langchain \
   langchain-core \
   langchain-openai \
   langchain-community \
   fastapi \
   faiss-cpu \
   tiktoken \
   rank-bm25 \
   pydantic \
   python-dotenv

Step 2: Configure the model

Production systems must configure retries, timeouts, and credentials through the environment.

Python
import os
from langchain_openai import ChatOpenAI 

api_key = os.getenv("OPENAI_API_KEY") 
if not api_key: 
    raise ValueError("OPENAI_API_KEY must be configured.")

llm = ChatOpenAI( 
    model="gpt-4o-mini", 
    temperature=0, 
    api_key=api_key, 
    timeout=30.0, 
    max_retries=2, 
)

Setting a low temperature improves consistency, while explicit timeouts and retry limits help the system recover gracefully from transient API failures.

Step 3: Route before you generate

Not every request requires a large language model. Deterministic logic can often answer simple questions. Routing inexpensive requests away from the LLM yields the most significant cost reduction in production systems.

Python
def classify_request(question: str) -> str:
    q = question.lower()

    if "status" in q:
        return "metrics"

    if "runbook" in q:
        return "retrieval"
   
    return "generation"

Step 4: Add a semantic cache

One of the simplest and most effective optimizations is an exact-match cache, which returns a previously generated response when the same question is asked against the same retrieved documents, avoiding unnecessary model calls.

Python
import hashlib

# Using an exact-match (lexical) cache
exact_match_cache = {}

def cache_key(question: str, sources: list[str]) -> str:
    """
    Generate a deterministic cache key from the user question
    and the retrieved document identifiers.
    """
    fingerprint = question + "|" + "|".join(sorted(sources))
    return hashlib.sha256(fingerprint.encode()).hexdigest()

# Example usage in the pipeline:
# key = cache_key(question, source_ids)
# if key in semantic_cache:
#     return semantic_cache[key]

Step 5: Budget your context

Most retrieval pipelines return far more text than the model actually needs. Instead of stuffing the context window with every retrieved document, establish a strict context budget.

Python
import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4o-mini")
MAX_CONTEXT_TOKENS = 2500

def build_context(chunks):
    context = []
    used = 0

    for chunk in chunks:
        tokens = len(encoder.encode(chunk.page_content, disallowed_special=()))

        if used + tokens > MAX_CONTEXT_TOKENS:
            break 

        context.append(chunk.page_content)
        used += tokens

    return "\n\n".join(context)

Step 6: Retrieve once

Repeated retrieval is a surprisingly common flaw in multi-agent systems. The rule is simple: retrieve once, reuse everywhere.

Python
from langchain_core.documents import Document
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

documents = [
    Document(
        page_content="Database latency often follows connection pool exhaustion.",
        metadata={"source": "db_runbook"},
    ),
    Document(
        page_content="Node pressure can increase API response times.",
        metadata={"source": "cluster_runbook"},
    ),
]

embeddings = OpenAIEmbeddings(api_key=api_key)
index = FAISS.from_documents(documents, embeddings)

retrieved_docs = index.similarity_search(question, k=4)
shared_context = build_context(retrieved_docs)

Now, every downstream agent consumes the same optimized context instead of launching its own redundant retrieval pipeline.

Step 7: Route models intelligently

Large models should solve complex problems. Everything else belongs to a smaller, faster model.

Python
from langchain_openai import ChatOpenAI

small_model = ChatOpenAI(model="gpt-4o-mini", temperature=0, api_key=api_key)
large_model = ChatOpenAI(model="gpt-4.1", temperature=0, api_key=api_key)

def choose_model(question: str):
    """Route requests to the most appropriate model based on complexity."""
    if len(question) < 200:
        return small_model
    return large_model

This strategy drastically reduces operational costs without noticeably affecting response quality.

Step 8: Estimate tokens before sending

Without token telemetry, optimization is just guesswork. Monitoring usage makes efficiency measurable and helps engineers detect cost regressions.

Python
import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4o-mini")

def estimate_tokens(messages):
    """
    Estimate input tokens for an OpenAI-style chat payload.
    Note: This is an estimate, not an exact billing calculation.
    """
    tokens_per_message = 3
    tokens_per_name = 1
    total = 0

    for message in messages:
        total += tokens_per_message
        for key, value in message.items():
            if isinstance(value, str):
                total += len(encoder.encode(value))
            if key == "name":
                total += tokens_per_name

    # Every reply is primed with additional assistant tokens.
    total += 3
    return total

Step 9: Validate responses

Production systems must return structured outputs to ensure downstream systems receive predictable, well-formed data.

Python
from pydantic import BaseModel

class AgentResponse(BaseModel):
    answer: str
    sources: list[str]

def validate_response(answer: str, sources: list[str]):
    """Validate and serialize the agent response using a structured schema."""
    response = AgentResponse(
        answer=answer,
        sources=sources,
    )
    return response.model_dump()

Step 10: Build the optimized pipeline

Finally, assemble the architectural components into a single workflow. Notice how failures degrade gracefully instead of crashing the service.

Python
import logging

from langchain_core.prompts import ChatPromptTemplate

logger = logging.getLogger(__name__)

def run_pipeline(question: str):
    """Execute the token-efficient AI workflow with graceful degradation."""
    try:
        route = classify_request(question)

        # Route deterministic requests away from the LLM.
        if route == "metrics":
            return {
                "answer": "Retrieve metrics directly from the monitoring system.",
                "sources": [],
            }

        # Retrieve context once.
        docs = index.similarity_search(question, k=4)

        context = build_context(docs)

        source_ids = [
            doc.metadata.get("source")
            for doc in docs
            if doc.metadata.get("source")
        ]

        # Check exact-match cache.
        key = cache_key(question, source_ids)

        if key in exact_match_cache:
            return exact_match_cache[key]

        # Select the most appropriate model.
        model = choose_model(question)

        # Keep trusted instructions separate from untrusted user input.
        prompt_template = ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    (
                        "Answer the user's question using ONLY the provided context. "
                        "If the answer cannot be determined from the context, say so."
                        "\n\nContext:\n{context}"
                    ),
                ),
                ("user", "{question}"),
            ]
        )

        chain = prompt_template | model

        result = chain.invoke(
            {
                "context": context,
                "question": question,
            }
        )

        payload = validate_response(
            answer=result.content,
            sources=source_ids,
        )

        # Cache validated response.
        exact_match_cache[key] = payload

        return payload

    except Exception:
        logger.exception("Token-efficient pipeline failed.")

        # Gracefully degrade instead of crashing.
        return {
            "answer": (
                "The AI pipeline encountered an error. "
                "Please continue using the standard operational workflow."
            ),
            "sources": [],
        }

What actually reduced token usage?

When teams instrument architectures like this, the largest savings rarely come from editing prompts. They come from eliminating unnecessary work.

The biggest improvements typically stem from:

• Retrieving documents once instead of multiple times.
• Caching semantically identical requests.
• Routing simple requests away from the LLM.
• Limiting context with explicit token budgets.
• Selecting the smallest suitable model.

These architectural shifts reduce cost and latency while making system behavior significantly easier to reason about.

Lessons learned

Several core principles consistently emerge when optimizing AI systems for production:

• Treat tokens like infrastructure: Tokens are a finite resource, just like CPU cycles or memory. Monitor them, budget them, and optimize them.
• Retrieval is usually the largest source of waste: Repeated retrieval often contributes more unnecessary tokens than verbose prompts. Share context whenever possible.
• Bigger models are not always better: Smaller, faster models effectively handle many operational tasks. Reserve larger models for genuinely complex reasoning.
• Caching is an engineering feature: A semantic cache is more than a performance optimization—it is a core architectural component that reduces cost, latency, and provider dependence.
• Measure before you optimize: Instrumentation must accompany every production deployment.

As AI systems mature, success will increasingly depend on engineering efficiency rather than raw model size. The hidden tax of AI agents is rarely a single expensive prompt; it is the accumulation of redundant retrievals, oversized contexts, unnecessary model calls, and repeated reasoning across distributed workflows.

“The most effective production AI systems are not the ones that generate the most tokens. They are the ones that generate only the tokens they truly need.”

By treating token consumption as a systems engineering problem, organizations can build AI platforms that are faster, less expensive, and highly scalable. Routing requests intelligently, budgeting context, sharing retrieval results, validating structured outputs, and introducing semantic caching are practical techniques that guarantee efficiency without compromising quality.

The most effective production AI systems are not the ones that generate the most tokens. They are the ones that generate only the tokens they truly need.

The post Stop the token bleed: building token-efficient multi-agent systems appeared first on The New Stack.

Consider a platform team managing a shared GPU cluster with a mix of B200s, H100s, and recently added B300s. Every Monday morning, the on-call engineer finds a queue of pending jobs from the weekend. Training workloads are stuck because they landed on H100s and triggered Out-Of-Memory (OOM) errors. Inference jobs sit idle because the small MIG (Multi-Instance GPU) slices are exhausted, even though larger slices sit empty right next to them.

Their fix? A 200-line Bash script running every 30 minutes to reconfigure MIG profiles, reschedule stuck jobs, and send a Slack alert when it succeeds, or a PagerDuty alert when it fails.

The root of the problem

Here is what was actually broken: Kubernetes treated every GPU as an identical unit. The resource limit nvidia.com/gpu: 1 was the extent of its awareness. The scheduler had no idea if it was handing a pod a 192GB B200 or an 80GB H100. A training job requiring 150GB of VRAM would land on an H100 and immediately OOM, while B200 nodes sat completely idle nearby.

“Kubernetes treated every GPU as an identical unit.”

The industry’s accepted “fix” relied heavily on node labels, taints, tolerations, and separate node pools per GPU type. Every workload manifest hardcoded hardware assumptions. Adding a single new GPU generation meant updating 40 different Helm charts.

The MIG illusion

MIG made this worse. MIG slices a single GPU into smaller, isolated partitions, each with dedicated memory and compute. Instead of one inference job monopolizing a B200, you can run seven smaller jobs on the same card.

In theory, this sounds great. But when you enable MIG in Kubernetes, each profile becomes a separate, rigid resource type (e.g., nvidia.com/mig-1g.10gb, nvidia.com/mig-3g.40gb). There is no fallback logic. You cannot instruct a job to “try a small slice first, and use a large one if nothing else is free.” When small slices run out, jobs sit pending, while large slices go to waste.

“When small slices run out, jobs sit pending, while large slices go to waste.”

This inefficiency was accepted as the cost of running GPU workloads on Kubernetes. Then, Kubernetes 1.34 shipped.

Dynamic Resource Allocation (DRA)

Kubernetes 1.34 introduced Dynamic Resource Allocation (DRA), fundamentally changing the scheduling model. GPU drivers now publish structured data. Instead of requesting a generic nvidia.com/gpu: 1, workloads can express explicit intent using Common Expression Language (CEL):

1. “Give me an H100 or better with at least 40GB of memory.”
2. “Give me a MIG slice: small if available, medium if not, or a full GPU if necessary.”
3. “Give me four GPUs connected via NVLink.”

Example 1: Hardware and memory requirements

“Give me an H100 or better with at least 40GB memory.”

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: h100-or-better-40gb
spec:
  spec:
    devices:
      requests:
        - name: gpu
          deviceClassName: gpu.nvidia.com
          count: 1
          selectors:
            - cel:
                # Attribute names are illustrative.
                # Your NVIDIA DRA driver must expose these fields.
                expression: >
                  device.attributes["gpu.nvidia.com"].memory >= quantity("40Gi") &&
                  device.attributes["gpu.nvidia.com"].generation in ["H100", "B200", "B300"]
---
apiVersion: batch/v1
kind: Job
metadata:
  name: training-job-h100-or-better
spec:
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
        - name: gpu
          source:
            resourceClaimTemplateName: h100-or-better-40gb
      containers:
        - name: trainer
          image: nvcr.io/nvidia/pytorch:24.12-py3
          command: ["python", "train.py"]
          resources:
            claims:
              - name: gpu

Example 2: Flexible MIG fallback

“Give me a MIG slice: small if available, medium if not, or full GPU if needed.”

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-prefer-small-then-medium-then-full
spec:
  spec:
    devices:
      requests:
        - name: gpu
          deviceClassName: gpu.nvidia.com
          count: 1
          selectors:
            - cel:
                # Prefer any acceptable MIG profile or full GPU
                expression: >
                  device.attributes["gpu.nvidia.com"].profile in [
                    "mig-1g.10gb",
                    "mig-2g.20gb",
                    "mig-3g.40gb",
                    "full-gpu"
                  ]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-service-flexible-gpu
spec:
  replicas: 2
  selector:
    matchLabels:
      app: inference-service
  template:
    metadata:
      labels:
        app: inference-service
    spec:
      resourceClaims:
        - name: gpu
          source:
            resourceClaimTemplateName: mig-prefer-small-then-medium-then-full
      containers:
        - name: inference
          image: nvcr.io/nvidia/tritonserver:24.12-py3
          args: ["tritonserver", "--model-repository=/models"]
          resources:
            claims:
              - name: gpu

Example 3: Topology constraints

“Give me 4 GPUs that are NVLink-connected.”

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: four-nvlink-connected-gpus
spec:
  spec:
    devices:
      requests:
        - name: gpus
          deviceClassName: gpu.nvidia.com
          count: 4
          selectors:
            - cel:
                # Attribute names are illustrative.
                # Some NVIDIA DRA setups may model this through ComputeDomains.
                expression: >
                  device.attributes["gpu.nvidia.com"].fabric == "nvlink"
      constraints:
        - requests: ["gpus"]
          matchAttribute: "gpu.nvidia.com/nvlinkDomain"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: distributed-training-nvlink
spec:
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
        - name: gpus
          source:
            resourceClaimTemplateName: four-nvlink-connected-gpus
      containers:
        - name: trainer
          image: nvcr.io/nvidia/pytorch:24.12-py3
          command:
            - torchrun
            - --nproc_per_node=4
            - train.py
          resources:
            claims:
              - name: gpus

The engineering takeaway

This architecture requires one manifest. It eliminates fragile node selectors and the need to duplicate job definitions for every new hardware generation. As GPU clusters become increasingly heterogeneous, mixing H100s, B200s, B300s, and whatever silicon drops next, the old integer-based scheduling model breaks down. DRA represents Kubernetes finally maturing to support the nuanced realities of production AI workloads.

The post Say goodbye to K8s GPU pain: How DRA changes everything appeared first on The New Stack.

The AI ecosystem is drowning in tutorials on how to build a retrieval-augmented generation (RAG) app in five minutes. The pitch is appealing but flawed: Chunk a document, run it through an embeddings API, load it into a vector database, and slap a UI on top. This setup works locally. It might even survive a beta test with friendly users. But launch it into a production-grade B2B SaaS environment, and the architecture collapses.

Enterprise applications don’t handle neat, static files. They ingest dynamic, unpredictable streams of live data bound by strict legal and compliance constraints. Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.

“Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.”

Here is exactly what breaks when a naive RAG setup hits production, and the architectural trade-offs needed to fix it.

Bottleneck 1: the synchronous ingestion trap

Synchronous data ingestion is the most prevalent architectural flaw in new AI products. A user uploads a 500-page compliance manual. The client makes a POST call to a web server, which parses the document, splits the text, iterates over a sequence of synchronous API calls to OpenAI or Cohere for vectorization, and writes those vectors to the database.

This approach introduces two critical failures:

• Timeouts: A 500-page document rarely finishes processing within standard HTTP timeouts (30 to 60 seconds) while waiting for the embedding API.
• Cascade failures: If the system hits rate limits or latency spikes, the entire ingestion operation fails, throwing a 500 error and losing the user’s document.

The fix: the batched fan-out pipeline

Production-grade AI pipelines require persistent events rather than simple HTTP calls. However, sending the whole 500-page document to be processed by one consumer from Kafka or RabbitMQ is a mistake. If a consumer spends 10 continuous minutes generating embeddings, it misses its broker heartbeat. Assuming the worker died, the broker kills the consumer and triggers a partition rebalance, creating an infinite loop of duplicated work and stalled processing.

“Synchronous data ingestion is the most prevalent architectural flaw in new AI products.”

Conversely, granular chunking, where every chunk becomes an individual Kafka message, launches a self-inflicted denial-of-service (DoS) attack on downstream services. A document with 1,500 chunks generates 1,500 individual messages. This instantly exceeds upstream requests per minute (RPM) limits and floods the pipeline with network overhead.

The engineering sweet spot is a batched fan-out approach:

• Asynchronous uploads: The web API stores the raw file in Amazon S3, triggers a document_uploaded event, and instantly returns a 202 Accepted status. This single, asynchronous path processes one-page invoices and 100-page SOC2 reports with equal reliability, eliminating the technical debt of maintaining separate “fast” and “slow” ingestion routes.
• Micro-batching: A lightweight “Spitter” consumer downloads the file, chunks it, and groups those chunks into optimized micro-batches (e.g., 64 chunks per batch).
• Controlled embedding: Embedding workers pull these batched events. To prevent concurrent workers from breaching upstream RPM limits, avoid fragile sleep() delays. Instead, enforce a token bucket rate limiter at the consumer level or strictly cap the number of active message broker partitions.

Python

# Conceptual snippet for architectural illustration
def handle_document_upload(event):
try:
raw_text = download_from_s3(event.file_uri)
chunks = semantic_chunking(raw_text)
except Exception as e:
# Log failure and raise so the message broker routes this to a Dead Letter Queue (DLQ)
print(f"Failed to process document {event.file_uri}: {e}")
raise

# Batch size heavily depends on the downstream embedding model's context limits
batch_size = int(os.environ.get("EMBEDDING_BATCH_SIZE", 64))

for i in range(0, len(chunks), batch_size):
chunk_batch = chunks[i:i + batch_size]
kafka.publish("embedding_tasks", {
"tenant_id": event.tenant_id,
"document_id": getattr(event, "document_id", event.file_uri),
"chunks": chunk_batch
})

This keeps individual consumer tasks short, respects upstream rate limits by maximizing payload density, and allows horizontal scaling of embedding workers during traffic spikes.

Bottleneck 2: the multi-tenant nightmare

Developers often treat multi-tenancy as an afterthought. The simplest way to handle multiple B2B tenants using a single RAG system is logical segregation, where all vectors reside in a large index, and each entry has a tenant_id associated with its metadata. Upon retrieval, the application filters results by adding a clause to the metadata payload.

Flaws of the approach:

1. Security vulnerabilities: Relying on application-level filtering creates an unacceptable risk. If an engineer omits or misconfigures a metadata filter, one client can access another’s confidential data. In highly regulated environments, this breaks compliance.
2. The noisy neighbor problem: If one customer uploads 10 million vectors to the shared index, memory usage skyrockets during vector searches. This degrades performance across the entire system, even for tenants with a handful of documents.

The fix: serverless compute-storage decoupling

Echo-chamber thinking assumes that the only solution is to provide each tenant with its own dedicated database cluster. This is prohibitively expensive and practically impossible to manage in a modern-day SaaS offering. 

The true gold standard here is using next-generation serverless vector databases like Pinecone Serverless or managed Qdrant implementations, which make a clear distinction between computing and storage.

Engineering takeaway: stop building complex multi-tenant routing logic in your application code. Push the isolation boundary down to the infrastructure layer using serverless namespaces.

Bottleneck 3: the semantic caching trap

Once ingestion is asynchronous and tenants are segregated, inference costs become the final bottleneck. Hitting an LLM API for every individual query is economically unsustainable.

The industry defaults to semantic caching: embed the user’s prompt, calculate its cosine similarity against previous prompts, and return a pre-calculated LLM response if the score exceeds a set threshold (e.g., 0.95).

Why semantic caching fails

Embeddings capture overall semantic meaning, but they miss specific contexts and entities. The prompts “What was the holiday policy in 2023?” and “What is the holiday policy for 2024?” share a near-perfect cosine similarity score. The core semantics match, but returning a cached answer feeds the user incorrect or contradictory information.

The fix: hybrid verification vs. native prompt caching

To scale without compromising accuracy, there are only two choices to consider: application-layer validation or infrastructure-layer optimization.

Strategy A: combined lexical filtering and intent routing

When using an application-layer caching system (for instance, Redis), you need to layer the semantics search on top of extremely light guardrails.

1. Exact-match filter: Apply a token-validation filter over vector similarity. In the case where the cached query is “2023”, and the current query is “2024,” throw out the cache hit right away.
2. Intent routing: Before serving a cached answer, use an inexpensive, fast model as an intent match router.

Python

Query A: {incoming_query}
Query B: {cached_query}

Do these queries have the exact same intent and require the exact same factual answer? 
Respond only with YES or NO.

Strategy B: infrastructure-level prompt caching

If the system cannot tolerate the added latency of an application-layer verification router, bypass custom caching entirely and offload the problem to the infrastructure.

Modern LLM providers natively support prompt caching. It is crucial to understand what is being cached here: it is not the user’s short question. When multiple users query the same corporate knowledge domain, the massive system instructions and the heavy retrieved context documents (often 10k+ tokens) are cached automatically at the provider’s inference layer.

“Stop approaching AI like a magic black box and instead approach it as a distributed systems problem.”

The application sends the full RAG query every time. The provider recognizes the repeated context block, slashes context token costs by up to 80%, and drops the time-to-first-token (TTFT) to milliseconds.  

The infrastructure surrounding a foundation model separates a prototype from a production AI system. Stop approaching AI like a magic black box and instead approach it as a distributed systems problem, and things will fall into place. Batched fan-out asynchronous queuing solves timeout and rate-limiting issues. Serverless namespacing resolves compliance risks. Prompt caching and intent routing secure unit economics. Designing a native AI product today means engineering for inevitable API failures, cross-tenant data attacks, and runaway LLM costs.

The post Can prompt caching tame RAG costs without sacrificing accuracy? appeared first on The New Stack.

Caching was one of the most critical optimizations in modern AI systems long before most teams realized it. Early prototypes of Retrieval-Augmented Generation (RAG) pipelines, AI copilots, and semantic search platforms often performed perfectly on small datasets and with limited traffic. 

But as soon as real production workloads arrived, tail latency, compounding infrastructure costs, and repeated retrieval operations started becoming impossible to ignore.

Our first instinct was that it was an easy fix. Redis would solve this effortlessly.

It was fast, simple, tested, and already trusted in high-scale systems for session storage, API caching, and rate limiting. Exact-match prompt caching dramatically reduced response times, allowing many repeated AI requests to be served in milliseconds without touching the expensive retrieval or inference layers again. For a while, Redis proved us right and solved almost every performance problem we had.

Until our workloads changed.

Traditional string-matching caches break down the moment your infrastructure becomes semantic. Human language variation means two users will ask for the exact same information using completely different wording. 

“Traditional string-matching caches break down the moment your infrastructure becomes semantic.”

Because Redis relies on exact string matches, it misses those connections entirely, creating duplicate, fragmented cache entries for identical intents. Before long, our hit rates tanked, memory utilization spiked, and we were stuck with a massive cloud bill for storing redundant data contexts.

That was when semantic caching via vector databases started to look attractive. On paper, it seemed like the perfect architectural evolution: match queries based on vector-distance math so that varied prompts could reuse old embeddings, context chunks, or past LLM answers.

Of course, production reality was far messier than the hype suggested. Vector database caching introduced its own set of problems: latency spikes, false-positive matches, embedding drift, operational complexity, and difficult tuning decisions around similarity thresholds. In some workloads, semantic caching significantly improved performance. In others, it became slower and more expensive than the Redis setup it was supposed to replace.

“Of course, production reality was far messier than the hype suggested.”

What we eventually learned is that Redis and vector databases solve fundamentally different caching problems. One optimizes exact retrieval speed. The other optimizes semantic reuse. Treating them as interchangeable technologies led to architectural mistakes that only became apparent under real production traffic.

The AI architecture we started with

Before the caching problems started appearing, our AI stack looked fairly standard for a modern Retrieval-Augmented Generation (RAG) system. The pipeline was designed around three major stages: embedding generation, document retrieval, and LLM inference.

A user query first entered the API layer, where preprocessing handled normalization, authentication, rate limiting, and conversation context assembly. Once the request was validated, the query was converted into an embedding vector using an embedding model. That vector was then used to retrieve semantically relevant chunks from a vector database before the final context was passed into the language model for response generation.

The simplified request flow looked like this:

• User sends a query
• Query is embedded into a vector
• Vector search retrieves relevant documents
• Retrieved context is assembled into a prompt
• LLM generates the final response
• Response is optionally cached

On a small scale, this worked perfectly. The real headaches started when traffic scaled and we noticed the exact same database queries and heavy inference workloads hitting us thousands of times an hour.

One of the first optimizations we introduced was Redis-based caching

The initial idea was straightforward: avoid recomputing expensive operations for repeated requests. We started by caching exact prompt-response pairs, embedding results, and frequently accessed retrieval outputs. 

Because Redis operates entirely in memory, lookup times were rapid, immediately reducing pressure on both the vector database and the LLM layer.

A simplified Redis caching flow looked like this:

const cacheKey = `llm_cache:${hash(userQuery)}`;

try {
const cachedResponse = await redis.get(cacheKey);
if (cachedResponse) {
return JSON.parse(cachedResponse);
}
} catch (cacheError) {
console.warn("Cache read failed, falling back to LLM:", cacheError);
}

const embedding = await generateEmbedding(userQuery);
const documents = await vectorSearch(embedding);
const response = await generateLLMResponse(documents);

try {
await redis.set(cacheKey, JSON.stringify(response), "EX", 3600);
} catch (cacheError) {
console.error("Failed to write response to cache:", cacheError);
}

return response;

The immediate results were excellent: near-instant response times for duplicate prompts, reduced API costs, and stable infrastructure that handled high traffic without requiring aggressive LLM scale-out. If a query matched an existing keyword-for-word, we skipped the entire expensive AI pipeline and served the answer straight from memory.

Why Redis looked like the perfect solution

Unlike vector indexes, Redis gave us clean, predictable metrics for memory usage, throughput, and latency characteristics under load. There were no similarity thresholds to configure, no ANN indexes to optimize, and no recall-versus-latency trade-offs to worry about. A cache key either existed or it didn’t. That predictability made the system easier to reason about during incidents and easier to scale under pressure.

We initially used Redis across multiple layers of the AI pipeline, including prompt-response caching, embedding caching, session state storage, rate limiting, temporary conversation memory, and caching frequently accessed retrieval outputs.

const cacheKey = `prompt:${hash(query)}`;
try {
const cached = await redis.get(cacheKey);
if (cached) {
return JSON.parse(cached);
}
} catch (cacheError) {
console.warn("Cache read or parse failed, bypassing cache:", cacheError);
}

We also started caching embeddings because generating embeddings became surprisingly expensive at scale. Even though embeddings were much cheaper than LLM inference, generating them repeatedly for popular queries still consumed a noticeable amount of compute resources.

const embeddingKey = `embedding:${hash(query)}`;
let embedding;

try {
const cachedEmbedding = await redis.get(embeddingKey);
if (cachedEmbedding) {
embedding = JSON.parse(cachedEmbedding);
}
} catch (cacheError) {
console.warn("Redis read failed, proceeding to generate new embedding:", cacheError);
}

if (!embedding) {
embedding = await createEmbedding(query);

try {
await redis.set(
embeddingKey,
JSON.stringify(embedding),
"EX",
86400
);
} catch (cacheError) {
console.error("Failed to write embedding to Redis:", cacheError);
}
}

As an added bonus, scaling horizontally with Redis clusters is a familiar process for most infrastructure teams. For a while, Redis significantly reduced both latency and infrastructure costs. Cache hit rates were high, GPU workloads dropped, and the vector database handled far fewer retrieval requests.

The architecture looked efficient enough that we initially believed Redis alone would resolve most of our AI caching challenges. That belief did not survive real semantic workloads for very long.

The problem was that AI workloads rarely behave like traditional web workloads for long. Users almost never ask the same question repeatedly. Instead, they ask semantically similar questions with slightly different wording, tone, or context. From Redis’ perspective, these were entirely different cache keys, even when the retrieval results and final answers were nearly identical. That limitation eventually pushed us toward semantic caching using vector databases.

Why we moved toward vector DB caching

Unlike Redis, vector databases do not rely on exact string matching. Instead, they compare numerical embeddings that represent the semantic meaning of text. This made it possible to retrieve cached results for prompts that were semantically similar, even when the wording was completely different.

Instead of hashing the raw prompt into a Redis key, we would generate an embedding for the incoming query and search for previously cached embeddings that were semantically close enough to reuse. If a sufficiently similar match existed, the system could skip large parts of the retrieval or inference pipeline.

The caching flow looked like this:

const embeddingKey = `embedding:${hash(query.toLowerCase().trim())}`;
const cachedString = await redis.get(embeddingKey);
let embedding;

if (cachedString) {
// Parse the stored string back into a workable array
embedding = JSON.parse(cachedString);
} else {
embedding = await createEmbedding(query);
await redis.set(
embeddingKey,
JSON.stringify(embedding),
"EX",
86400
);
}

This approach immediately solved one of Redis’ biggest weaknesses: wording variation. Queries that previously produced separate Redis entries could now reuse cached retrievals or responses if their embeddings were sufficiently close in vector space. This meant significantly higher cache hit rates for real conversational workloads.

The payoff from semantic caching was immediate, especially for conversational traffic where users ask the same question ten different ways. A string-matching setup like Redis misses completely if a user changes a single word. With a vector database, prompts like “How can I speed up vector search?” and “Best ways to optimize semantic retrieval performance?” resolve to the same underlying intent, allowing us to recycle the same cached response, context chunks, or embeddings seamlessly.

We also saw potential cost reductions beyond response caching alone. Embedding reuse became more effective because semantically similar prompts often generated nearly identical retrieval behavior. Retrieval outputs themselves could also be reused across related queries, reducing load on the vector search layer and decreasing the number of repeated context assembly operations.

Semantic caching appeared especially promising for RAG systems, AI copilots, internal knowledge assistants, search-heavy AI applications, and conversational agents with repeated intent patterns because these workloads frequently involve semantically similar queries that can benefit from intelligent cache reuse.

At first, the results were promising. Semantic caching immediately optimized our hit rates and reduced redundant retrieval calls across varied prompts. However, scaling this layout under full production traffic quickly exposed a brand new category of latency and performance constraints.

Where vector DBs started breaking

The advantages of semantic caching were real, but so were the new problems it introduced. As traffic increased and vector indexes grew larger, the system began to develop issues that were harder to predict and debug than the Redis problems we had dealt with earlier.

The first major issue was latency instability. Unlike Redis, which provided highly predictable exact-match lookups, vector similarity search performance degraded unpredictably under load, query complexity, metadata filters, and concurrency levels. Under heavy workloads, some semantic cache lookups became significantly slower than expected, especially when the system searched across millions of embeddings.

A typical semantic lookup now involves multiple operations:

• Generating an embedding
• Running ANN similarity search
• Evaluating similarity thresholds
• Retrieving metadata and cached responses

Even before LLM inference occurred, the cache layer itself was becoming computationally expensive.

False-positive matches also became a serious problem. Two prompts could appear semantically similar in vector space yet require very different responses in practice. This occasionally caused cached responses to be reused in contexts where they were only partially relevant or subtly incorrect. For context, a query about optimizing vector search for low-latency chat applications might accidentally reuse cached retrievals intended for large-scale offline analytics systems simply because the embeddings appeared highly similar.

The hardest part was tuning similarity thresholds correctly.

const SIMILARITY_THRESHOLD = parseFloat(process.env.CACHE_SIMILARITY_THRESHOLD || "0.93");
const cacheKey = `llm_cache:${hash(userQuery)}`;

try {
const cachedResponse = await redis.get(cacheKey);
if (cachedResponse) {
return JSON.parse(cachedResponse);
}
} catch (cacheError) {
console.warn("Cache read failed, falling back to semantic search:", cacheError);
}

const embedding = await createEmbedding(userQuery);

const result = await vectorIndex.query({
vector: embedding,
topK: 3,
});

if (result.matches && result.matches.length > 0) {
const bestMatch = result.matches[0];

if (bestMatch.score >= SIMILARITY_THRESHOLD) {
return bestMatch.metadata?.cachedResponse ?? bestMatch.cachedResponse;
}
}

const response = await generateLLMResponse(result.matches);

try {
await redis.set(cacheKey, JSON.stringify(response), "EX", 3600);
} catch (cacheError) {
console.error("Failed to write response to cache:", cacheError);
}

return response;

Another benefit was the reduction of repeated embedding and retrieval workloads. Since semantically related prompts often produced highly similar retrieval patterns, the infrastructure handled fewer redundant searches overall. This became especially valuable under high concurrency, where reducing repeated vector searches significantly lowered infrastructure load.

Semantic caching also improved the user experience in some scenarios. Similar prompts tended to receive more consistent responses because they reused previously validated retrieval contexts rather than generating entirely fresh retrieval paths every time.

For a while, vector database caching looked like the clear evolution of AI caching systems. The architecture appeared smarter, more adaptive, and better aligned with how humans naturally communicate.

But the more we pushed into production, the more we started discovering the hidden costs of semantic similarity itself. Finding a stable similarity threshold proved incredibly fragile. Low thresholds maximized cache hits at the expense of precision, while tight thresholds neutralized false positives but destroyed cache utility. This sensitivity turned threshold management into a major operational bottleneck, directly impacting downstream inference costs and system accuracy.

if (match.score >= 0.90) {
return match.cachedResponse;
}

Embedding drift introduced another long-term challenge. As embedding models changed over time, older cached vectors gradually became less compatible with newer embeddings. Semantic relationships shifted, reducing retrieval accuracy and forcing expensive re-indexing operations across the cache layer.

Operational complexity also increased substantially compared to Redis. Maintaining vector indexes required tuning ANN algorithms, balancing shards, handling index rebuilds, and monitoring recall accuracy as workloads changed. The infrastructure became harder to reason about because semantic correctness was no longer deterministic.

We also discovered that semantic caching consumed resources differently than traditional caching systems. Even cache hits still required embedding generation and vector search operations before matches could be identified. Unlike Redis, where a successful lookup was nearly free, semantic cache hits still carried noticeable computational overhead.

In production, the system began to reveal an uncomfortable reality. Semantic caching solved the exact-match problem, but it introduced an entirely new category of latency, accuracy, and operational challenges that traditional caching systems rarely encounter.

Redis vs Vector DB: The real production trade-offs

Once both systems had been running in production long enough, the comparison between Redis and vector database caching became much clearer. Neither technology was universally better. Each one is optimized for a completely different type of workload, and the real trade-offs only become visible under large-scale AI traffic.

Redis dominated in raw speed and predictability. Exact-match lookups were extremely fast, operationally simple, and relatively easy to scale. If a query had already been seen before in the exact same form, Redis almost always delivered the lowest possible latency. Cache hits are often completed in milliseconds with minimal computational overhead.

Operationally, Redis was also easier to maintain. Debugging cache misses was straightforward because the behavior was deterministic. A cache key either existed or it did not. Infrastructure teams already understood replication, sharding, persistence, and monitoring strategies because Redis has been battle-tested for years across traditional distributed systems.

Its weakness was semantic rigidity. Humans don’t write identical strings. If someone adds a typo or changes a single word, Redis drops the ball and treats it as a brand-new cache entry. Vector databases fixed that exact-match rigidity by letting us cache responses based on meaning. On paper, it’s a dream for RAG pipelines and copilots. In production, though, you realize you’re just paying a different tax. Vector lookups have actual computational weight. Unlike Redis, where an O(1) RAM read takes single-digit milliseconds, checking a semantic cache means you’re stuck waiting on an embedding model call and a graph traversal step just to see if you have a match.

“Unlike Redis, where an O(1) RAM read takes single-digit milliseconds, checking a semantic cache means you’re stuck waiting on an embedding model call and a graph traversal step.”

Worse, you lose determinism. With Redis, a key is either there or it isn’t. A vector cache forces you to manage a fuzzy threshold where the system occasionally treats two completely distinct user intents as “similar enough,” blindly serving bad data. Our infrastructure bill transformed too: we went from a system that was heavily memory-bound on RAM to one that aggressively chewed through compute just to optimize and search indexes.

The hybrid architecture that finally worked

After months of experimenting with both systems independently, we eventually stopped trying to choose between them and began using Redis and vector databases as complementary layers rather than competing technologies.

The final architecture used a multi-layer caching strategy. Redis handled ultra-fast exact-match caching for highly repetitive requests, session state, temporary conversation memory, and hot-path retrievals. The vector database handled semantic reuse for prompts that were conceptually similar but not textually identical.

The request flow became layered. The system first checked Redis for an exact-match cache hit, and if that failed, it moved on to a semantic vector cache lookup. If the semantic lookup also missed, the request proceeded through the full pipeline of retrieval and inference, after which the resulting output was stored back into both cache layers where appropriate.

A simplified hybrid flow looked like this:

try {
const exactCached = await redis.get(cacheKey);
if (exactCached) {
return JSON.parse(exactCached);
}
} catch (cacheError) {
console.warn("Redis read failed, proceeding to semantic search:", cacheError);
}

const embedding = await createEmbedding(query);
const semanticMatch = await vectorIndex.query({
vector: embedding,
topK: 1,
});
const SIMILARITY_THRESHOLD = parseFloat(process.env.SEMANTIC_CACHE_THRESHOLD || "0.93");

const bestMatch = semanticMatch.matches[0];

if (bestMatch?.score >= SIMILARITY_THRESHOLD) {
  return bestMatch.metadata?.response;
}


const response = await generateLLMResponse(query);
try {
await redis.set(cacheKey, JSON.stringify(response), "EX", 3600);
} catch (cacheError) {
console.error("Failed to write exact match to Redis:", cacheError);
}
await vectorIndex.upsert([
{
id: crypto.randomUUID(), // Most vector DBs require a unique ID
vector: embedding,
metadata: {
response,
},
},
]);
} catch (vectorError) {
console.error("Failed to upsert semantic cache to vector index:", vectorError);
}
return response;

This hybrid approach solved several problems simultaneously. Redis continued to handle the lowest-latency exact cache hits, protecting the infrastructure during traffic spikes and repetitive workloads. The vector cache improved semantic reuse without forcing every request through expensive ANN searches unnecessarily.

The layered design also improved reliability. Even if vector search latency increased temporarily, Redis still absorbed a large share of repeated traffic. Likewise, if exact-match cache hit rates dropped, semantic caching still recovered some of the lost reuse efficiency.

The architecture became easier to optimize because each layer had a clearly defined responsibility: Redis optimized speed, the vector database optimized semantic understanding, and the inference layer handled only true cache misses.

Over time, we also became more selective about what entered the semantic cache. Not every response benefited from semantic reuse, especially highly dynamic or context-sensitive outputs. Restricting semantic caching to stable retrieval patterns improved both precision and infrastructure efficiency.

Production lessons we learned

The biggest lesson was that AI caching behaves fundamentally differently from traditional application caching. Human language introduces semantic variation that exact-match systems struggle with, but semantic systems introduce probabilistic complexity that exact-match systems avoid.

In production, we learned that Redis and vector databases are not competing solutions but tools optimized for different layers of AI caching. Redis excels at fast, deterministic exact-match retrievals, while vector databases are better suited for semantic reuse in variable, intent-driven workloads. The most stable systems are not built on choosing one over the other, but on combining both in a layered architecture that matches the nature of AI traffic.

Ultimately, there is no “best” caching system for AI workloads. Redis and vector databases solve fundamentally different problems, and treating them as interchangeable leads to architectural inefficiencies at scale. 

Redis delivers speed and predictability for exact-match scenarios, while vector DB caching enables semantic reuse where user intent matters more than exact wording. In real production systems, the most reliable approach is not replacement but combination.

The post Why smarter AI caching sometimes makes everything slower appeared first on The New Stack.

Modern businesses are in a constant, uphill battle against what to do with unstructured data: PDFs, contracts, scanned images, customer call recordings, meeting videos, and more. Traditional document automation workflows that rely heavily on template-based extraction or rigid rules used to make sense. But document formats have changed; they’re diverse and don’t fit standard formats, making costly, brittle, traditional systems a relic of the past. 

“Modern businesses are in a constant, uphill battle against what to do with unstructured data.”

Enterprises demand faster, more accurate processing, which raises the question: How can we reliably turn messy, multimodal content into structured, actionable insights without a mountain of manual effort?

That’s where Amazon Bedrock Data Automation (BDA) comes in.

What is Amazon Bedrock Data Automation (BDA)?

Amazon Bedrock Data Automation (BDA) is a generative AI-powered, fully managed service on Amazon Web Services for end-to-end document and media automation. It enables users to automate the extraction, classification, and transformation of unstructured content across modalities such as documents, images, audio, and video.

“At its core are Foundation models which enable intelligent extraction and understanding of content.”

At its core are Foundation models (FMs) which enable intelligent extraction and understanding of content. It allows users to configure standard output for common use cases, or even define custom extraction logic using blueprints tailored to your business. BDA is designed for scalability, accuracy, and auditability, making it ideal for enterprise workflows.

Walk-through: creating a project, standard output & custom output using blueprints

1. Create a project via console

In the Amazon Bedrock Console, navigate to Data Automation → Create Project.

Enter the name of the project:

2. Standard output:

Standard output gives you the model’s default, unstructured response (text, image, audio, or video) directly from the Data Automation pipeline.

In standard output, each modality has its own options for what is needed as an output. 

Document:

Image & Video:

Audio:

Now let’s test Document Modality for Standard Output:

First, click on “Test” in the upper right corner.

Next, select the document from the system, sample, or S3 and choose the modality from the dropdown menu. 

Click on the “Generate results” button:

After processing, it will show the summary and content of the document:

Custom output (blueprints):

Custom output lets you define a structured, predictable format using blueprints, which ensures the output matches your exact schema, fields, and business rules.

Let’s test custom output using blueprints for the same document:

Navigate to “Custom output” and click on “Add Blueprint”:

From here, two options will appear. You can either use LLM power to generate the blueprint (where it inspects the document), or you can choose to enter field names, instructions, and other information manually.

Below is a blueprint generated by LLM which has pulled all possible fields and tables from the document:

It has extracted the information using the blueprint as demonstrated below, including the  Field name, Instruction, and Results:

It also provides the Extraction type (which can be Explicit or Inferred), Confidence percentage, and other relevant information.

Additionally, it can extract information in the form of a table, such as an account summary or transaction information:

Code examples

Amazon Bedrock Data Automation (BDA) Utility Module
Description:
    Helper functions to create BDA projects, blueprints, invoke jobs,
    monitor job status, and fetch results.

import boto3
import time
import json
import botocore

class BedrockDataAutomation:
    def __init__(self, region="us-east-1"):
        self.bda = boto3.client("bedrock-data-automation", region_name=region)
        self.runtime = boto3.client("bedrock-data-automation-runtime", region_name=region)

    # ------------------------------------------------------------
    # BLUEPRINT OPERATIONS
    # ------------------------------------------------------------
    def create_blueprint(self, name, schema, description="", stage="LIVE"):
        """
        Create a BDA Custom Output Blueprint from a JSON schema.
        """
        print(f"Creating blueprint: {name}")

        response = self.bda.create_blueprint(
            blueprintName=name,
            blueprintStage=stage,
            type="DOCUMENT",
            schema=json.dumps(schema)
        )
        return response["blueprint"]["blueprintArn"]

    # ------------------------------------------------------------
    # PROJECT OPERATIONS
    # ------------------------------------------------------------
    def create_project(self, name, description, standard_output_config, custom_output_config=None):
        """
        Create a BDA Project with Standard or Custom Output.
        """
        print(f"Creating project: {name}")

        response = self.bda.create_data_automation_project(
            projectName=name,
            projectDescription=description,
            projectStage="LIVE",
            standardOutputConfiguration=standard_output_config,
            customOutputConfiguration=custom_output_config or {}
        )
        return response["projectArn"]

    # ------------------------------------------------------------
    # INVOCATION OPERATIONS
    # ------------------------------------------------------------
    def invoke_project(self, project_arn, profile_arn, input_s3_uri, output_s3_uri, blueprints=None):
        """
        Invoke a BDA project using async invocation.
        """
        print(f"Invoking project: {project_arn}")

        kwargs = {
"inputConfiguration": {"s3Uri": input_s3_uri},
"outputConfiguration": {"s3Uri": output_s3_uri},
"dataAutomationConfiguration": {
"dataAutomationProjectArn": project_arn,
"stage": "DEVELOPMENT"
},
"dataAutomationProfileArn": profile_arn
}

        if blueprints:
kwargs["blueprints"] = blueprints

        response = self.runtime.invoke_data_automation_async(**kwargs)
       invocation_arn = response["invocationArn"]

       print("Invocation ARN:", invocation_arn)
       return invocation_arn

    # ------------------------------------------------------------
    # JOB STATUS POLLING
    # ------------------------------------------------------------
    def wait_for_job(self, invocation_arn, poll_interval=10):
        """
        Poll until job finishes.
        Returns final status object.
        """
        print("Polling job:", invocation_arn)

        while True:
            try:
                resp = self.runtime.get_data_automation_status(
                    invocationArn=invocation_arn
                )
            except Exception as e:
                print("Error fetching status:", e)
                raise

            status = resp["status"]
            print(f"Status: {status}")

            if status in ("SUCCEEDED", "FAILED", "CANCELLED"):
                return resp

            time.sleep(poll_interval)



# --------------------------------------------------------------------
# EXAMPLE USAGE
# --------------------------------------------------------------------
if __name__ == "__main__":
    bda = BedrockDataAutomation(region="us-east-1")

    # 1. Create Blueprint
    blueprint_schema = {
        "type": "object",
        "properties": {
            "account_holder": {"type": "string"},
            "balance": {"type": "string"},
            "transactions": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "date": {"type": "string"},
                        "description": {"type": "string"},
                        "amount": {"type": "string"}
                    }
                }
            }
        },
        "required": ["account_holder", "transactions"]
    }

    blueprint_arn = bda.create_blueprint(
        name="BankStatementBlueprint",
        schema=blueprint_schema,
        description="Extract fields from bank statements."
    )

    # 2. Create Standard Output Config
    standard_config = {
        "document": {
            "extraction": {
                "granularity": {"types": ["PAGE", "LINE"]},
                "boundingBox": {"state": "ENABLED"}
            },
            "outputFormat": {
                "textFormat": {"types": ["PLAIN_TEXT", "CSV"]}
            }
        }
    }

    # 3. Create Project with Custom Blueprint
    project_arn = bda.create_project(
        name="BankStatementProject",
        description="Process PDF bank statements",
        standard_output_config=standard_config,
        custom_output_config={
            "blueprints": [
                {
                    "blueprintArn": blueprint_arn,
                    "blueprintStage": "DEVELOPMENT",
                    "blueprintVersion": "1"
                }
            ]
        }
    )

    # 4. Invoke the project
    # Ensure you replace <ACCOUNT_ID> with your actual AWS Account ID
    profile_arn = "arn:aws:bedrock:us-east-1:<ACCOUNT_ID>:data-automation-profile/us.data-automation-v1"
    invocation_arn = bda.invoke_project(
        project_arn=project_arn,
         profile_arn=profile_arn,
        input_s3_uri="s3://your-bucket/input/statement.pdf",
        output_s3_uri="s3://your-bucket/output/",
        blueprints=[
            {
                "blueprintArn": blueprint_arn,
                "version": "1",
                "stage": "DEVELOPMENT"
            }
        ]
    )

    # 5. Poll job status
    final_status = bda.wait_for_job(invocation_arn)
    print("Final status:", json.dumps(final_status, indent=4))

Types of document blueprints

When processing documents, BDA supports five core automation types:

1. Classification: invoice, bank statement, ID card, contract, HR letter, etc.

2. Extraction: Extract entities, fields, tables, metadata.

• Example: From a bank statement → Date, Description, Amount, Balance.

3. Transformation: Modify or restructure data.

• Example: Convert Home Address into separate fields -> street, city, ZIP code, etc.

4. Normalization: Standardize data values.

• Example: Convert multiple date formats (MM/DD/YYYY → YYYY-MM-DD).

5. Validation: Validate extracted fields against rules.

• Example: Amount must be numeric; dates must match the format; balances must reconcile.

Use cases that illustrate business value

Real-world scenarios where BDA provides significant ROI include:

• Financial Services: Automate processing of bank statements, invoices, and loan applications, reducing manual labor and speeding up reconciliation or underwriting.
  
• Insurance: Ingest and extract data from claims forms, medical reports, and damaged-asset photos.
  
• HR / Legal: Process resumes, contracts, and offer letters; extract structured data, including skills, clauses, salaries, and parties.
  
• Customer Support: Transcribe and summarize calls, extract intent and sentiment, and feed those insights into CRM or case systems.
  
• Security & Compliance: Analyze CCTV footage or meeting recordings to detect key actions, summarize context, and flag compliance issues.

BDA proves itself flexible and powerful, as it supports both standard outputs for basic workflows and fine-tuned custom schemas via blueprints. It is scalable and robust, with projects that enable batch processing and versions (development vs. live) for safe testing. It’s also audit-friendly, providing structured fields with types, normalization rules, and validation logic. 

“Compared with rule-based systems, foundation models achieve better semantic extraction across the board.”

A true key benefit is that BDA is multimodal across formats. Users can use the BDA framework to process documents, images, audio, and video. And, best of all, it’s highly accurate. Compared with rule-based systems, foundation models achieve better semantic extraction across the board. 

Amazon Bedrock Data Automation empowers businesses to transform unstructured, multimodal content into structured, trustworthy, and actionable data. With minimal setup, highly customizable blueprints, and a scalable project-based architecture, BDA helps organizations reduce manual workload and unlock insights faster.

The post Template-based data extraction is dead. Here’s what comes next. appeared first on The New Stack.