AI agents are multiplying as corporations adopt the technology in record numbers. Smarter underlying models, better tool use, and improved multi-agent collaboration have pushed agents to evolve beyond impressive demos into practical technology that companies marshal in production environments. But the job’s not finished. 

As companies deploy more agents, more often, and against longer tasks, the plumbing that provides their AI ephemera with the required information is buckling.

Here’s the problem: AI agents are sending waves of queries against company data, creating concurrency issues and exposing just how difficult it can be to ensure a company’s AI-legible information is fresh, served only when relevant, and quickly available.

Join the live conversation: On September 24 at 12 p.m. Eastern/9 a.m. Pacific, Whit Walters, Field CTO and Lead Analyst at GigaOm and author of Defeating the Integration Tax report, joins Bonnie Chase, Director of Product Marketing at Vespa.ai, to discuss what happens when retrieval architecture meets that workload.

And crucially, they will explore in this live conversation what changes when a team rebuilds it as a unified layer instead of a fragmented one.

Register for our free event on September 24

You might be asking yourself: How has this problem not been solved yet? Google famously handles tens of thousands of search queries every second; how difficult can it be to serve agents the information that they need when we’ve solved the human version of the same problem? It’s no small challenge, and it’s why retrieval engineering is a labor category you’ll hear more about in coming quarters.

So, why is the problem worse with AI? Agents don’t ask a single question. They may retrieve data, reason against it, and then go back for more context. That doesn’t sound too complicated, until we recall that companies often stitch multiple systems together to provide their agents with required information. In practice, that means fusing vector databases, ranking tools, and serving layers into a single hybrid retrieval system that serves ever more agentic queries.

Worse, when several agents ping the same cobbled-together architecture at once, relevance drift becomes a real issue. You might do all the work to get your company or team up and running with agents, only to see the effort fail because of stale data, generic answers, or even truncated results as retrieval plumbing stumbles.

Your AI agents can’t scale successfully if they get dumber the more agents you deploy. So join the conversation on September 24, where we’ll break down how you can solve your retrieval engineering woes.

What you’ll take away:

• Why agent workloads create a fundamentally different retrieval challenge than added concurrency alone
• The specific failure modes at agent scale — latency stacking, stale context, relevance drift
• Why fragmented retrieval stacks amplify those failures
• What a unified retrieval architecture looks like in practice

The post Want to scale AI agents without breaking anything? Retrieval engineering is the answer. appeared first on The New Stack.

Every product team is chasing the same moment: The user opens a page and thinks, this understands me.

A shopper who loves floral prints should see more floral prints. A user who follows local politics should open their app to see news about local politics. A job candidate who keeps clicking remote roles should not keep getting shown in-office jobs.

That is not a niche feature anymore. It is the baseline expectation. Users decide quickly whether a product system understands them, and they rarely care whether the failure came from search, recommendations, merchandising rules, or stale data.

Here is the uncomfortable truth: Most teams do not have a personalization quality problem. They have a personalization architecture problem.

Personalization is not a widget bolted onto search. It is a ranking decision. The system has to decide, for this user and this request, what deserves the next slot. That means weighing the user, the item, the context, and the business goal at the same time. In many stacks, the ranking layer is the one place that cannot see all of those signals together.

The hard part is not collecting signals. The hard part is combining them while the user is still there.

Why personalization is hard in the first place

To put the right item in the right slot, a system has to understand several things at once:

• Intent: What is the user asking for right now?
• Item quality: What does each candidate actually contain or represent?
• User history: What has this person clicked, bought, read, watched, or ignored?
• Availability: Is the item in stock, fresh, nearby, legal to show, or ready to ship?
• Business priority: What should the business promote, protect, or de-emphasize?
  

Those signals often disagree. The most relevant item may not be the most profitable. The most profitable item may be out of stock. The user may say “running shoes,” but their behavior says “trail running, wide fit, under $120.”

They also move on different clocks. Product attributes change slowly. Inventory and price can move throughout the day. Preferences shift with every click. External context — weather, breaking news, a championship game, a cultural moment — can matter without warning.

Personalization means folding all of that into one ordered list, on every request, in milliseconds. The signals themselves are not the bottleneck. Query-time ranking is.

The usual stack makes the problem harder

Most personalization systems are assembled from tools that were each designed for one slice of relevance.

Keyword search engines are excellent at lexical matching. They are good when the query language and catalog language line up. But shoppers, readers, and job seekers rarely speak in neat index terms. You indexed “athletic performance running footwear”; they typed “running shoes.” Synonym rules can help, but they do not scale gracefully across long-tail language, changing catalogs, and new user behavior.

Vector databases start from the opposite side. They are good at semantic similarity: “Find me things like this.” That is powerful, but nearest-neighbor search is not the same thing as personalization. Real ranking has to blend semantic similarity with live behavior, stock, price, margin, freshness, eligibility, and business rules.

Re-rankers, recommendation services, feature stores, and rule engines are usually added to glue everything together. That is where fragmentation creeps in.

When retrieval and ranking live in separate systems, the ranker often works from a partial, stale, or precomputed view of the world. Click history, session context, and the user’s live preference vector arrive too late. Business rules become filters or overrides instead of ranking signals. Fresh inventory or price changes require coordination across multiple systems.

Every hand-off adds latency. Every boundary creates another place for signals to drift. Every “quick rule” becomes another hard constraint that can accidentally turn “show the closest match” into “show nothing.”

“Every hand-off adds latency. Every boundary creates another place for signals to drift.”

The deeper issue is a timing assumption. Many architectures were built around offline ranking: process the catalog, compute scores in a batch job, and serve those scores until the next rebuild. That works when preferences are stable. It breaks when the most valuable signal is the click that happened two seconds ago.

What changes when ranking happens in one real-time pipeline

A real-time personalization architecture treats retrieval, ranking, and inference as one serving problem.

That is the core idea behind Vespa’s approach: Text search, vector similarity, structured filtering, ranking expressions, tensor computation, and model inference can live inside one query pipeline. Instead of retrieving somewhere, enriching somewhere else, and ranking at the end, the system can rank with the relevant signals while it is still deciding what to return.

That architectural choice changes the shape of the problem.

1. Retrieval is hybrid from the start

Lexical search, semantic search, and structured filtering can run together instead of being reconciled after the fact. A product query can combine text, embeddings, filters, session behavior, and item attributes in one request.

That matters because personalization is rarely one signal. The user’s query still matters. So does semantic similarity. So do category, availability, price, and business constraints. Hybrid retrieval keeps those signals in play before ranking starts.

2. Ranking can express the actual objective

A personalization score should not be trapped inside one similarity function. It should be a formula that reflects the product’s goals.

That formula might combine BM25, vector similarity, user affinity, stock level, margin, popularity, discount depth, freshness, rating, distance, or a weather term. Some of those signals need normalization first. Some should matter only for certain categories or users. Some should be tested as weights.

The important part is that they are all terms in the same ranking expression, not scattered across services.

A simplified version might look like this:

final_score =
    0.30 * lexical_relevance +
    0.25 * semantic_similarity +
    0.25 * user_affinity +
    0.10 * availability +
    0.10 * business_priority

In production, the formula can be more nuanced. But the principle is simple: personalization, relevance, and business logic belong in the same scoring decision.

3. Model inference can run where the data lives

Some signals should come from learned models rather than hand-tuned rules: propensity to buy, churn risk, quality prediction, fraud risk, query classification, or a learned-to-rank model.

When inference runs in the serving path, those model outputs can become ranking features instead of delayed batch scores. That reduces the need to ship data to a separate inference service, wait for a response, and stitch the score back into ranking.

4. Updates become immediately useful

“Real time” should not mean “after the next index rebuild.” If inventory changes, stock should be rankable immediately. If a user clicks two yellow dresses, “yellow” should matter on the next request. If a merchandising team adjusts a ranking weight if the weight is exposed as a query-time input, the experiment should start producing useful feedback right away.

That is the difference between personalization as a nightly job and personalization as a live ranking decision.

Tensors make the personalization concrete

The most useful mental model is simple: represent the user and the item in the same feature space, then rank by how well they match.

In Vespa, tensors make that practical. A tensor can be a scalar, a dense vector, a sparse map of feature-weight pairs, a matrix, or a more complex structure. That means the same framework can represent semantic embeddings, product attributes, user preferences, business objectives, and model features.

For example, each item can carry a sparse feature tensor:

{
  "floral": 0.90,
  "yellow": 0.70,
  "short_sleeve": 0.80,
  "crew_neck": 0.65
}

Each user can carry a tensor with the same feature names:

{
  "floral": 1.00,
  "yellow": 0.37,
  "short_sleeve": 0.33,
  "crew_neck": 0.31
}

Because the two tensors share a shape, personalization becomes a dot product: multiply matching features, sum the result, and use that score inside ranking.

In a Vespa rank profile, the core expression is compact:

# schema: item attributes stored as a sparse tensor
field item_features type tensor<float>(feature{}) {
    indexing: attribute | summary
}
 
# rank profile: the user's live preferences arrive as a query tensor
rank-profile personalized {
    inputs {
        query(user_features) tensor<float>(feature{})
    }
    first-phase {
        expression: sum(query(user_features) * attribute(item_features))
    }
}

That one expression is not the whole ranking function. It is the personalization term. BM25, vector similarity, stock, margin, freshness, distance, or a model score can be added as other terms with their own weights.

The user tensor is where real-time behavior becomes powerful. Click a floral item, and the “floral” weight rises. Click two yellow items, and “yellow” rises; the application feeds click events into the user profile. The next query can use those updated preferences immediately, without waiting for a nightly profile build.

Business goals stop fighting personalization

In fragmented stacks, business rules often become blunt instruments: boost this category, hide that brand, force these items to the top, filter these out. That can satisfy a short-term merchandising goal while damaging relevance.

When business logic is part of the ranking expression, it can be more subtle. You can boost overstocked inventory without ignoring intent. Promote umbrellas when rain is forecast without turning every search into an umbrella search. Give new sellers a small exploration boost. Prioritize destocking before a new product line launches. Surface team merchandise during a championship run.

“When business logic is part of the ranking expression, the user still gets relevant results. The business still influences outcomes.”

The user still gets relevant results. The business still influences outcomes. The difference is that both are expressed as ranking signals instead of competing systems.

That also makes experimentation easier. A merchandising or growth team can test weights, traffic splits, and ranking profiles without asking engineering to rewrite the whole pipeline. Relevance becomes a controllable growth lever rather than a fragile side effect.

The same pattern applies beyond commerce

The examples above are easy to picture in apparel, but the architecture is not commerce-specific. Personalization is the same ranking problem in many products:

• Content feeds: Blend topic affinity, freshness, engagement, creator quality, and business rules.
• News: Rank by reading history, topic interest, locality, freshness, and source diversity.
• Jobs: Match candidate preferences such as remote work, seniority, compensation, location, and tech stack against role attributes.
• Geo search: Treat distance as one normalized ranking term alongside relevance, quality, and preference.
• Video and audio: Combine embeddings, viewing history, metadata, freshness, and learned ranking models.

Different domains need different features. The architecture pattern is the same: retrieve candidates, rank with the signals that matter, update those signals as behavior changes, and keep the decision close to the data.

Scale doesn’t have to be the trade-off

The natural concern is that a more expressive ranking system must be slower. In practice, that does not have to be true.

Vespa was built for large-scale serving from the beginning: billions of documents, high query volume, and low-latency ranking. The reason this works is multi-stage ranking. The system does not run the most expensive logic across every possible result. Instead, it uses a fast first phase to narrow the candidate set, then applies more precise ranking to the smaller group that remains.

For example, a cheap first phase narrows a huge candidate set. Then, once the candidate set is smaller, Vespa can apply full-precision scoring, richer tensor operations, business logic, and model inference where they matter most.

The result is a practical balance: speed across the full corpus, accuracy in the final ranking, and enough flexibility to personalize each query without turning the serving stack into a chain of fragile services.

What’s next

Personalization is not failing because teams lack data. Most teams already have plenty of signals: query intent, clicks, product attributes, inventory, margin, freshness, location, and business priorities. The harder problem is that those signals often live in different systems, move at different speeds, and arrive too late to influence the final ranking decision.

That is why personalization should be treated as a ranking problem. When retrieval, ranking, personalization, and business logic are split across separate systems, the ranker is forced to work with stale or incomplete context. The user moves faster than the architecture can respond. Every new signal becomes another integration project.

A unified real-time ranking pipeline changes that. User behavior, item attributes, semantic similarity, lexical relevance, inventory, and business goals can all become parts of the same scoring function. Tensors make those signals directly comparable and usable at query time. Instead of bolting personalization onto the end of the system, personalization becomes part of the decision the engine makes for every query.

The goal is simple: rank each result with the best context available, at the moment the user asks. That is when personalization stops feeling like a feature and starts feeling like relevance.

The post Personalization is a ranking problem — architecture makes it work appeared first on The New Stack.

Public AI assistants have become so commonplace that software vendors are increasingly adding AI search, conversational experiences, and AI agents to their own applications. From eCommerce and customer support to enterprise software, AI is rapidly becoming the primary interface to many applications.

Companies that build products around proprietary information are particularly well positioned to benefit from this shift. Whether they provide financial intelligence, market intelligence, legal research, scientific publishing, or business information, their products help professionals make better decisions by transforming trusted information into actionable insight. 

AI allows these organizations to deliver that expertise through entirely new user experiences. Increasingly, they compete not only on the quality of their proprietary information, but on how intelligently they retrieve, understand, and transform it into customer value.

“Increasingly, they compete not only on the quality of their proprietary information, but on how intelligently they retrieve, understand, and transform it into customer value.”

A new competitive battleground is emerging. As AI becomes the primary interface to proprietary knowledge, the ability to retrieve, verify, rank, and assemble information is becoming almost as important as the proprietary information itself. 

Much of the industry’s attention has focused on increasingly capable language models, but those models are only as effective as the context they receive. Designing retrieval workflows that consistently deliver trusted, relevant, and up-to-date information is rapidly becoming one of the defining engineering challenges for AI-native applications.

Retrieval engineering: optimizing the workflow

For decades, search engineering has focused on helping people find the right information. Whether searching a website, a legal database, or a financial research platform, the challenge was to retrieve the most relevant results while balancing competing priorities such as relevance, latency, scalability, and cost. The search system’s job was to retrieve relevant information. The human’s job was to evaluate it.

AI fundamentally changes that role.

Instead of retrieving information for people to evaluate, retrieval systems increasingly assemble the context that large language models and AI agents use to investigate, reason, and act. Every retrieval decision now becomes part of an automated workflow in which relevance, freshness, latency, and trust directly influence the final answer.

“Prompt engineering influences how a language model reasons. Retrieval Engineering determines what it has to reason about.”

This shifts the engineering challenge away from individual technologies and towards the retrieval workflow itself. The goal is no longer simply finding relevant documents, but orchestrating retrieval, ranking, filtering, inference, and real-time updates so they work together efficiently. We believe this emerging discipline deserves its own name: Retrieval Engineering. Prompt engineering influences how a language model reasons. Retrieval Engineering determines what it has to reason about. Both matter, but as AI applications become increasingly autonomous, the quality of retrieval increasingly determines the quality of the outcome.

As AI applications evolve from conversational assistants to deep research systems and autonomous agents, optimizing workflows rather than individual components becomes increasingly important. A single user request may trigger dozens—or even hundreds—of retrieval operations before a response is generated.

The challenge isn’t vector search

Vector databases solved an important problem by making semantic retrieval practical at scale. But semantic retrieval is only one stage of a much larger workflow.

Production AI applications increasingly combine vector similarity with keyword search, structured filtering, business rules, personalization, machine-learned ranking, and real-time inference to assemble the context that language models depend on. The engineering challenge is no longer selecting the best retrieval technology—it is orchestrating increasingly sophisticated retrieval workflows that remain accurate, responsive, and cost-effective.

Many organizations address this by combining specialist technologies. A vector database provides semantic retrieval. A search engine handles lexical matching. Additional services provide reranking, personalization, and inference. This works well initially, but every additional component introduces another network hop, another operational dependency, and another source of latency. The problem is no longer vector search. It is engineering an efficient retrieval architecture.

From components to platforms

This shift is changing how retrieval infrastructure is designed. Instead of optimizing individual components in isolation, engineering teams increasingly need to optimize the retrieval workflow as a complete system—balancing retrieval quality, latency, freshness, scalability, and infrastructure cost.

“The problem is no longer vector search. It is engineering an efficient retrieval architecture.”

That is why AI Search Platforms are emerging. Rather than stitching together retrieval, ranking, inference, and serving from multiple independent services, they execute the workflow within a single distributed architecture. The optimization problem changes from integrating components to engineering the workflow itself.

AI has transformed the user interface. It is now transforming the retrieval infrastructure behind it. For organizations building applications around proprietary knowledge, the next competitive advantage will not come solely from larger language models or better embeddings. It will come from building retrieval workflows that consistently deliver trusted, relevant, and timely context at scale.

Retrieval Engineering is rapidly becoming one of the disciplines defining the next generation of AI-native applications.

If you’re interested in exploring these ideas in more depth—including Retrieval Engineering, AI Search Platforms, and the architectural patterns behind AI-native information platforms—we cover them in our ebook, Building AI-Native Information Platforms.

The post Is retrieval engineering becoming AI’s next bottleneck? appeared first on The New Stack.

Agentic systems usually have two jobs: Build context, then use that context to produce an answer or action.

Many failures that look like LLM problems start in the context-building step. The answer the LLM gives is limited by the context it was given, or it finds through tool calls. If the agent model cannot find the right sources, then improving the generation model will not improve the overall system.

“Many failures that look like LLM problems start in the context-building step.”

A client, Specstory, wanted to give users the ability to ask questions from the agent’s history. For example, why a team chose Authlib for authentication and what alternatives they considered. The chatbot needs the right prior conversations, decisions, and tradeoffs from a large corpus of coding sessions. The model and system prompt help only after those chat turns have been retrieved and are in context.

If retrieval ranks implementation snippets above the discussion where the team weighed alternatives, the agent can still produce a confident answer. It may find code that imports Authlib and a few inline comments, then describe the decision based on implementation evidence rather than the actual trade-off discussion.

The same pattern showed up in an AnkiHub operator review in our private community. A request for help studying based on lecture slides only works if the agent’s tool calls retrieve the right flashcards. The hard part is not finding any related cards. A lecture on the function of the heart may match hundreds of cards. Ranking decides whether the core cards make it into context or whether the system has to raise top_k and flood the prompt.

The exact setup changes by product. The context-building step might use local search, semantic search, web or API calls, or database queries. It might be handled by an agent, a fixed workflow, or application code. The process stays the same: gather the right context, then generate from it.

For example, a coding agent runs rg, opens files, reads logs, and inspects tests before writing a patch. A research agent searches the web and internal notes before writing an answer. A study assistant searches deck facts and user context before suggesting what to learn next.

When context building fails, the symptoms look like generation failures.

Retrieval failures mimic generation bugs

A better model helps with reasoning and writing, but it cannot give a better answer without the right context.

“A better model helps with reasoning and writing, but it cannot give a better answer without the right context.”

The Mixedbread OfficeQA-Pro Eval shows the same pattern at the benchmark scale. OfficeQA-Pro uses 89,000 pages of financial documents, dense tables, scanned PDFs, and questions that require reasoning across documents. Giving Codex better search tools reduced tool calls and improved answer quality.

Plain-text tools like grep and rg work (ish) on flat code files. They do not work well when context lives in PDFs, tables, chat histories, multi-modal inputs, web results, and permissioned data. In those cases, the agent needs a retrieval that can combine exact terms, meaning, metadata, permissions, and ranking quality.

Retrieval needs traces and evals

Once retrieval enters the architecture, the next question is whether it finds the right information.

For that, you need traces and evals. For each retrieval step, the minimum trace is the input, the outputs, and a way to label whether each output was relevant.

For a coding agent using rg the input is the command, the output is the returned snippets, and the label says which snippets helped, which were noise, and which relevant files were missing.

For product retrieval, the step might be BM25, semantic search, hybrid search with reranking, a generated SQL query, or something else. Capture the query or arguments, the returned documents or chunks, and whether those results were helpful.

Trace each retrieval step by itself, then evaluate the full context-building pass. The local trace answers “Did this query return useful material?” The full trace answers “Did the system collect everything the model needed before generation?” If it did, failures are a generation problem. If not, it’s a retrieval problem.

You cannot know where the failure started or what to fix without traces.

Different failures need different fixes

“Improve retrieval” is too broad to be useful, as different problems require different solutions. If a relevant document is missing, the trace should show where it disappeared: query building, retrieval, filtering, ranking, or final context assembly.

The failed step, plus what the trace shows, tells you what change to make.

The right fix depends on what the system was trying to retrieve. A decision-history question requires the decision, the alternatives, and the chats in which the team worked through them. A study question depends on the lecture material, deck metadata, semantic matches, and the user’s study context.

The architecture

Once you trace individual retrieval calls, the full architecture has a simple shape: fan out to context-building tools, then fan in to generate the final output.

The retrieval layer might be a search engine, a vector database, an SQL query, a local file tool, a web search API, or a custom service. The pattern stays the same: build candidate context, narrow it, rank it, assemble it, then generate from it.

Give agents human search controls

Semantic search compares embeddings (numerical representations of meaning). It helps when wording differs, but most retrieval intents also depend on structured constraints. A meeting search box can use semantic search over transcripts and notes, but a useful interface also lets someone filter by person, date, project, and source. 

A finance search may need the latest filing, a specific quarter, or an official source in addition to the closest semantic match. In e-commerce, the best semantic match for “32×30 cargo pants” may be an out-of-stock product. The system still has to decide whether to hide it, return it with a backorder note, or show it so the user can check later. That product decision is a retrieval decision because it changes which candidates reach the agent.

In a chat interface, those controls are in the tool schema, query planner, or app logic. If an agent runs the search, it needs arguments for the same constraints a human would set with filters, sliders, tabs, and sort menus.

A retrieval system usually needs several controls working together:

Here, a chunk means a small piece of source content, and a candidate is a chunk returned by the first search step. Ranking is the scoring step that orders those candidates. Context assembly then selects which chunks and structured fields to include in the model prompt.

Better ranking improves precision, which means a larger share of the returned chunks is useful. If the relevant chunks are near the top, the system can pass fewer chunks to the model, use fewer tokens, reduce latency, and expose the model to less noise. If the right chunk is ranked 40th and the context only includes the top 10, the system behaves as if the retrieval missed it.

People and agents use the same basic search path: ask for results, inspect what comes back, and decide what to use. A person can skim ten search results, compare titles, snippets, dates, domains, and URLs, and decide whether the result set looks right. They can open the third result, ignore the rest, and search again with a better query. An agent usually receives a bounded set of returned documents and reasons from the context. If the right source falls below the cutoff, the agent may answer from partial context. To avoid this, the system has to retrieve more candidates, run more searches, or pass more evidence into the model.

A missed document can change what the agent searches for next. Suppose someone asks why the team chose Authlib. If the first search misses the transcript where the team compared Authlib with alternatives, the agent may search the codebase instead. It finds imports, callback handlers, tests, and maybe a comment. Then it asks follow-up questions about OAuth configuration. The context starts to look complete, but it supports the wrong answer. It explains how Authlib was used and why it makes sense in the codebase, not why the team chose it.

“The context starts to look complete, but it supports the wrong answer.”

But ranking cannot repair every search problem. If the agent failed to request the latest filing, a reranker may faithfully select an older document with a closer wording match. If the tool has no date_range, person, source_type, size, or in_stock argument, the model has to impose hard constraints in the search text and hope that retrieval infers them. A hard filter gives the model less to infer, making semantic search more reliable.

Scale changes the retrieval problem

Search systems already have tools for this: indexes, filters, facets, sorts, caching, bounded reranking, and freshness jobs. Agent systems need the same discipline.

A human might search, adjust a date filter, scan the first page, then search again. One agent request can do that many times in seconds: rewrite the query, run keyword and semantic search, inspect thin results, issue follow-up searches, fetch sources for citations, and ask for more context before answering. With many concurrent users or agents, the retrieval layer can become a bottleneck.

Humans often wait through a slow search if the result is good. Agent systems often turn slow and uncertain search into more work. When ranking is weak, teams compensate by raising top_k, running keyword and semantic searches in parallel, adding reranking, fetching more source documents, and passing larger evidence bundles to the model. That can improve answers, but it moves the cost into tokens, latency, and retrieval load. A better ranking lets the system return fewer, better candidates, rather than making every request carry a larger pile of possible evidence.

With a small corpus, you can still search comprehensively quickly and cheaply, even with fully agentic approaches. That’s what I recommend when you’re starting and don’t have much data. Don’t add complexity until you need it. But with millions or billions of chunks, every extra retrieval call, candidate, ranking pass, and returned token adds up quickly.

Multi-stage retrieval is the production shape

Most production systems should split retrieval into stages, even when the UI is a chat box.

Each stage leaves a different repair path. If the agent chose the wrong filters, changing the embedding model will not help. If the latest document was available but the tool never sorted by date, the fix belongs in the search arguments or retrieval API. If the right source was present but below the cutoff, the fix belongs in the ranking. If the right source came back but was dropped before generation, the bug is in context assembly.

As retrieval becomes a core part of agent architecture, teams increasingly need infrastructure that can combine semantic search, exact matching, filtering, ranking, and large-scale retrieval in a single system. Depending on requirements, this may involve search and retrieval platforms such as Vespa, Elastic, or Coveo, each of which supports different approaches to ranking, retrieval, and operational scale. 

The important point is not the specific technology choice, but recognizing that retrieval quality has become a first-class engineering concern. As agent workloads grow, retrieval systems are increasingly determining the accuracy, cost, latency, and reliability of the overall application.

The post Why retrieval quality is becoming the defining challenge in AI agent architecture appeared first on The New Stack.

A recent GigaOm CxO Decision Brief explores how AI retrieval architectures are evolving beyond flat vector databases as organizations combine semantic search, ranking, personalization, and machine learning inference in production systems.

Vector search changed the AI infrastructure landscape by making semantic retrieval practical at scale. By converting text, images, and user behavior into embeddings, organizations could move beyond exact keyword matching and retrieve information based on meaning. But production AI systems rarely stop at vector similarity.

A real-world query often requires multiple signals to be evaluated simultaneously. Semantic relevance may be one factor, but so are structured attributes, business rules, personalization signals, freshness, access controls, recommendation logic, and machine-learned ranking models. As organizations move from AI experimentation to production-scale applications, the challenge is no longer simply finding similar items. It is in combining all of the signals that matter while maintaining low latency and operational simplicity. This is where tensors are attracting increasing attention.

While vectors represent information as a single dimension of numerical values, tensors provide a more general framework for representing and operating on complex, multi-dimensional data structures. They offer more control in how relevance is computed, allowing dense embeddings, sparse features, metadata, and model outputs to be evaluated together within a unified retrieval and ranking process. For organizations building large-scale retrieval systems, this raises an important architectural question: is a flat vector store sufficient, or does the next generation of AI applications require something more expressive?

“Tensors provide a more general framework for representing and operating on complex, multi-dimensional data structures.”

A new GigaOm CxO Decision Brief, “The Tensor Advantage in AI Search,” explores this question in depth.

Among the findings:

• Production AI systems increasingly depend on combining semantic, lexical, behavioral, and business signals rather than relying on vector similarity alone.
• Architectural fragmentation between vector databases, search engines, rerankers, and feature stores introduces latency, operational complexity, and synchronization challenges that become more significant as workloads scale.
• Emerging retrieval models, including multi-vector and late-interaction approaches, place new demands on infrastructure that were not anticipated when first-generation vector databases were designed.
• Tensor-native architectures provide an alternative approach by treating multidimensional data structures as first-class citizens rather than forcing them into simpler vector abstractions.

The paper also examines the infrastructure, operational, and organizational implications of these architectural choices, including benchmark data, deployment considerations, and the trade-offs engineering leaders should evaluate when planning future AI retrieval systems.

“Retrieval is evolving from a nearest-neighbor problem into a ranking and decision-making problem.”

As AI applications become more sophisticated, retrieval is evolving from a nearest-neighbor problem into a ranking and decision-making problem. Understanding the role tensors play in that transition may be one of the most important architectural discussions facing engineering leaders today.

Download the GigaOm CxO Decision Brief to explore the findings in full.

The post Why AI retrieval and ranking need more than vector search appeared first on The New Stack.