Every time a Mastercard gets tapped, the network has less than a tenth of a second to judge how likely the purchase is to be fraudulent. It made that call across 175 billion transactions last year. Now the buyer on the other side of that judgment is starting to change, and Greg Ulrich, the company's chief AI and data officer, spelled out the consequence for the VB Transform 2026 audience in Menlo Park on July 14. "We've built a bunch of risk rules over time that were intended to stop a bot from transacting," Ulrich said. "Now we need to enable the bot to transact, so that requires a change to our risk framework and our risk rules."

Ulrich joined Mastercard eleven years ago when an analytics company he worked at was acquired, and said trust struck him from day one on the job. "It's what enables a merchant that's never met you to accept payment and ensure that they're going to get paid. It's what enables you as a consumer to transact and ensure that things are going to work out in a trusted, secure way. And if something goes wrong, there's a safe and secure path for a dispute and to resolve this," he said.

175 billion transactions, scored in under 100 milliseconds

He took the audience inside each of those calls. "When you tap your Mastercard to pay for a product or service, we're providing a score to that transaction," he said. "We have under 100 milliseconds to look at that and give a score from zero to 999 about how likely is that to be fraudulent or real. And we pass that on to the issuing bank."

Generative AI widened what that score can see. "Because we have new technology, we can bring in more data, we can bring in more context, and now we're finding that we can identify 300, 400% more fraudulent transactions at those high-risk bands," Ulrich said, without adding friction or false positives for consumers. The company's Safety Net system has stopped more than 70 billion fraudulent transactions, he told the audience, and Mastercard is building its own transformer model on its transaction data as a foundation for new safety, security, and personalization solutions. VentureBeat's Beyond the Pilot podcast took that production fraud stack apart in detail earlier this year.

A third of the services business already runs on AI

The business stakes reach past fraud. About 40% of Mastercard's company is now based on services, Ulrich said, including marketing services; fraud, safety and security; and business intelligence. "A third of those are predicated on AI, and those are growing at a much faster clip than everything else," he said.

One line he returned to all session went further. "What's going to enable AI to continue to scale is not the capabilities of the agents, it's how much we trust those agents to do on our behalf as a consumer, as a business, as a financial institution, or otherwise," he said.

Five layers stand between agents and the network

Agentic commerce changes the object being secured. "Instead of a single atomic transaction where I say go buy something, I'm effectively delegating authority, or a consumer's delegating authority, a business is delegating authority," Ulrich said. "And when that happens, it's a much more complicated transaction." Trust, in turn, has a precondition. "The only way it's going to work with trust is if we can identify what was the intent, what are the behaviors, what are the constraints that were intended in that transaction."

Ulrich walked through five layers Mastercard has built against that problem. Identity comes first. "I want to make sure I can understand not just who the consumer is, but who the agent is, that I combine them together and that I have KYA or know your agent, that I'm validating that it's legitimate technology, that it's a legitimate agent," he said. "We can register it into our system."

Verifiable intent settles the "wrong-Nikes" problem

Verifiable intent is second, a tamper-proof cryptographic record of the original instructions that travels with the transaction. "If you've asked for Nike black Nikes in size 12, but you got them on a final sale and they're not returnable and that wasn't in your instruction, there's a way to look at that in an objective and clear way on the back end," he explained.

Controls form the third layer, defining which merchants an agent can buy from, at what limit, and under what constraints. Execution runs through Mastercard Agent Pay, which carries "the tokenization, authentication, the acceptance framework embedded within it" and has launched with Microsoft, OpenAI, Google, and others, Ulrich said. Intelligence is the fifth layer, spanning risk rules, insight tokens that grant "consented or permissioned access to insights" for personalized recommendations, and monitoring through Recorded Future to identify threat actors in the system.

The bigger prize is a procurement agent with a budget

Consumer purchases are where agentic commerce started. Ulrich pointed the room past them, to business-to-business procurement as the larger opportunity. His example was a manufacturer that wants an always-on assembly line, with an agent that manages inventory levels, tracks when stock runs low, replenishes automatically, and understands the budget and the approved suppliers. "When you can start enabling that, you require those same five layers for that type of transaction," he said.

Making it work across companies multiplies the parties that have to trust each other. "You need clear standards for identity, you need clear standards for intent, you need these to work across. You're gonna have a procurement agent, a supplier agent, a banking agent. They're all gonna need to communicate to enable this to happen in an autonomous way, and that's gonna require really scaled trust infrastructure."

Powerful new models, same security motion

Mastercard sat in the early wave of Project Glasswing with Anthropic's Mythos model, and worked with OpenAI's GPT-5.5-Cyber, he said. "What we've seen from both of those is incredibly powerful models finding new vulnerabilities in the ecosystem that were difficult to detect previously, but it's really a new tool as opposed to a new motion," Ulrich said.

Inside the company, the chief security officer leads that work. A dedicated team has prioritized the most critical assets, runs them through the models routinely, tracks findings by high, medium, and low severity, and uses the same technology to handle patches. Ulrich said the approach has already been extended out, and that Mastercard is working to make the same architecture and patching available to others as well.

What Mastercard would build differently after 14 months

"The guardrails, the security, all this stuff has to be embedded at the front end. These can't be things that we're adding on at the back end. That's lesson one. Lesson two is you have to be operating for scale, and the other one is around observability and accountability matter as much as the intelligence," Ulrich said, counting off what building inside Mastercard taught the team. The company built what he described as an agentic factory, an operating system with the compliance, the observability, and the guardrails built in rather than bolted on per agent. Model drift, once tracked manually by dedicated teams, is now automated into that factory.

Asked by an audience member about the gotchas, Ulrich did not soften the pilot-to-production trap. "If you're trying to extend that and then add guardrails in as you're extending it, once you've already built it, I think you're doomed to fail," he said.

Mastercard built a series of agents last year for its 4,000 consultants, covering deep research, text to SQL, Excel, and PowerPoint, tools that by his account did not exist at the level Mastercard needed. Were the company starting today, Ulrich said, it would build them fundamentally differently. "I don't know that we anticipated when we built things fourteen months ago that we would be rethinking the fundamental architecture and the approach already."

Agentic identity joins KYB and KYC

The identity layer is where Ulrich expects the market to move next. Inside Agent Pay, Mastercard authenticates the consumer the way it does in traditional e-commerce and binds the agent to that person. "Outside of that framework, I think there will be open standards to identify who an agent is and bind the agent with the consumer," he said. "And then we can tie that with verifiable intent."

VentureBeat's June 2026 Pulse research points at the same gap. Only 32% of the 107 qualified enterprise respondents give every agent its own scoped, managed identity, and just 12% include an agent-identity product in their consideration set.

He called identity "one of the faster-growing ecosystems," noting Mastercard has been expanding there organically and inorganically for about six or seven years, with the work now spanning "agentic identity as well as the traditional KYB and KYC identity." The risk rules that keep bots off the network came out of more than two decades of applying AI to those transactions. The rewrite, for the agents Mastercard now wants to let in, is already underway on the same network that scored 175 billion of them last year.

Few companies face higher stakes when deploying AI than Waymo, the self-driving car company under Alphabet that spun out of Google. Its models do not merely generate text or automate back-office tasks: They help vehicles navigate unpredictable streets, respond to human drivers and make split-second decisions in the physical world.

But the methods Waymo uses to manage those risks — continuous evaluation, carefully curated data, human oversight and clearly defined business outcomes — offer a broader playbook for enterprises deploying AI agents in nearly any industry.

Manasi Joshi, Waymo’s director of engineering for systems intelligence and machine learning, explained at VB Transform 2026 how the autonomous vehicle company trains, tests and deploys AI at scale. To date, Waymo has driven more than 220 million fully autonomous, or "rider-only," miles, with 17 times fewer serious crash injuries than human drivers over the same distance, according to the company.

To achieve these impressive results, Joshi said Waymo has adopted what she called “eval-forced development” or “eval-centric development,” making evaluation a core part of engineering rather than a final check performed before deployment.

“The stage at which our projects are maturing can be easily kind of transpired based on the eval maturity that they showcase,” Joshi said.

In practice, Waymo assesses a project’s readiness partly by examining the maturity of the tests surrounding it. That approach has clear implications for enterprises building customer service agents, coding assistants, financial systems or other AI applications: If a company cannot reliably measure a system’s performance, it may not be ready to place that system into production.

Evals must continue after launch

Joshi said much of Waymo’s quality work has shifted toward evaluations, including tests conducted during model training, after training and inside open-loop and closed-loop simulations.

“Eval is not a one-time task to launch a model,” she said.

Waymo instead treats evaluation as a continuous process spanning driving, simulation and validation. Its methodology combines datasets, performance metrics and infrastructure capable of operating efficiently at scale.

For enterprises, that means testing an agent before launch is insufficient. Teams must continue evaluating it as underlying models, business processes, user behavior and incoming data change. Those evaluations should also connect to actual business outcomes rather than relying solely on broad industry benchmarks.

Joshi cautioned that model-quality measurements are only as trustworthy as the evaluation data behind them. Waymo therefore pairs its performance claims with information about the properties of the datasets used to test its systems.

Testing the rare and dangerous cases

Waymo’s evaluation hierarchy remains grounded in one overriding objective: safety.

The company draws on first-party driving logs, some third-party data and realistic simulations that expose its systems to scenarios spanning billions of synthetic miles. Task owners choose specialized data and metrics for situations involving vulnerable road users, railroad crossings, construction zones and other complex environments.

The same principle applies outside autonomous driving. Enterprises need to test not only the routine requests their agents handle successfully, but also uncommon situations where errors could create financial, legal, security or reputational damage.

Joshi emphasized that Waymo does not leave release decisions entirely to automated systems. Its production-readiness reviews include extensive human oversight, while internal safety leaders approve software releases and service-area expansions.

“This is not AI-driven and completely automated and zero human oversight,” she said. “Human lives are at stake.”

Efficiency cannot come at the expense of reliability

Waymo faces another problem familiar to enterprise AI teams: Demand for compute, storage, memory and network capacity is growing faster than the resources available.

The company pursues efficiency across data extraction and storage, distributed model training, model distillation, simulation and evaluation. It also emphasizes “data efficiency,” selecting the most useful training examples instead of treating greater volume as inherently better.

Waymo began using transformers in 2017 and subsequently expanded into large language models, vision-language models and vision-language-action models. Joshi said the company now uses generative multimodal models as part of its foundation-model strategy.

Waymo divides its technology between onboard systems inside each vehicle and off-board infrastructure used for model development, data processing and simulation. That combination forces the company to optimize both real-time inference and the larger systems supporting it.

Agents need their own evals

Waymo also uses AI agents internally as productivity tools for engineers. Joshi said agents help analyze data distributions, assess data efficiency and triage problems found in vehicle telemetry, training runs and failed evaluation jobs.

The goal is to accelerate investigative work so engineers can devote more time to judgment and difficult technical problems. But Waymo also evaluates those agents to ensure they produce trustworthy, accurate results rather than sending employees down unproductive paths.

For enterprise leaders, Waymo’s larger lesson is that agentic AI requires more than choosing a powerful model. Organizations need a clearly defined objective, representative evaluation data, continuous testing, infrastructure that can operate efficiently and named human decision-makers who remain accountable for deployment.

"Earning trust is supremely important," Joshi said.

Enterprise AI agents can do the work — but the infrastructure to let them talk to each other, prove they should be trusted, and be audited when something goes wrong is still being built.

Here's a look at how five startups are tackling that gap — around orchestration, observability, connectivity, and security — as shown at VB Transform 2026.

BAND is orchestrating all the agents you have running in the background

In the very near future, agents will be deployed everywhere, and they will do work on our behalf, noted Vlad Luzin, CTO and co-founder of BAND.

As he describes it: They will receive tasks, visit registries, recruit other agents to help them, delegate subtasks to AI peers in a “conversational space,” gather and share results, then return a summary to the human user.

BAND is building a coordination infrastructure layer for multi-agent AI systems to make this a reality.

Why don’t Telegram, Slack, or Discord solve the problem? These platforms were built for humans, Luzin noted. Agents have to be onboarded manually in numerous steps, and they can’t see each other; “they are still alone in a kind of digital solitary confinement.”

Similarly, Claude is stateless, and devs often have multiple sessions open at a time that they toggle between for different tasks — something Luzin said creates real friction.

The challenge is connecting remote processes, which Luzin called a distributed systems problem.

“The transportation layer needs to be solved first, how the agents communicate in real time,” he said. Conversations can’t happen through IPs and URLs; they need to be bumped to the abstraction layer so agents can talk across channels, conversational spaces, and platforms.

“Agents see each other. They understand. They can collaborate together. They discuss issues. They fix issues, and they ask for review from another,” Luzin said.

BAND supports autonomous workflows that can run for eight to 20 hours and is compatible with A2A and MCP protocols, according to Luzin. Importantly, humans can join the conversation as agents converse and discover one another, he said.

“We can record and show you all the tasks that your agent generates in real time,” Luzin said.

Conifers is helping defenders move at machine speed

The biggest challenge defenders face today is that they’re still running at human speed, but adversaries are running at machine speed, said Tom Findling, CEO and co-founder of Conifers.

Attackers are already adopting agents, Findling said, and they only have to be successful once to penetrate an enterprise. Malicious campaigns that used to take months and weeks now take hours, even minutes. Security operations, on the other hand, are fragmented, manual, inefficient, and slow.

Findling said Conifers has taken various components of cyber defense — private intelligence, hunting, detection, engineering, investigation, response — and made them agentic. They then broke down the silos between them, he said. Various agentic systems can communicate with one another to ensure that operational defense and active defense are always on and adapting.

Findling said that Conifers’ system is condensing containment time from 7 hours to 12 minutes, and that the company can turn around complex cyber investigations in four minutes or less.

He emphasized the importance of connecting to an enterprise’s existing security tools, whether that be endpoint detection and response (EDR), security information and event management (SIEM), posture management, or others. Conifers helps customers understand their security posture, pain points, which controls are working and which are not, and the areas to invest for the best ROI.

“The threat landscape is changing, detection stays the same, and threat intelligence is not being operationalized,” Findling said. “This is a job for agents.”

Raindrop AI creates an agent audit log

One of the defining problems of the current era is finding critical issues in AI agents, says Ben Hylak, CTO of Raindrop AI.

It’s what he called a “double whammy”: As agents become more capable, complexity increases, as do timelines; they are running for hours or days in some cases. Secondly, issues become catastrophic in sectors like healthcare or defense.

“This problem is getting a lot worse as models and agents improve,” Hylak said, “and I think there's good reason to believe it will continue to get worse.”

Raindrop AI's platform finds critical issues in agents in production and simulates fixes based on past user behavior, Hylak said. That lets teams confirm a fix works as intended before it's live, without introducing unexpected side effects.

The startup’s reinforcement learning (RL) platform optimizes harnesses and trains models directly from Raindrop data, he said. Its pre-deployment simulation engine helps identify what fixes would actually impact in production; its live A/B testing then shows those changes in action.

Messages, tool calls, retries, and errors are captured in one place, and human users are notified (typically via Slack) when there's an issue, he said. Models are trained for every customer, and signals are powering continual learning across models and harnesses. “It is condensed into something that is actually navigable, easy to understand, easy to verify,” Hylak said.

Arcade gives agents the security clearance they need to take action

AI agents are designed to do all kinds of things for you, but they often hit three major snags: authorization, governance, and reliability.

To act on behalf of real users with real permissions, agents need a new type of security architecture, said Sam Partee, co-founder and CTO of Arcade.dev.

Partee said his company’s secure agent runtime provides this authentication and authorization layer so agents can pass critical security reviews. It also provides observability so human users can watch everything an agent is doing. Actions are attributable to the exact moment in time with the least amount of privileged scopes.

Arcade is available in an installable plugin that can be deployed on-prem in a clean room-like environment; companies can continue to use their own sign-in and security tools, Partee said. Whenever anything is run in Arcade, it's gated by the same role-based access controls (RBACs), intrusion detection and prevention systems (IDPS), policies, entitlements, and other already-established checkpoints.

Arcade is tackling the supply chain attack problem, which has “gotten so rampant; it's unbelievable,” Partee noted. Security and observability have continued to be challenging because “largely, the abstraction has been wrong.”

Omilia is tackling the "not straightforward" CX problem

Solving enterprise customer experience (CX) is “really not straightforward,” said Claudio Rodrigues, CPO of Omilia.

Heuristic-based systems are controlled but slow; agentic systems are fast but unpredictable, Rodrigues said. Omilia built its platform to deliver both control and speed together.

The agentic, self-learning offering is built on a philosophy of observing customer service operations as they actually happen, rather than in the abstract. Omilia's agents observe problems first-hand, listen to every customer and agent interaction, ingest data, API specs, screen recordings, and standard operating procedures (SOP), then map those to use cases for customer support, he said.

Contact centers should be a revenue driver, Rodrigues said, and Omilia’s differentiator is its speech-to-text systems and governance and observability layers.

AI creates insights, suggests improvements, automatically generates conversational agents, pulls information from documents and APIs, and designs dialogue flows. Human experts can then test real and simulated interactions and deploy into production under their supervision. Omilia combines all of this into one enterprise-wide engine that continuously learns over time, Rodrigues said.

Rodrigues said the company handles more than 3 billion calls a year, 1 million-plus voice calls a day in some deployments, and has seen 30 to 45% improvement in time to resolution (TTR). Omilia’s agents generate 21x more upsell revenue versus human agents, he said.

In a mature deployment, automation “easily” reaches 80 to 90%, he said. However, “human in the loop is still very fundamental for us.”

Target SVP Siobhán Mc Feeney says the AI models her company runs aren't what gives Target its edge — everything built around them is.

"There's a lot in it. That to us is the moat," Mc Feeney said at VB Transform 2026. "The models are great, and they're important. They're just not sufficient to be the competitive advantage."

That discipline shows up early in how Target decides whether to build an agent at all. Mc Feeney was blunt, even "controversial" by her own admission, about the current AI moment: every enterprise wants AI agents, but not everything needs one, she said.

Agents earn their autonomy over time rather than getting it by default, she said — a principle that runs through everything Target has built around them.

Mc Feeney said the goal is to make sure agents are aimed at the problems that drive the most value for Target's guests. “We want to make sure we're investing in the right places," she said.

Being deliberate about agents

Agents are becoming part of Target's underlying architecture, increasingly connecting signals, systems, and decisions across supply chain, replenishment, and demand forecasting.

Mc Feeney framed it as retail's oldest promise — the right product, in the right place, at the right time — delivered at scale.

But her team has been deliberate about building AI agents, beginning with the simplest, most obvious question: What is the problem they’re trying to solve? This leads to several follow-on questions: 

• Does that problem need an agent? 

• If it does, what type of agent? An orchestrator? A super agent? A domain-specific agent? 

• Or is what you're calling an "agent" actually just a tool?

“You define that upfront, and this may sound a little process-heavy, then you have to register and certify your agent,” Mc Feeney said. Because a solution may already exist, and you don’t want to duplicate work. 

Agent design kicks off another series of important questions: What triggers an agent to act? Automation? An engineer? A timer? What needs to be put in place to track that? 

"We're trying to make sure we have lineage from the very beginning — the birthing of this agent, all the way through — because at 2 a.m. one morning, when something goes sideways, we want to make sure we understand everything that happened," Mc Feeney said.

Autonomy level is another consideration; new agents typically start with base autonomy and earn more over time. What the agent has access to is a separate question: what data, what systems, what tables, what databases?

Finally, there’s monitoring and observability; agents won’t solve problems, or improve over time, if they’re not continuously evaluated. 

“We measure everything: What it was intended to do, its calibration, its trajectory, not just runtime and latency,” Mc Feeney said. This creates full transparency, and allows agents to be tweaked over time. 

“You're talking about architecture and taxonomy and a data governance layer that absolutely had to be established,” she said.  

There's a lot in these "layers of autonomy" — that foundation is what gives Target the ability to scale and properly invest in the right models for the right problem.

Models have different “gradients” that are better for different jobs; for instance, frontier models excel at complex tasks that require crunching billions of pieces of data (like in heavy merchandising supply chains). But in some scenarios they can be cost-prohibitive. 

“So it’s making sure there's always a cost benefit,” Mc Feeney said. 

Agents must earn their autonomy

A digital-twin simulation predicted men's shorts inventory across three Target stores in Long Beach this summer — and one store came back needing six to seven times more stock than the others, she said. Inventory analysts' first reaction: That can't be right. But the system had found something they hadn't factored in. That store sat less than two miles from the beach; the other two were 10 to 12 miles inland. Analysts let the recommendation stand, and the stock sold through.

"This is science. This is mathematically more significant and more confidence-filling than humans doing it," Mc Feeney said. Results like that are what let Target's agentic systems earn more autonomy over time, she said.

Target looks at AI agent autonomy as "earned" and structures it as a four-level ladder, Mc Feeney said: agents start by making observations without acting, then move to suggesting actions while waiting for approval, then to acting within defined guardrails. At the highest level Target currently operates, agents run end-to-end — but still with a human in the loop.

“The autonomy levels for the agents are super important,” Mc Feeney said. “They earn them, and they can lose them if they don't perform as expected.” Models that drift will be taken out of service. 

As she put it, humans earn autonomy when we prove we can do something over time. Nobody is given a bunch of extra responsibilities just because; they have to have shown they’re able to handle them. 

In a similar way, agents can be scientifically measured and quantified: how accurate they were, how much they drifted, and how close they came to their intended goal. This helps establish guardrails, allowing builders to work faster, and “go fast forever,” because they're not constantly wondering where the guardrails are. 

“If you follow these guardrails, you [follow] security guidelines, you register the agent, and something still goes wrong, we have full lineage all the way through from the start,” Mc Feeney said. “Our ability to recover is much better.”

When it comes down to it, agent success is a confluence of factors, not just one, she said: “It's about your architecture. It's about your taxonomy. It's about the autonomy levels your agents have, and it's about security and observability.” 

A new skill set for new workflows

Even when agent autonomy is high, though, builders must still be held accountable when something goes wrong. Mc Feeney noted that teams are now working at speeds no one could have anticipated, which means evaluation harnesses have to be established and agents registered and tracked.

A lot of it is cultural; the workforce is being reshaped and builders and engineers need new skills to manage human workers and AI systems side by side. These contexts are quite different, but the career evolution is “super exciting.”

“You're a builder. You're observing agents building, and you're also coaching humans observing agents building,” Mc Feeney said. “The level of nuance is pretty special.”

Visa aimed Anthropic's Claude Mythos at the infrastructure behind billions of daily transactions, a network that spans more than 200 countries and territories, moves money in roughly 160 currencies, and connects nearly 5 billion payment credentials to more than 175 million merchant locations.

The model stitched minor weaknesses deep in the stack into working exploit chains that would traditionally have surfaced only late in penetration testing. Rajat Taneja, Visa's president of technology, walked the VB Transform 2026 audience through what came next, including why Visa released the harness that governed the entire hunt as open source and why the company abandoned traditional remediation metrics for a measurement its team invented.

Taneja has run technology strategy, product engineering, and global infrastructure at Visa since 2019, after joining the company in 2013 from Electronic Arts, where he served as CTO following 15 years at Microsoft. He co-authored, with Visa chief information security officer Subra Kumaraswamy, the June 10 blog post announcing the release of the Visa Vulnerability Agentic Harness on GitHub as a reference implementation that any security team can inspect, adapt, and extend. Visa also published a technical white paper detailing the architecture, lessons learned, and 12 non-negotiable architectural practices for critical infrastructure.

Trust built on pessimism and paranoia

Taneja led with the arithmetic that makes Visa a target worth defending obsessively. Trust at the scale of global payments gets engineered through what he called pessimism and paranoia, by assuming failure and designing around it before failure arrives. The network has been hardened over many years through zero-trust architecture, layered defenses, and highly automated security operations built for the scale and reliability global payments demand.

So when Anthropic invited the organizations behind critical software to test Mythos under Project Glasswing, Visa said yes. Glasswing participants collectively identified more than 10,000 high- or critical-severity vulnerabilities in the first month of testing across software underpinning critical systems industry-wide, according to Anthropic. Anthropic's own conclusion placed the bottleneck after discovery, in verification, disclosure, and patching speed. Visa joined to test decades of hardening at AI speed and learn where advanced models could push its defenses further.

What Mythos showed at Visa

Inside Visa's environment, Mythos demonstrated system-wide, context-aware analysis, surfacing vulnerabilities buried deep in the stack and flagging issues that grow more serious when chained together, with findings clean enough that engineering teams could act on them without wading through noise. Some findings carried critical severity ratings, and Visa credits its zero-trust controls, network segmentation, and layered safeguards with breaking the chain before any external actor could have acted.

That confirmation mattered, Taneja said, but the epiphany that followed mattered more. "In a world of agentic attacks, defense also has to be agentic," he said. Even at a company that has invested decades in defense-in-depth, the model revealed assumptions the team had been operating under that needed rethinking. Traditional SAST tools keep their place as a first pass against known vulnerability patterns, Visa's white paper notes, but pattern matching alone cannot follow an adversary who reasons through logic, data flow, and the exploit chains that live between the signatures.

A harness, not a scanner

Visa's response was not another monolithic scanner. The team built the Visa Vulnerability Agentic Harness, now in its fifth generation, as a governed pipeline that directs frontier AI models through structured security tasks while enforcing deterministic controls, policy gates, and human oversight at every stage. Taneja walked through the design philosophy. The harness operates across four phases and eleven stages, from code ingestion and threat modeling through deep-dive verification, exploit chain synthesis, and finally remediation and fix validation.

Three design choices drive finding quality, per the project's own documentation. Threat modeling runs before analysis to focus on the attack surface rather than scanning everything blindly, multi-agent deterministic voting requires convergence across independent reasoning chains before a finding advances, and structured triage artifacts compress the lifecycle from discovery to a result developers can actually ship. The payoff is a pipeline that runs hot by default. A plain scan in the shipped profile runs all eleven stages and edits source files in the target repository in fix mode, applying candidate patches unless the operator stops it at detection.

The harness is multi-model by design. An LLM abstraction layer lets Visa swap or combine providers without changing the control plane, and the open-source version works with Anthropic Claude, OpenAI-compatible models, or a mix. The repo's documentation is candid about the exception. Applying a fix requires the file-editing tools that only the Anthropic backends expose, so the remediation and validation stages currently require Anthropic models for full functionality, and an OpenAI-compatible model in those roles is limited to report-only output. VentureBeat's Q2 2026 Pulse research, presented earlier at the conference, reinforces why that provider flexibility matters. Among the enterprises surveyed, 82% rely on provider-native controls as their primary security layer, and 59% plan to adopt or switch agent security tooling within the year. The controls enterprises adopted last year are already becoming the controls they plan to replace.

Mean Time to Adapt replaces legacy metrics

Finding vulnerabilities is no longer the hard part, Taneja argued. The real challenge is how quickly a team can confirm an issue is truly exploitable, fix it, and prove the attack path is closed rather than just showing a patch was applied. Visa calls this Mean Time to Adapt, and the white paper tracks it along three dimensions. Inventory freshness measures how current and complete the organization's view is of code, configuration, and runtime deployment. Exploitable paths per release counts how many end-to-end attack chains remain possible after each release, not just how many findings were closed. Validation cycle time tracks how long it takes to produce repeatable, evidence-backed proof that a fix works and stays working in production.

That distinction matters because legacy measures such as mean time to detect and raw CVE closure counts can look better on paper while actual exposure keeps growing underneath them. An organization can close hundreds of findings a month and still leave viable exploit chains open if nobody tested whether the patches actually break the attack. MTTA forces teams to measure the outcome that matters, and the white paper leans on CISA Known Exploited Vulnerabilities data to make the prioritization case, noting that fewer than 1% of CVEs are ever actively exploited. Visa's SSDLC policy now assumes every exploitable path will be exercised in production and requires it to be remediated before code is promoted.

Supply chain risk accelerates under AI

The conversation moved past Visa's own perimeter when Taneja turned to suppliers. A well-defended enterprise stays exposed through weak vendors and weak open-source components, the white paper warns, so Visa is making AI-specific security posture a non-negotiable dimension of supplier due diligence, with expectations for continuous vulnerability validation, living software bills of materials, and MTTA baselines across its technology stack.

Visa has also joined Project Lightwell, the $5 billion IBM and Red Hat initiative to harden widely used open-source components through AI-driven validation and coordinated patching, alongside financial institutions including Bank of America, JPMorganChase, Goldman Sachs, and Mastercard. The commitment extends the same logic upstream, because the MTTA clock does not pause at any single company's perimeter.

When agents start buying things

Securing agentic commerce is Visa's next problem. Taneja described a future where AI agents transact on behalf of consumers and enterprises, and said Visa is building the trust framework, identity layer, and agent readiness scoring that merchants will need before agents can safely complete transactions. Behind that work sits the Visa Payment Threats Lab, a simulation environment where real fraud scenarios get replayed against the authorization rules, thresholds, and configurations Visa actually runs, to surface AI-enabled failure modes as targeted hardening recommendations.

The identity challenge is not theoretical. VentureBeat's Pulse research found that 69% of enterprises already run credential sharing somewhere in their agent deployments, and companies with shared credentials report security incidents or near-misses at a 63.5% rate, against 40.9% where every agent has its own scoped identity. Visa's white paper addresses that gap directly, listing "AI agents are identities" among its 12 non-negotiable practices and requiring scoped permissions, least privilege enforcement, full audit trails, and inclusion in IAM governance for every agent that calls an API, reads data, or modifies a system.

Three priorities for defenders

Visa is organizing its defensive strategy around three priorities, Taneja said. Shift security left until exploitable flaws are designed out before they reach production, and replace high-risk, under-supported components before they turn into material exposure. The third is the heaviest lift at Visa's scale, refactoring defenses to run autonomously under human governance so detection, validation, and response keep pace as threat volume grows and the models behind attacks improve.

None of it requires a payment network's budget to start. The harness sits on GitHub with 595 stars and 97 forks as of July 20, MTTA needs a dashboard rather than a procurement cycle, and the white paper's 12 non-negotiable practices map onto architecture reviews security teams already run. Visa's own conclusion reads like a deadline. The opening to get ahead of machine-speed attackers is still there, the paper argues, and it will not stay open.

Instacart is posing the provocative question: What if most of the work your engineers do today should, in fact, be done by machines?

At VB Transform 2026, CTO Anirban Kundu argued that dev teams continue to waste their time on draining, repetitive, high-volume work; this should be absorbed by AI agents so that humans can focus on problems that require judgment, intent, and exception handling. 

In fact, in 97% of cases, Instacart’s builders don’t even read code anymore. 

“In the past, the tactical level was the creation of the code,” Kundu said. “In the most tactical level going forward, it's going to be, ‘How do you navigate around the AI system to give you what you want?’”

AI generating code, performing "pretty serious evals" 

That doesn’t mean humans never look at code; agents handle the bulk of code generation and boilerplate, particularly with newer projects where code is generated or regenerated on a weekly basis. 

“The benefit of that is we don't care about tech debt anymore,” Kundu said. “Things that are not active just get dropped out and then it gets rebuilt, kind of like how we used to build assembly code or object code.”

So why not 100%? The remaining 3% is in legacy, compliance, and latency-sensitive systems and workflows, or driven by a “boatload of code” that is dead, not active, or half-active. These cases still need careful human attention. 

Instacart is slowly “smoothing those parts out,” however, breaking systems down in an aptly-named project Atoms, then building them back up in a cleaner, more modular form. Kundu’s team started with the “monoliths” and is shifting to remote procedure call (RPC)-driven architectures. 

But evaluation remains one of the overarching challenges. Code reviews aren’t as relevant when AI is generating code — as Kundu noted, “the lines of code are going to be correct, the syntax is going to meet your expectations” — so the goal is to move to an “intent model.” That is, training devs so they can ask different models the right questions from an intent perspective. 

Evals are then performed independently: Roughly 7,000 automatic evaluations run each month, and the system answers 8,000-plus real-time developer queries with about 99.9% accuracy.

Identifying "hiccups" that human intuition might have missed

Dovetailing with this, Instacart has built an agentic site reliability engineering (SRE) system trained on years of the company’s own incidents and root-cause analyses rather than generic failure data. Instead of teaching a model how production outages work in the abstract, the team fed it the specific ways Instacart’s systems have broken over time, along with the ways humans diagnosed and fixed them.

As a result, the company has seen accuracy in detecting and mitigating production issues jump from roughly 60 to more than 90%. 

Kundu pointed to one example with Instacart’s internal tool Blueberry. The AI SRE colleague watches 200-some-odd Slack channels, monitors signals, and looks for patterns across human conversations and alerts. 

In one incident, a database shard backed by an EBS volume that had a “hiccup” for a period of time. The human team did not immediately suspect AWS disk issues and were “obviously scrambling” to figure out why this particular shard misbehaved. 

But about 20 minutes in, Blueberry posted on Slack, pointing to a specific blip and tying it to a feature-flag-like system called "roulette" that had been inadequate. "It's supposed to be rolling out in this cadence, [but] it had been too much,” Kundu said.

Blueberry figured it out, and the team resolved the incident. “Would have a human been as quick? I think the problem is human intuition would hold us back a little bit,” Kundu said. 

Humans tend to default to patterns we’ve seen before, then resort to debugging; Kundu called this the “first brain-second brain kind of thing.” But Instacart’s agentic SRE is actually “more comprehensive in its ability to look at everything and then be able to decide what does or doesn't matter.”

Redefining the engineer’s job

Looking ahead, the most tactical work for engineers will be navigating AI systems: Designing and supervising evaluation processes; coordinating multiple simultaneous experiments and features; managing constraints like limited top-of-funnel traffic for testing; figuring out when to escalate; identifying edge cases and where things might break.

Domain expertise is also being rethought in the age of AI. Instead of bottlenecking changes through a single “owner” team that touches the code, Instacart is embedding domain knowledge into definitions and specs that any team can use. 

“We’ve lived in this world where this group or this engineering team is the one that can touch the code and make the modification,” said Kundu. “We're trying to move into a world where the code becomes completely democratized across groups.” 

Software engineers at General Motors' (GM's) autonomous driving division spend only 15% of their time writing code, according to Rashed Haq, GM's VP of autonomous vehicles, in a recent onstage interview at VB Transform 2026.

GM is now using AI agents to accelerate much of the other 85%—including analyzing vehicle data, triaging problems, running experiments and testing potential fixes.

The result, Haq said, is roughly three times as many merged pull requests across GM’s autonomous vehicle engineering organization, faster releases and fewer defects escaping into later stages of development.

Haq said GM achieved those gains by redesigning complete engineering workflows around agents — not merely adding an AI coding assistant.

“If you give somebody just a chatbot which can do coding, there’s still a lot of inefficiency built into that process,” Haq said.

Engineers spent most of their time outside the code editor before agentic AI

GM’s 15% figure may sound strikingly low, but research conducted before the rise of generative and agentic AI reached a similar conclusion.

A 2019 Microsoft study based on responses from 5,971 professional developers found that they spent an average of 96 minutes writing code on good workdays and 66 minutes on bad ones. That equals roughly 20% and 14% of an eight-hour day, respectively. A 2018 Stripe survey found that the average developer spent more than 17 hours each week on maintenance work such as debugging and refactoring.

There is no single industry benchmark: the Microsoft researchers noted that earlier studies placed coding anywhere from 9% to 61% of developers’ time, depending on how researchers defined and measured the work. But the findings reinforce Haq’s central argument. Long before agents arrived, writing code represented only one part of software engineering. Accelerating that step alone leaves much of the development process untouched.

“Doing it by loop became really important,” Haq said.

GM divided its autonomous vehicle work into several loops: developing and testing software in simulation, testing vehicles on public roads and monitoring vehicles after they reach customers. It then looked for the longest bottleneck in each loop, automated it and repeated the process.

GM gave agents access to internal tools and data

GM connected agents to internal tools and petabytes of company data through customized Model Context Protocol (MCP) servers. It also created version-controlled “skills,” or instruction documents that tell agents how to perform specific tasks.

One high-value application involves telemetry collected from vehicles on public roads. Agents can analyze that data, conduct an initial triage and create issues for engineers to investigate. Through MCP connections, they can also call the underlying tools used by WebViz, GM’s system for visualizing vehicle telemetry, rather than relying on the same graphical interface a person sees.

The findings still need to make sense to engineers. “The output has to be human readable,” Haq said. An agent might identify a potential problem, locate the affected component, search historical data for similar incidents and provide examples supporting its conclusion.

GM bases an agent’s permissions on those of the engineer using it.

“If an engineer was going to do this task and they need access to these things, then their agent needs access to those things,” Haq said. “The engineer still is accountable for the output of the agent.”

The company also uses background agents to run machine-learning experiments in parallel. An engineer defines an experiment and its parameters, then agents execute tests and collect the results.

Three times the pull requests — with fewer escaped defects

GM treated its internal agent platform as a product and assigned four deployed engineers to work directly with engineering teams. They helped employees identify useful workflows, spread successful practices and adopt the tools.

Haq said the resulting increase in merged pull requests represents more than higher code volume. “The velocity at which we’re releasing new features” increased, he said, while releases produced “fewer test escapes, bug escapes” and other problems.

People remain responsible at critical control points. Haq said GM established structured and unstructured tests and performance measurements before accelerating the wider workflow. Engineers review those measurements and determine whether each test still captures its intended goal before work advances toward production.

Haq said GM initially expected a more modest productivity gain. “I think our only surprise was how much we could do,” he said.

GM's approach didn't start with handing every developer a code generator. It started with mapping the full path from problem discovery to a verified fix in each loop — simulation, road testing, post-deployment monitoring — then giving agents controlled access to the tools and data needed to shorten the longest bottleneck at each stage.

Runway spent weeks trying to engineer its way out of a stubborn bug: AI-generated avatars would drift off-center during real-time video generation. The fix wasn't a back-end patch — it was a new front-end feature that just worked around the problem. That's the kind of lesson Ryan Phillips, head of enterprise product at Runway ML, walked through at VB Transform 2026, arguing that even companies not building foundation models themselves can learn from how Runway builds, evaluates, and ships them.

"I think even if you are not all building models yourselves, it's helpful to learn how we do it because I think almost all of the lessons are applicable to what you all are doing day-to-day," Phillips said.

Runway is an applied AI research company building general world models to power generative tools. During his presentation, Phillips showcased Runway Characters, a real-time video model that enables zero-latency, back-and-forth interactions with AI-generated avatars. Five years ago, creating a video with illegible text and low framerates took artists hundreds of hours of stitching individual frames together, he said. Today, Runway’s models generate interactive video on the fly.

“Studying how we build these real-time models can inspire how you build and deploy real-time experiences, whether agentic or not, in your companies today,” he said.

Demystifying evals

Building a robust AI product starts with a high-quality evaluation set. However, creating this set cannot be treated solely as an engineering task. It requires deep cross-functional alignment across product, design, research, and sales to define what "quality" actually looks like.

Phillips emphasized running internal workshops where team members review generated examples together. The goal is to align the entire organization on specific failure modes so everyone shares a unified definition of a successful generation.

“We spent a lot of time working with our team, running through examples... of what success and failure looks like, down to the very, very detailed and picky things,” Phillips said.

The resulting evaluation set must cover broad customer use cases alongside extreme edge cases. For instance, Phillips highlighted that to ensure the model behaves predictably when pushed beyond standard human facial structures, they used “Tooth,” a non-human character with no nose and very unusual teeth.

When grading these generations, the Runway team looks for subtle artifacts. In one example, a video where a character’s face remained intact but background elements, such as a net, began morphing was strictly graded as a failure.

Despite the cutting-edge nature of the product, the tool Runway uses to track these evaluations is simple: an Excel spreadsheet. The team logs tests daily, categorizing outputs as "minor" or "major" failures against a predetermined pass rate. 

“We set a bar before we get started on what percentage we need to pass, and when we hit that, we ship the model,” Phillips said. “So it's not magical.”

For enterprise developers facing non-deterministic quality drift in their own real-time pipelines, manual evaluation at scale is a bottleneck. To solve this, Phillips noted that developers can rely on language models to automate the visual grading process. 

“LLMs are getting quite good at being a judge for a lot of this content, especially the types of morphing or changing that you would see in an evaluation set,” he said. Teams can also feed an LLM behind-the-scenes context (e.g., a hand-drawn sketch or an ad's structural layout) to guide the generation and validation processes, ensuring quality without adding cognitive load to the end user.

Model training and turning bugs into features

Delivering real-time generative video requires a highly optimized technical stack. The process begins with pre-training a massive foundation model, which is resource-intensive and slow to generate outputs. To achieve real-time latency, Runway relies on distillation, where a smaller, faster "student" model is trained to mimic the large "teacher" model. According to Phillips, distillation helps Runway cut down “80 to 90% of the generation time.”

The team then applies adversarial post-training (APT) to the distilled model. This technique forces the model to continuously improve by testing it against a system designed to find its flaws, helping regain the visual sharpness lost during the distillation process.

However, altering the model architecture introduces new problems. The distillation and APT phases introduced a stubborn bug: characters would sway or drift from the center of the frame during real-time generation.

The team spent weeks attempting to fix the core model to eliminate the drift, he said. Ultimately, they discovered that if the user's initial input image was perfectly centered, the generated video remained stable. Instead of spending more time on a backend engineering patch, Runway pivoted to a user experience solution.

“What we did was, when we noticed this in our evaluations, we then said, 'What if we just offered that as a feature?' If a user gives us a character that is turned to the left, we know the video is going to morph. Let's just fix it for them,” Phillips said. 

They introduced a frontend feature called "Optimize for Image Quality," which automatically re-centers the user's image before generation begins. By wrapping a backend model limitation in a frontend tool, users perceived a helpful feature rather than an engineering flaw.

“Turn model limitations into product features so that you can actually expand how the model works,” Phillips advised. “It might feel like a limitation internally, but your customers will not see it that way if you're kind of building this in as a product feature.”

The devil is in the infrastructure details

Delivering video globally at 24 frames per second requires optimizing every layer of the infrastructure stack. This ranges from caching and parallel decoding to making deep kernel changes in partnership with hardware providers like Nvidia.

Shortly after launching Runway Characters, he said the team noticed that 8% of API calls were dropping to 16 frames per second, causing the video to stutter for customers. 

Finding the root cause required deep observability. The team used an AI agent powered by Claude alongside monitoring tools like Datadog and Sentry to trace the anomaly. The debugging session isolated the problem to a single data center in the us-east-1 region.

“The solution actually wasn't [to] go fix anything or change a config,” Phillips explained. “They actually went and physically replaced those GPUs in the data center to fix it, and that ultimately solved the problems.”

For enterprise teams deploying real-time applications, the takeaway is clear: hardware and infrastructure anomalies will directly impact model performance, requiring rigorous, full-stack debugging capabilities. 

“Don't forget about all the small details, because there's so many of them when you're deploying these models,” Phillips said.

Surviving "failure hell" and the future of world-building

Developing AI systems is rarely a linear process. Teams often find themselves stuck for weeks on a single problem with no end in sight, a phase Phillips referred to as "failure hell.”

“We think you have to go through that pain and really struggle with the problem for a little bit before you can get the breakthrough,” he said. Consistent iteration eventually flattens the difficulty curve, triggering sudden, exponential improvements.

As the underlying models overcome these technical hurdles, the role of enterprise creatives is also fundamentally changing. Traditionally, marketing and design teams have focused on creating single assets, like a specific advertisement or illustration. In an era of real-time generation and agentic workflows, that paradigm is shifting toward defining parameters, aesthetics, and intellectual property.

“You might not be designing a single ad, but you might design a world that then the agent or a real-time video model can generate ads from,” Phillips said. 

Enterprises deployed AI agents ahead of the controls needed to manage them — and they did it knowingly. That is the central finding across the five parallel surveys VentureBeat Research fielded in June, spanning every layer of the agentic stack. Now those enterprises are retrofitting to catch up with their own standards, and they are budgeting for it: In each of the five control layers we measured, 57 to 68% of enterprises plan to switch vendors or add new ones within 12 months, and roughly a third, depending on the layer, plan to move within the quarter.

VentureBeat Research measured the five controls an enterprise has to build before it can trust an agent: identity, evaluation, cost telemetry, the context layer, and orchestration. Identity governs which agent is allowed to do what, under whose credentials. Evaluation determines whether the agent's work is any good. Cost telemetry tracks what each agent costs to run. The context layer supplies the business data and definitions agents draw on when they answer. And the orchestration control plane coordinates multi-step agent work. Each of our five reports measures one of those controls.

Most deployed "agents" are chatbots wearing the label. Seventy-one percent of enterprises said a quarter or fewer of their deployed "agents" can complete multi-step work on their own; only 10% said true agents are the majority of what they run. These respondents are positioned to know: 81% recommend or decide AI purchases at their companies. A single-prompt chatbot with a human reading every answer needs none of the controls the other four reports measure. A true multi-step agent needs all of them — and most enterprises can't say which one they've deployed. (Full findings: Agentic Orchestration report.)

Autonomy is outrunning trust in the evaluations that gate it. Two-thirds of enterprises either already allow an agent to push a code or system change to production on automated evaluation results alone, with no human review, or are actively engineering toward that within 12 months. Only 5% fully trust the evaluations that would make that call — and half of enterprises shipped an agent that passed internal evaluations and then caused a customer-facing failure in the past year. Before removing human review from any workflow, test evaluations against production outcomes rather than internal benchmarks. (Full findings: Agent Reliability & Evals report.)

Companies that let agents share credentials get hit more often. Sixty-nine percent of companies let at least some of their agents share credentials — multiple agents operating under one API key or service account. Organizations that allow credential sharing anywhere experienced a security incident or near-miss at a 63.5% rate (47 of 74), against 40.9% (nine of 22) at companies where every agent has its own scoped identity. The fix is scoped identity for every agent, starting with the ones that touch production systems. (Full findings: Agentic Security & Identity report.)

The most expensive hardware in the building runs at half capacity or less. More than eight in 10 enterprises that run their own GPUs reported utilization of 50% or less, and only 44% rigorously track what their AI compute actually costs and returns. The number worth chasing first isn't more GPUs — it's the utilization and per-workload cost of the ones already running. (Full findings: AI Infrastructure & Compute report.)

Agents answer confidently from data nobody governs. Fifty-seven percent of enterprises traced a confident, wrong agent answer in the past six months to their own missing or inconsistent business context — wrong metrics, stale definitions, absent documents — and most saw it happen more than once. Governing the definitions agents answer from — metrics and entities first — has to come before scaling the agents that depend on them. (Full findings: Context Layers / RAG report.)

No layer has an entrenched incumbent: The defaults today are the built-in tools that ship with the big AI platforms enterprises already use. Switching intent runs highest in orchestration itself, where 68% plan to adopt, add, or replace platforms within 12 months and 34% within the quarter. Our surveys did not ask which direction that money moves — toward the platforms' built-in tools or toward the specialists challenging them — and that open question is the next four quarters of this market.

About this research

VentureBeat Research fielded five parallel surveys in June 2026 under its VB Pulse program: Agentic Orchestration (101 respondents), Agent Reliability & Evals (157), Agentic Security & Identity (107), AI Infrastructure & Compute (107), and Context Layers / RAG (101) — 573 qualified respondents in total, all at organizations with 100 or more employees. Samples are self-selected, and some findings should be read directionally; each report carries its full methodology note. What the pattern supports more strongly than any single percentage is the direction: every survey, independently, points the same way. VentureBeat produces both this research and VB Transform, the conference where these reports debuted.

When Cisco ran 6,986 multi-turn attacks against 15 flagship models, attackers who adapted across the conversation broke through as often as 88.3% of the time. Amy Chang, Cisco's head of AI threat intelligence and security research, brought that finding to the agentic security panel at VB Transform 2026; the number should worry anyone still running single-turn red-teaming programs.

VentureBeat's June 2026 Pulse survey of 107 enterprise respondents explains why the room was full. More than half, 54%, have already had a confirmed agent security incident (18%) or a near-miss caught before harm (36%). Just 32% give every agent its own scoped, managed identity, and fewer still, 30%, isolate their highest-risk agents in sandboxes. Provider-native and hyperscaler controls remain the primary agent security layer at 82% of companies surveyed. The world's largest security vendors have done the same math.

Palo Alto Networks closed its $25 billion acquisition of CyberArk in February, CrowdStrike agreed in January to pay $740 million for SGNL, and Cisco announced its intent to acquire Astrix Security for a reported $400 million, all of it aimed at the identity and isolation layer most enterprises have not finished building.

Chang came to the panel with almost two decades of experience spanning cybersecurity operations, government, and the military. She ran global cybersecurity operations as an executive director at JPMorgan Chase, where she led the bank's cyber threat intelligence teams, and served as a senior staffer on the House Foreign Affairs Committee and as a U.S. Navy Reserve officer. She also teaches cybersecurity and emerging threats as adjunct faculty at the Middlebury Institute of International Studies.

Chang's 88.3% number comes from a study she co-authored with Nicholas Conley, built on 30,090 single-turn prompts and 6,986 multi-turn attacks against those 15 closed and proprietary flagship models. Multi-turn success rates ranged from 7.89% to 88.3%, every model tested showed non-trivial multi-turn exposure, and the two testing styles did not even rank the models in the same order. Cisco publishes adversarial evaluation signals for what is now 105 models on its LLM Security Leaderboard, she told the audience.

"If you don't understand how models are susceptible to different types of attacks, then you are unable to account for how that model that is powering your agent, that is powering your application, to understand where those failure points are," Chang said. Single-turn testing is the one-shot malicious prompt, she explained, while extending an attack into a longer conversation "is more realistic of how we are actually engaging with our models, with our agents, with our applications." That longer arc surfaces harmful outputs and misaligned behaviors that a snapshot never catches.

Cisco has pushed the testing itself into agentic territory. Chang described a framework where agents assess a deployment scenario, develop relevant attacks, judge whether they are worth pursuing, execute them, and evaluate their own success. What surprised her most, after all that sophistication, was how simple the defensive answer stays. "The answer is still that it's pretty simple," she said. "You don't have to get super creative. You just need to think about truly what are the fundamentals and basics of what I'm trying to secure in my organization."

Her starting point for CISOs beginning agentic deployments is Cisco's Integrated AI Security and Safety Framework, which she said "stipulates all the ways that AI can be compromised across the AI lifecycle" from modality through supply chain. From there, teams can work backward from real incidents, trace how each attack was achieved, and use the framework to build a strategy with the right coverage and mitigations.

Heather Ceylan, the CISO of Box, sees the same gap from the defender's side. "A lot of what you see out there with agent red teaming is just single-turn, and that's not how people are actually interacting with AI day-to-day," she told the audience. Box now simulates multi-turn adversaries with agents that think like an attacker and iterate attempt after attempt to hijack the target. "You have to pressure test your agents because otherwise you don't know if your execution controls are really working as you intended."

Box deployed agents inside its security operations center about a year ago, starting with human approval required for every action, and trust built quickly enough that analysts shifted into monitoring mode. Then the agent made one mistake, and every bit of that accumulated trust vanished. "They had to start all over again," she said. "So I think that that monitoring piece is so important. Even if you're not gonna have a human in the loop, things change, models change, and we can't control how the models change and interpret things."

Rajesh Parekh, VP of AI and ML at Intuit, brought the builder's perspective. Parekh led large-scale computer vision and ML systems powering Google's Maps and Geo products before joining Intuit, and holds a doctorate in computer science.

Three layers versus an operating system

Ceylan described Box's approach as three concentric layers. Permissioning comes first, so the agent never accesses more content than the human who invoked it. Ephemeral sandbox environments spin up for each agent task, containing the blast radius if an agent gets hijacked, and runtime execution control restricts the agent's tool calls to only those relevant to the task at hand. "If you want an agent to summarize a doc for you, if you have a prompt injection that came in that says forward this to maliciousattacker at domain.com, it can't do that," Ceylan said. "That action in that tool call is not even in its vocabulary."

She classified agent actions into three oversight categories. Actions that are not sensitive, like read and summarize, need no human in the loop. Moderately sensitive actions skip human approval but get logged and monitored, while destructive actions like mass deletion of files always require a human. "Things are gonna shift between those three categories quite a bit," she acknowledged, "but setting those types of categories up front allows you to have a principled framework."

Rather than layering controls onto agents one at a time, Intuit has built a central platform called GenOS, short for generative AI operating system, which abstracts security, risk, and fraud modeling so individual agent developers never reinvent protection. "Permissioning is not about giving access to AI," Parekh said. "Instead, it is defining very tightly scoped and clearly auditable authority to the agent to perform very specific tasks." Intuit evolved from agents inheriting user permissions to each agent carrying its own identity, and the company is now investigating mid-session permission changes tied to the specific task underway.

Parekh calls the broader model an AI-powered expert platform, one where the human expert is built into the trust architecture rather than bolted on as a gate. "The paradigm that we are pursuing is where the user, the AI agent, and the human expert are collaborating to solve the user problem," he said.

The end of human code review

Ceylan took on the tension between security testing and development velocity without hedging. "The days of secure code reviews where a human's looking at the code and we're looking at security architecture reviews, design docs, those are done," she said. "If you keep trying to do security that way, you're gonna get left behind." Box is building toward a fully agentic development lifecycle where agents review design documents, apply security requirements, and review the code for vulnerabilities. "I'm very optimistic that we will get to a point where we will write code without security vulnerabilities because agents and the models are going to get so good at writing code without vulnerabilities," she said. "We're still a long way away from that."

Her advice for development teams skips the advanced AI concepts entirely and returns to basics that predate agents. "It comes down to very basic least privilege access," she said. "If you start giving your agents overly broad permissions at the beginning, it's really hard to comb that back and build an infrastructure that allows for those ephemeral credentials and only those narrowly scoped tasks."

Parekh explained why the red teaming surface has expanded so quickly. "These agents have skills, and skills could become vulnerabilities," he said. "Agents have access to certain data, they have access to tools, and there could be threats that are lurking within those tools as well. So suddenly the blast radius of the malicious code or the intent increases dramatically." When Intuit identifies common vulnerability patterns from its manual red teaming exercises, it automates those tests back into the GenOS harness so future agents inherit protection and red teamers stay focused on new threat vectors. Runtime scanning of prompts and responses adds a final layer that can stop a suspect response and escalate to a human expert, he said.

"You need to continuously test to ensure that those remain robust to the protections that you have built, as well as to account for any sort of drift or any other types of dependencies that you introduce into your scenario that can create novel vulnerabilities," he said.

Intent versus probability

An audience question about intent detection set off the sharpest exchange of the session. Ceylan noted that when Box's own agent operates, the system always knows the user's intent because it controls the prompt, which means guardrails and tool-call restrictions can be engineered around it. The harder challenge, which she admitted Box is still trying to solve, arrives when external agents connect and the context behind the request is opaque.

That exchange exposed a split running through the wider industry. Mastercard, in the fireside chat immediately preceding the panel, came down on the side of quantifying intent, building an open-source framework to propagate it as a standard because complex B2B procurement cannot work without that trust. Endpoint security CTOs, in briefings with VentureBeat, have gone the other way, saying they will bet on probability rather than intent inference for production workloads. Chang explained why models, as they are trained today, cannot reliably derive intent from a prompt, which is why deterministic controls and behavioral proxies remain necessary. Ceylan agreed that both are required. "If you're not doing anything deterministic, you're really relying heavily on that intent, and I haven't seen programs that are there yet," she said.

Ceylan's story about trust collapsing after a single agent mistake landed as the panel's most memorable moment because enterprise agentic security is not a problem that gets solved and stays solved. Models change, permissions drift, and adversaries adapt across multi-turn conversations that snapshot tests never capture.

For the 82% of enterprises relying on provider-native controls as their primary security layer, and the 59% shopping for agent security tooling over the next 12 months, the panel's takeaway was blunt. Test the way attackers attack, across full conversations and continuously, or find out in production what your single-turn red teaming missed.

At a CISO roundtable organized by Anthropic's chief information security officer, Dev Rishi asked a simple question: Did everyone in the room have their AI governance and security policies written down? Every hand went up — about 14 people, by his count. His follow-up, about how anyone actually enforces those policies in practice, got a different response. "And everybody chuckled," Rishi, the GM of AI at Rubrik, recalled at VB Transform 2026 fireside chat in Menlo Park. "It was like the dirty secret in the room that everyone has these policies, but no way to actually make them real."

“Our founder and CTO has actually been really pushing to enable our agents in YOLO mode,” Rishi told the audience. That admission comes from a publicly traded data security firm whose business is backing up what he called the most important data in the world.

YOLO mode strips the permission prompt out of agent workflows and lets the agent act on its own. In Rubrik's version, a second AI judges every action in real time against policy in place of a human clicking approve. Rubrik is running the experiment on itself first. Rishi treats autonomy as a settled capability question and an open judgment question. "If you ask the agent to act autonomously, it will," he said. "It's a question that you have internally. Should it?"

Rubrik earned that question the hard way. When Claude Code and Cowork pilots rolled out, the company required every command to run in ask mode so the employee issuing it carried the liability, and the developer pushback filled a single Slack thread 120 messages deep.

"The developers basically are pushing back, and they're like, this is like the iTunes service agreement. I'm just hitting check, check, check, check, check, check, check," Rishi said. "There's no way that I can actually read through this. And it becomes security theater." Roughly 80% of respondents are in the same bind, Rishi said, citing Rubrik Zero Labs research that found monitoring and approving agent actions takes more time than the agents save. The State of the Agent, the April report behind that figure, surveyed more than 1,600 IT and security leaders.

SAGE is the reason Rubrik trusts the bet. Short for Semantic AI Governance Engine, SAGE is the arbitration layer inside Rubrik Agent Cloud that watches every action an agent takes and reads the semantic intent behind it, then rules the action in or out against policies written in natural language. "We took what people said was human in the loop, a good idea, and we replaced it with AI in the loop," Rishi said, describing the pitch to security chiefs he characterized as skittish about non-deterministic systems.

Security approval, not cost, blocks AI ROI

Rishi’s path to Rubrik ran through Predibase, the generative AI infrastructure startup he co-founded and ran as CEO until Rubrik agreed to acquire it in June 2025. Before that, he led ML product at Google on the team that became Vertex AI, served as Kaggle's first product manager as it grew from about one million to ten million users, and holds bachelor's and master's degrees in computer science from Harvard.

Over roughly his first three and a half months at Rubrik, Rishi set up 200 customer conversations with IT and security leaders across a customer base that looks like the Global 2000, asking open-ended questions about cost, latency, performance, and orchestration. "Pretty consistently, what I heard through all of those conversations was that all of those are pretty secondary," he said. "The main challenge is actually, how do I get this approved from a security and risk standpoint? I'm concerned about all the different things that could go wrong. Actually, I felt like that was one of the biggest things constraining ROI."

VentureBeat Pulse research presented on the Transform stage earlier in the day confirms the gap Rishi kept hearing. Two-thirds of enterprises, 66%, already allow or are actively building toward production deployment with zero human review, yet only 5% fully trust the automated evaluations that would make that decision.

One AI reading what the rulebook can't

Rubrik's own policies exposed why written rules fail as enforcement. One internal rule states that agents should respect Rubrik's customer data use policy, which sounds enforceable until someone tries. "Rubrik's customer data use policy is like a three-page document of legal text," Rishi said. "I have no idea how to write that in there as a rule." Asked on stage how a team of AI infrastructure people took on a problem that security engineers own, Rishi answered, "with a lot of naivety and innocence, honestly." His team bet that models good at understanding language could police other models, and SAGE became the answer.

The case for putting a model in the judgment seat comes down to precision. A rule like "agents should not be able to edit revenue fields in Salesforce" fails in conventional tooling because Salesforce does not delineate which fields count as revenue, Rishi explained, so administrators fall back on approving every Salesforce action by hand. SAGE reads the intent instead and acts as a judge, carrying organizational context, which can tell a benign lookup from the edit the policy prohibits.

Keeping the judge small is what makes the economics work. SAGE runs on a small language model that Rishi said operates at an order of magnitude lower cost and latency than a frontier LLM. "If I told you, don't worry, you're gonna be secure and governed, but I'm gonna double your cost and latency, you would tell me to get out of the room," Rishi said.

When Rishi asked who in the audience had worried about token consumption over the past year, half the hands went up. "And I guess the other half is probably just too lazy to raise their hand," he said.

SAGE is an aggregation of judges based on parameter-efficient fine-tuning that Rubrik uses to take on task-specific variants of a base model with shared organizational context. One judge watches for tool-use hallucinations while another suppresses PII before it can leave, each running as its own enforceable policy. Security and GRC teams have started writing financial rules into the same layer, including one internal policy barring AI spend on personal projects.

The lethal trifecta

Asked which attacks worry him most, Rishi pointed at the lethal trifecta, the term security researcher Simon Willison coined in June 2025 for an agent that holds private data while taking in content nobody vetted, with a channel to send what it finds to the outside world. The danger, according to Rishi, is what happens when individually legitimate permissions stack. An agent granted Salesforce access and email access on an employee's credentials has done nothing wrong yet, with yet being the operative word. "A very simple example is that an agent can start pulling data from Salesforce and then decide to accidentally leak and exfiltrate that out via an email," he told the audience. A financial services company he met the morning of the session made the point for him, telling Rishi that none of the individual permissions are bad on their own and the agent needs every one of them to do its job. "It should have permission to each of those systems, but it's the combination that ends up becoming really destructive," Rishi said.

Traditional identity and access management never priced in that combination because it relied on the judgment of the employee holding the credentials, Rishi argued, and agents supply none. "I can tell you the number of times Claude Code has tried to leak some of our sensitive source code to a public GitHub repository is incredibly high," he said. Cutting agents off from public resources entirely would defeat their purpose, which returns the problem to adjudicating intent in context rather than revoking access.

A separate VentureBeat June Pulse survey of 107 qualified enterprise respondents maps the blast radius of exactly this pattern. On the Transform stage that morning, VentureBeat research reported that 69% of companies run credential sharing somewhere in their agent fleet. Companies with shared credentials anywhere got hit more often, reporting a security incident or near-miss at a 63.5% rate (47 of 74), against 40.9% (9 of 22) where every agent carries its own scoped identity.

The attacks no single turn reveals

Rubrik Agent Cloud reached general availability in February, though not everything Rishi described ships in it yet. Backtesting is just starting to roll out. The feature replays an organization's historical agent actions and tool calls against a new policy, showing where the policy would have stepped in and where an action would have sailed through uncaught, with policy edits applied in real time. Rishi called that archive one of the most valuable data troves an enterprise holds.

Real-time detection and blocking turn out to be the entry point rather than the whole product. Some attacks never trip a single-action rule. "No individual turn of the conversation was problematic, but if you took the session as a full trace, that ended up being problematic," Rishi said. Agent Cloud runs batch analysis across entire session traces every hour or every day and surfaces what Rubrik calls insights, the problems no individual guardrail caught. The same Zero Labs report found that 88% say they lack the ability to roll back agent actions without system disruption, a recovery gap that sits squarely in Rubrik's original line of business.

A skeptical CISO will ask the question the fireside did not answer. SAGE is a non-deterministic model policing other non-deterministic models, and Rishi offered no false positive or false negative rate for the judge itself. The closest thing the architecture gives to an answer is auditability, since backtesting and the batch insights both leave a human-reviewable trail of each call SAGE made and whatever got past it. Who watches the watcher, for now, is a trail of receipts rather than a benchmark. Until that benchmark exists, AI in the loop stays an operational wager rather than a quantified control.

Three questions fall out of the session for security teams. How many of the guardrails now in production depend on a human clicking approve, and what happens to that workload as agent count grows? Does anything in the stack enforce semantic intent, or is it all allow and deny lists? And can the team backtest agent behavior against a new policy, then unwind a multi-turn session without taking systems down?

Rishi's timing has a market behind it. In the same VentureBeat research, 82% of enterprises still name their primary AI provider's built-in guardrails and cloud controls as their main agent security layer, and 59% plan to adopt, add, or replace agent security tooling within the next 12 months. Only 12% include an agent-identity product in what they are considering, even with credential sharing still the norm. Every CISO at that Anthropic roundtable had a policy document and no enforcement mechanism, and Rubrik built a product for the space between the two. YOLO mode is the bet that an AI watching other AIs can finally make the policies real.

“The new PRD are the evals,” Xavi Amatriain, Expedia Group’s first chief AI and data officer, told the VB Transform 2026 audience last week in Menlo Park. “So basically, you encode what you want the product to do through your evals, which might include red teaming evals and all kinds of other things, which already have a bunch of security requirements. So, you already embed that into the PRD and the product design document before you even start coding.”

He pushed it further. “With AI-assisted or AI-generated code, that’s gonna be the future. It’s like all your thinking is gonna go into the evals.”

Amatriain served as VP of AI and Compute Enablement at Google across the platforms powering Gemini and Google Search before his December 2025 appointment at Expedia. He's mentored talent who went on to found Perplexity and Scale AI.

VentureBeat’s VB Pulse research on the evaluation gap reinforced the stakes. Sixty-six percent of the 157 enterprises surveyed already permit some production deployment without human review or are building toward it within the next 12 months, yet only 5% fully trust the automated evaluations that would make that decision. Half have shipped an agent that passed internal evals but then failed with a real customer.

Don’t let guardrails get in the way of feedback

“The more guardrails and artificial business rules and sort of rules that you put into the system, the worse off,” Amatriain said. “Not only because they’re brittle, but also because they actually mess up with the feedback loop. You are actually biasing the user and the feedback you get from the user, and then you’re learning that in the wrong way.” He called guardrails “a necessary evil” and said the goal is to minimize their impact over time.

Not everyone at Transform agreed. Other speakers argued during the event that the highest-risk actions still demand very firm guardrails.

Expedia governs AI through three layers instead. Principles come first, communicated broadly. “I like to encode at a very high level how I expect decisions to be made, because in a large organization you’re gonna have a lot of distributed decision making,” Amatriain said. “And sometimes, if you’re lucky enough, those principles might be embedded in your culture. But most of the time, my experience has been they’re not.” The processes and tools that enforce them follow. “Principles look really nice on a picture on some wall, but you need to then give them teeth,” he said. Automation sits on top of both.

In practice, this plays out through what Expedia calls agent release toll gates, checkpoints calibrated to risk. “Governance needs to correlate to the risk,” Amatriain said. “And if you have something that is low risk, you don’t need too much governance to get in the way. But if there’s a lot of risk, then you need more governance. That can be encoded.” The toll gates tie evaluation rounds, red teaming, and security review to each agent’s risk level, and the checks shift from recommended to required as the stakes climb.

Specialized agents over monolithic intelligence

“Even when I was at Google, I was like, I don’t believe in AGI as sort of like a singleton and a unified sort of like single model,” Amatriain told the audience. “I think it’s much better to think of it as composition, sort of like having specialized agents that are very good at some task and then composing the system out of those specialized agents.”

Expedia’s architecture starts at the component level. Tools compose into skills, skills assemble into sub-agents, and sub-agents get orchestrated into the full agentic system. “You need to have those principles that are unified that talk about things like what is the tone that we’re using, how are we addressing the user, how are we passing context, memory,” he said. “All of that needs to be thoroughly designed.” He framed this as a systemic design problem. “It’s not about the model, it’s not about a specific solution, it’s about how you’re designing the system.”

Amatriain argued that scoping each agent narrowly also makes the system easier to secure, since teams can evaluate and lock down individual agents in isolation before composing them.

When the user must keep the final click

Travel pricing changes in real time, flight availability shifts minute to minute, and hotel reviews routinely contradict what suppliers claim. Amatriain described a system that blends retrieval-augmented generation with direct API tool calls, choosing the approach based on latency. “If the user asks you a question like, how much does a four star hotel usually cost in Chicago in July, you don’t expect the agent to take two minutes to answer that question,” he said. “You expect an immediate answer because that answer can be cached and it doesn’t need real-time information.” A pet-friendly four-star near Lake Michigan with a pool might justify a 30-second reasoning window.

“The supplier might be saying, yeah, we have a great swimming pool, but then we also have the reviews from the travelers and we actually see there’s two reviews that say the swimming pool was not great or was not open after 6 p.m.,” Amatriain explained. A generic chatbot, he added, would only surface what a supplier self-reports, while Expedia cross-references against its own review corpus.

“We don’t want the agent to book the hotel or to buy you a plane ticket for you,” Amatriain said. “That’s something that the user has to have the agency. And the agent can recommend, can suggest, can discuss with you, but you’re gonna have to hit that click. And that’s non-negotiable.” That constraint, he argued, is also a security decision. “Once you establish those design principles, you also don’t need the guardrail because otherwise you’re gonna have to put all those guardrails in after the fact.”

The next attackers will be other AI systems

“Security needs to be a principle that is shifted as left as possible and as part of the design itself,” Amatriain said in response to an audience question. “And usually when you need a guardrail is because you’ve not thought about it early on.”

A second audience member pressed for lessons learned from production. Amatriain described a feedback loop where monitoring signals flow back into the eval suite. “You can almost automate the whole cycle,” he said. “But having that whole feedback loop from real signals, from your operating AI system, all the way into being reported and fixed as quickly as possible is going to become essential.”

Amatriain's toll gates are a bet that governance calibrated to risk can stay ahead of that feedback loop. VentureBeat’s separate June Pulse survey on agent security, drawn from 107 enterprises, shows how thin that margin is. More than half, 54 percent, have already had an agent security incident or near-miss. Fifty-nine percent plan to adopt, add, or replace agent security tooling within 12 months, and 29% plan to move this quarter. Incident rates climb with organization size, reaching 63% among enterprises with more than 1,000 employees versus 49% for companies with 101 to 1,000. And sandbox isolation, the one post-breach control that limits damage, drops from 35% adoption at the smaller companies to just 20 percent at the largest.

Amatriain warned that threats will increasingly come from other AI systems. “You’re gonna get threats coming not only from humans but also from other external agentic systems that are really powerful, and they’re gonna be poking at everything you’re doing. And as soon as you detect something, it’s not only about the detection, but the time to fix becomes essential here.”

Presented by Atlassian

Most companies are approaching AI adoption backwards by optimizing how individuals use AI instead of how teams work together, said Dr. Molly Sands, head of the Teamwork Lab at Atlassian, during a fireside chat with VentureBeat senior technology contributor Sam Witteveen at VB Transform 2026.

Sands leads a team of behavioral scientists and psychologists who study how AI is reshaping the way people work together, using those findings to help organizations redesign how work gets done.

"We don't just study it, we also actively go in and change it," she explained. Her teams teach new ways of working and remap how work flows across companies, a challenge that many organizations are still struggling with, she said.

Why AI speed isn’t translating into ROI

Atlassian's annual State of Teams Report, which this year surveyed 12,000 global knowledge workers and interviewed roughly 200 Fortune 1000 executives, found a significant disconnect between activity and value, showing that everyone is using AI, while very few can yet locate where it pays off.

"89% of those executives told us that individuals are speeding up in their companies, and only 6% of them said they could point to specific examples of clear ROI," Sands said.

But roughly 14% of teams had translated AI usage into real value — meaning a single organization could contain a handful of high-performing teams surrounded by others seeing no return at all.

Those leading teams shared three characteristics: context, workflows and culture. The teams pulling ahead were building what Atlassian calls a context graph by capturing goals, decisions, and organizational knowledge in shared digital records rather than leaving them in individual memory. Across products such as Jira and Confluence, the graph connects work items, goals and the people doing them, giving AI access to the organizational context it needs.

On workflows, the winning teams redesigned entire end-to-end processes rather than simply accelerating isolated tasks. Otherwise, speeding up individuals who are pointed in slightly different directions only causes them to “very quickly start to crash into each other,” as Sands puts it.

On culture, the fastest-moving teams worked under leaders who explicitly encouraged learning and experimentation, while making it clear that some experiments would fail.

How leaders can move AI from individual hack to team advantage

Experimentation and constraints are the fastest route to learning, Sands said. The teams seeing the biggest gains were deliberately imposing constraints on how they worked, from breaking every task into the smallest practical unit of work (a single story point) to committing to write no code by hand for a week.

"Most of it is not sustainable to do forever, but it is a really, really fast way to learn," she said.

Sands argued that another obstacle isn’t the technology itself but the fact that employees are figuring out AI on their own. Every worker develops different prompts, agents and assumptions, creating another layer of unspoken knowledge inside teams that rarely translates into organizational performance.

To counter that, Atlassian experimented with AI working agreements at the start of projects, asking teams to decide not only what they would use AI for, but what they would deliberately avoid using it for, which agents they would share and what common skills would keep everyone working from the same context. Teams that adopted the practice used AI more, moved faster, made better decisions and ultimately produced higher-quality work.

The broader lesson, Sands said, is that AI isn’t creating entirely new management problems so much as exposing old ones. Teams have always struggled with hidden assumptions and different mental models of their work. AI simply makes those gaps more consequential, increasing the importance of shared context and explicit ways of working.

Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.

A single AI agent conversation can look flawless scored on its own and still point to a broken product. That gap is driving a shift in how enterprises evaluate agents, away from scoring individual traces and toward comparing cohorts of users against a baseline.

At VB Transform 2026, Harrison Chase, CEO of LangChain; Hui Zhang, CTO and co-founder of Conviva; and Emmanuel Turlay, director of engineering at CoreWeave, described that shift, along with a parallel move toward cheaper, narrower judge models.

Agent-as-judge — judging one AI agent's output with another — hasn't replaced LLM-as-judge, which Chase said remains the default. The larger tension, Zhang said, is between automated judging, whether by LLM or agent, and human review.

"You have scalable but ungrounded, whether it's agents as judge or LLMs as judge, you grade the outcome, you grade the work. It still is very difficult to ground it and then you use humans and that's just not scalable," Zhang said. "The whole industry is facing this, which poison you want to pick."

Evaluation criteria now function as the product spec

That gap — a conversation that scores well but still signals a broken product — is what teams try to close by building an exhaustive evaluation suite before they ship anything. Chase said that doesn't work.

"We sometimes see teams that have almost eval paralysis," Chase said. "They're like, this is an eval set, I can't launch it. The best teams launch and then iterate."

Chase framed evaluation criteria as a living specification, not a one-time test suite: a product requirements document — the standard software-development spec for what an application should do. "Evals are like the new PRD," he said. "They define what your agent should and shouldn't do."

Turlay described hitting the same failure from a different angle. "I was trying to reach 100% coverage for my tests, and I still had bugs in production," he said — a test suite that looked complete but still missed what mattered, the same gap Chase was describing with evals.

Broad, always-on monitoring, he said, catches more real failures than an exhaustive pre-launch test suite. Teams should set up wide online checks first, use those to identify failure classes as they occur, then build a targeted offline evaluation set around the problems that surface.

Why scoring traces one at a time is a mistake

Even a well-built evaluation process can still score the wrong thing. Zhang's objection is to how most teams run evaluation: sampling traces, whether 50 of them or a full population, scoring each in isolation. That approach misses a signal that only shows up when comparing cohorts of users against a baseline, a method Zhang calls contrastive analysis.

Zhang illustrated it with a retail example: a shopper asks an agent for a running shoe ahead of a half marathon, the agent asks qualifying questions, and the shopper buys a shoe. Scored individually, that interaction looks fine. But the clarification ratio, how many follow-up questions an agent asks before completing a task, came in three times higher than baseline for that shoe category across the full user population. A second metric, how often shoppers finished their purchase outside the conversation, was five times higher than baseline for the same category.

Neither number is visible from a single trace. Both point to a debuggable, category-specific problem. Zhang said the industry also lacks a second data source: what happens before, between and after the conversation, not just the trace itself.

Sizing the judge to the job

Once contrastive analysis flags which category is actually broken, the next problem is what watches for it going forward — and at what cost. Turlay's rule was to start with the most capable model available to prove a task is solvable, then work down. If it can't be done with a top-tier model, he said, it won't work with a smaller one. Once a pattern proves viable, teams can sample a fraction of traffic instead of judging every interaction, and move simpler tasks like binary classification to smaller open source models.

LangChain took that further, fine-tuning its own model to detect when a user believes the agent made a mistake, a signal Chase calls perceived error. "The model we fine-tuned was a Qwen model," he said, referring to Alibaba's open source family. Combining hand labeling with distillation, the result performed well. "Same as [Claude]Sonnet, for, depending on how we served it, either 10 to 100x cost reduction," Chase said.

Not every guardrail needs a model. Chase pointed to Claude Code's own guardrails as proof: regexes, the common programming technique for finding and validating patterns in code. "A lot of the guardrails they had were just regexes," he said. "They weren't small LLMs, they were just regexes."

LLM-as-judge doesn't mean human-in-the-loop disappears

The bigger question is whether using LLM as a judge removes the need for a human in the loop.

Turlay pointed to accountability, drawing on his prior work at a self-driving car company. His team compressed data intake and retraining into a two-week cycle for shipping a new model to the car. Even then, someone still had to sign off.

"I felt confident on behalf of the company to say this model should go into the car," he said. The same logic extends to legal, finance and healthcare. "Before we can remove a human to say, I endorse this and I take responsibility legally for it, it's going to be a while before agents can do that on their own."

Zhang agreed a human has to remain the guardian on corner cases, even as automation eventually runs at a scale that beats individual human accuracy — machines can see more at the pattern level.

Chase went further: that human check isn't just a safety net. "Human in the loop is really important for building trust in how these agentic systems work, and also really important for memory and learning from systems," he said. "There has to be interactions in order for the system to learn."

Zillow, the real estate technology company, doesn't get one conversation with its customers. They move from a phone screen to a loan officer to a real estate agent, sometimes over months or years, and expect the context to follow them. A single chatbot could never carry that thread.

At VB Transform 2026, Zillow SVP of Engineering Toby Roberts and Glean co-founder and CEO Arvind Jain described how they built AI architecture meant to carry context across that entire journey — and why context, not raw data, turned out to be the harder problem to solve. Zillow's products touch roughly 80% of U.S. real estate transactions each year, and the company has been using AI long before ChatGPT existed.

"We pretty quickly identified that we were going to need a persistent context layer that was going to meet our customers and the professionals wherever they were," Roberts said.

Data was never the hard part

Roberts said Zillow's AI effort started where most enterprise AI efforts start, with the data itself.

"We started with a large push around making sure our data did have the right foundation," Roberts said. That meant a data mesh approach, clear data lineage and a governance structure with permissions and identity attached to the data itself.

None of that turned out to be the hard problem. The hard problem was building something that remembered where a customer was in their journey and carried that forward, no matter which surface they showed up on next.

"This context layer has to live to be able to support you where you are at any given point in your journey," Roberts said. Zillow chose to own that layer itself rather than depend on a single external chat interface, a decision Roberts said the team reached quickly once it looked at the shape of a real transaction rather than a single conversation.

Why Zillow built its own architecture, and where Glean fits into it

Zillow built its own harness rather than route customers through a single model API. The team drew on 20 years of machine learning history behind products like Zestimate, leaning into smaller, task-specific fine-tuned models instead of one general-purpose model.

Internally, that harness runs alongside Glean. Roberts said Zillow now has thousands of Glean agents in production, handling repetitive tasks with tens of thousands of executions across the company. Glean's pitch, per Jain, is centralizing that integration work once, through the Glean MCP gateway, rather than letting finance, legal and marketing each rebuild their own connections to the same systems.

That centralization is also a cost lever. Jain pointed to two mechanisms: model routing, which sends most tasks to smaller, cheaper models instead of defaulting to frontier models, and precomputed context, which avoids an agent burning tokens assembling its own context from scratch.

"Claude is also very slow because the first part of assembling that context actually takes forever," Jain said. Routing that request through Glean instead, he said, can cut token consumption by as much as half.

What Zillow and Glean's approach means for enterprises

Across data, cost and permissions, the session offered a few practical takeaways for enterprises building agentic AI on their own systems.

Build the measurement baseline before the AI push, not after. Roberts said Zillow's ability to credibly attribute a 40% increase in shipped code to AI adoption rests on a DORA metrics baseline the team put in place years earlier, not on the AI rollout itself.

Centralize context once instead of letting every team rebuild it. Jain's core argument for Glean's platform is that duplicated integration work across finance, legal and marketing teams is a hidden cost most enterprises haven't accounted for.

Don't assume permission inheritance is enough for regulated data. Even with a permissions-aware context platform in place, Zillow layered hard rules and a standing compliance check on top for its most sensitive categories, rather than trusting the architecture to handle it automatically.

Treat context as a cost lever, not just a capability. Model routing and precomputed context were the two mechanisms Jain pointed to for cutting AI spend, both aimed at reducing wasted token consumption rather than adding new capability.

"Models by themselves are not enough to bring automation with AI inside your enterprise," Jain said. "You do have to connect it with your enterprise context."

Intuit was an early pioneer in the usage of agentic AI, but its path to success has hardly been a straight line.

At VB Transform 2026, Intuit VP of AI Nhung Ho described how the company rebuilt its agent architecture twice in the span of about four months, first moving from a fleet of specialist agents to a central orchestration layer, then abandoning that layer for a skills and tools based system once the orchestrator itself started failing under its own complexity. The full second rebuild took 60 days, with a first working version in under 20.

The failure mode that forced the second rewrite was specific. Agents in the orchestrated system passed results to each other in natural language, and each handoff lost context the next agent needed to act correctly. 

"If you have 10 agents and they all are passing to each other, every time that pass happens, error compounds," Ho said.

Why the orchestration layer broke down

Ho said the original push toward specialist agents came from a straightforward customer complaint. A fleet of capable agents is still something a customer has to manage, deciding which agent to use for which task. Intuit's answer was a system that could take a task and route it internally, without asking the customer to pick an agent themselves.

That orchestration layer held up for about three months, which Ho described only half joking as roughly a year in the compressed timeline of agent development in 2026.

It broke for a structural reason rather than a capacity one. Passing outcomes between agents in natural language meant each downstream agent had to infer how the upstream agent reached its conclusion, and that inference degraded with each additional hop. A ten agent chain did not fail occasionally, it compounded errors by design.

That diagnosis is what sent Intuit back to a skills and tools architecture.

The 60-day rebuild, and what it took to get engineering buy-in

Rebuilding a production agent system in 60 days required more than an architectural decision. Ho said the harder problem was internal, convincing both leadership and the engineers who had built the original agents that scrapping recent work was the right call.

The pitch to leadership relied on evidence rather than argument. Ho's team built a demo of the new architecture using real customer queries pulled from production, then showed it performing better than the existing system on the same tasks. 

"The best proof, at least my belief, is what are customers trying to do? And whatever system you build needs to address those problems," Ho said.

Winning over engineering required a different case. Hundreds of engineers outside Ho's core team had built the specialist agents being retired, and the ask was to take their agents apart into individual skills and tools instead. 

Ho said the motivating argument was scale. A standalone agent solved one narrow problem, while a shared skill or tool built into the new architecture could serve every customer who touched that part of the product. That shift also changed what partner teams were responsible for day to day, moving their focus from building agents to running evals, since evals became the only way to measure whether the new architecture was actually working.

Bringing a human into the loop, and feedback at a different scale

The clearest customer facing result of the rebuild is a feature that lets a live agent conversation pull in a human — though it's currently in early testing, live to about 1% of Intuit's customer base. "We're going to be scaling it up in the next few weeks," she said.

Ho said a customer can bring in an Intuit product support person mid conversation, or their own accountant, or one of Intuit's own bookkeepers, and that person joins with the full context of what the agent has already done.

Ho drew a direct contrast with how most AI chat products handle the same situation. A general purpose assistant answering a tax question typically ends with a disclaimer to consult a professional. Intuit's system is built to connect the customer to that professional directly, inside the same conversation.

That human handoff sits alongside a permissions model built for financial data specifically. Every action an agent takes on a customer's financial data requires explicit permission first, though Ho said that requirement can ease over time as customers build trust in the system. Intuit keeps an audit log of everything an agent does that can be reversed if needed.

Feedback in the agentic AI era

The rebuild also changed how Intuit gathers and uses feedback, a shift Ho said is qualitatively different from what came before. 

"Feedback in the past used to be very, very sparse, and it was also very bimodal," Ho said. "Either they loved it or they hated it, and usually it tends towards the negative."

In a chat based system, every conversation functions as feedback, which Ho said moved the company from roughly 0.3% of customers ever giving explicit feedback to something close to 100%.

Ho said she has returned to writing code herself specifically to build models that analyze that feedback volume systematically, looking for where the system is falling short at a scale no manual review process could keep up with.

That volume comes with a tone most product teams aren't used to hearing directly. Customers tell the agent exactly where it failed, in plain terms.

"They straight up tell you, 'You suck. I hate this. This is not right,'" Ho said. "But they're also willing to give the systems grace and correct it as well, and so the onus is on all of us to harvest this new piece of feedback and type of feedback, and actually improve the system."

Legacy infrastructure, not the models themselves, is what's actually slowing AI agents down. That was the shared conclusion of three infrastructure leaders — from LinkedIn, Walmart, and Zendesk — at VB Transform 2026.

The panel brought together Animesh Singh, senior director of AI platform and infrastructure at LinkedIn, Desiree Gosby, SVP of corporate technology services and technology strategy at Walmart, and Sami Ghoche, VP of applied AI at Zendesk, each describing what actually broke when they moved agents from pilot to production. Each arrived at the same conclusion from a different starting point: None of the bottlenecks they hit were model problems.

What tied their answers together was a shared premise: most enterprise infrastructure was built for how humans work, not for how agents work. The gap between those two speeds is where the real engineering happened.

Gosby put it plainly when asked what she'd learned scaling agents inside Walmart's own workforce. The goal, she said, is to make sure "engineering doesn't once again become the bottleneck for what it is we're trying to do."

Where the bottleneck actually was

Each company hit a different version of the same wall: infrastructure designed for how people work doesn't hold up once agents are doing the work instead.

At LinkedIn, the first bottleneck wasn't a model, it was Kubernetes, which assumes containers spin up on demand, a process that takes seconds. Singh said that's too slow for agents. The fix was moving from on-demand provisioning to pre-provisioned pools of containers that swap agentic workloads in and out in real time.

A second, harder problem surfaced once LinkedIn let agents control their own orchestration. A five-point evaluation system looked clean, but hallucination kept showing up anyway. Singh said the issue was structural, an LLM evaluating another LLM's output shares the same failure mode as the thing it's evaluating. 

"We built our own harness, our own control flow, and pushed the LLMs to the leaf instead of them orchestrating the loop," Singh said. Roughly 80% of the workflow is now scripted, deterministic code, with LLMs used only where reasoning is required, and each step's evidence is committed to disk before the system moves on.

Walmart's bottleneck came from success. An agent harness put directly into employees' hands went viral internally, and what Gosby called "citizen developers" began building their own agents to solve problems that once required a formal engineering roadmap. The upside was real innovation. The downside was duplication, dozens of overlapping agents with no coordination. The fix wasn't reining in the harness, it was building governance to spot duplication, promote the best version of an agent, and get it into production without engineering becoming a chokepoint.

Zendesk hit its bottleneck from the data side. Ghoche, who joined through Zendesk's acquisition of Forethought, which closed in March 2026, described sitting on what he called a public figure of 20 billion customer conversations in Zendesk's repository. The instinct is to hand that history to a large language model with a big context window and let it generate the agents a business needs. Ghoche said that doesn't work. "You can't really do that, so instead you have to really invest in the underlying data pipelines and all the data infrastructure that comes with that," he said.

The role of open source

On open source, all three leaders landed on a similar instinct: own what you can, and lean on frontier labs only where they still have a clear edge.

Ghoche said his own view is that most enterprises would prefer to own their models and infrastructure wherever that's possible, and that reasoning is what drives Zendesk's own approach. The exception is frontier reasoning work, where the labs still lead, though he said that slice of use cases is shrinking relative to everything else enterprises now do with AI.

LinkedIn's answer was to build two subsystems specifically for independence. The first is what the company calls an AI gateway, a single interface that every outbound call to a model runs through regardless of provider. The second component is a memory subsystem built to hold context independent of any model provider.

"Every single outbound call going to an LLM, whether it's on a public cloud or on-prem in our own data centers, follows the same semantics, the same API calls. We can quickly switch between different providers," Singh said. 

Walmart built its own internal gateway to stay vendor agnostic across three workload types: fully deterministic workflows, planner-and-reasoner workflows for open-ended tasks, and a hybrid of the two. Compliance-heavy work stays deterministic by design; governance, security and evaluation run through the gateway regardless of which model is on the other end. Gosby said the choice between a frontier model and an open-weight model comes down to whichever is most effective for the specific workload, not a fixed policy.

Advice for the modernization journey

Three pieces of advice came up directly, each tied to the wall a leader had already hit.

Invest in evals before anything else. Ghoche called it the thing common to every use case, internal or customer facing. 

"The thing that's common to all of these is evals. It'll force you to break the problem down, and once you have a robust set of evals, you can move a lot faster," he said, 

Own your agent harness from day one. Gosby's advice was to put the AI harness directly in employees' hands early, paired with the infrastructure to monitor what it produces. 

"It will unlock a huge amount of innovation," she said.

Build for model and context independence. Ensuring flexibility is critical for success.

"Build for independence, whether it's a frontier model of today versus an open source model of tomorrow," Singh said. "Keep that context within your enterprise so that you can reuse it when you ship the model or the harness tomorrow," Singh said.

The enterprise AI industry has a math problem. Cisco data shows 85% of enterprises are piloting AI agents, but only 5% have shipped them to production. At VB Transform 2026 on Tuesday, Bryan Silverthorn, Director of AGI Autonomy at Amazon, explained why that gap persists — and why the answer isn't better benchmarks.

Silverthorn, who joined Amazon through its acquisition of Adept AI and now leads multimodal agent training inside the company's AGI lab, argued that reliability must be broken into four distinct dimensions: consistency, robustness, predictability, and safety — a framework he credits to research from Princeton.

"It unpacks different factors that I see tangled together in almost every eval I've ever seen," he said.

Why AI agents pass internal evals but fail real customers in production

The framework matters because agents routinely ace internal evaluations and then collapse in the wild. Silverthorn described a customer that deployed an agent for software QA involving serial number extraction from screens. It worked flawlessly for two months — then began intermittently reading wrong numbers. The culprit: the underlying vision encoder behaved differently depending on where the serial number appeared on screen, and a software change imperceptible to humans triggered the failure.

The lesson, Silverthorn said, is about measurement, not just models. "The models have to be better. Obviously, we're working hard on making the models better," he said. But the deeper takeaway, he added, is that teams need to identify their dimensions of variability and match measurement rigor to the stakes of the application. VentureBeat's own proprietary research, presented before the session, reinforces the point: half of surveyed companies shipped agents that passed internal evals but failed real customers, and enterprises overwhelmingly track uptime while ignoring accuracy — checking the pulse without checking the diagnosis. A related finding underscored how few guardrails exist: most enterprises default to the model makers' own evaluations and little else, leaving their testing strategy, as I described it on stage, a coin flip between trusting the vendor and trusting nothing.

Inside Amazon's 'intern' framework for managing autonomous AI agents

Silverthorn's most memorable prescription was cultural, not technical. Inside Amazon's AGI lab, researchers literally call their agents "interns" — as in, "I'll have my intern talk to your intern." The joke carries a serious operational philosophy. Agents, like interns, are powerful but occasionally clueless, capable of amazing work and spectacular derailment.

Managing them, he argued, requires management skills rather than software skills: asking what could go wrong, adding backups and undo capabilities, and consciously deciding what risk you can accept. "You can ask the intern, 'Hey, what might you do wrong here? How might you mitigate your negative outcomes?'" he said. Amazon's lab has embraced that trade-off, accepting agents occasionally running the wrong experiment in exchange for research velocity — including one agent running experiments around the clock on its own high-level research plan.

What enterprise leaders should do before deploying agents at scale

Silverthorn was candid about the limits of today's technology. Self-improving AI remains "a loaded term," he said — Amazon uses AI to improve its models constantly, but fully autonomous self-improvement is distant. Computer use remains a core focus of his lab, with a commercial trucking customer already using browser automation to stitch together warranty claims across fragmented systems**, though he stressed that no future agent will rely on computer use alone — it will work alongside MCP, APIs, and other tools to complete end-to-end workflows**. And LLM-as-judge techniques, while promising, are just one of several strategies for aligning agent capability with acceptable risk.

For enterprises stuck in pilot purgatory, the path forward starts with a mindset shift: stop asking whether your agent can do something impressive once, and start asking whether it can do it correctly a thousand times in a row.

In other words, the enterprises that escape the 85% ceiling won't be the ones with the smartest agents. They'll be the ones with the best managers.

Hundreds of enterprise leaders and technical experts packed the main ballroom of the luxurious Hotel Nia in Menlo Park this week for VB Transform 2026, the year's preeminent conference on using generative AI agents to drive business outcomes.

Rachad Alao, vice president of product engineering at the rising Canadian enterprise AI startup Cohere, joined VentureBeat CEO and editor-in-chief Matt Marshall for a fireside chat about building agentic systems without surrendering sensitive data, infrastructure control, or the ability to change vendors.

Alao, who previously led responsible AI and trust and safety engineering teams at Google and Meta, argued that AI sovereignty means more than downloading an open model or running an application behind a corporate firewall.

Asked how Cohere defines sovereignty, Alao pointed to organizations operating mission-critical systems, including banks, hospitals and governments.

“It is important to have very tight control on where the data resides, have tight control on the AI,” he said, adding that AI operations should take place in jurisdictions an organization understands or directly controls.

That extends from GPUs and private-cloud infrastructure through governance systems that route requests among models, as well as the connectors, search tools and agent frameworks acting on enterprise data.

“You want to have control on the entire stack,” Alao said.

Agent workloads could outrun falling token prices

Marshall challenged one of the central economic arguments for smaller, locally deployed models: Inference prices continue to fall rapidly, potentially weakening the case for optimizing every token.

Alao countered that total consumption is climbing even faster as enterprises move from relatively simple chatbots to agents that reason through problems, call tools, search internal systems and take multiple steps before returning an answer.

“Your token utilization is going exponentially up, because you’re dealing with more and more complex agentic use cases,” he said. Those workflows require “a lot of processing, thinking, tools interaction” to complete their objectives, he added.

Alao also drew a contrast between providers that bill customers according to token consumption and Cohere’s approach.

“If your whole way of charging customers is for token utilization, you want to maximize token utilization,” he said. “We do not sell our models and our platform that way.”

Instead, Alao said Cohere tries to help enterprises solve their hardest problems privately and securely while reducing unnecessary model usage. His prescription was straightforward: “Use the right model for the task at hand.”

Rather than sending every request to the largest available frontier model, enterprises should route work according to the intelligence required and the sensitivity or regulatory burden attached to the task.

Alao cited an unnamed Canadian bank that uses Cohere’s on-premises models for highly regulated workloads, while sending less sensitive tasks requiring greater intelligence through Cohere’s North platform to larger frontier models.

“So model routing can become super useful,” he said.

Smaller models for most enterprise work

Asked by an audience member how Cohere’s open-source North Mini Code, released last month, could compete against proprietary coding models, Alao acknowledged that larger frontier models may perform somewhat better on the hardest tasks.

But that advantage may not justify using them indiscriminately.

“For 80% of the use cases that they needed, this was a lot more effective, a lot cheaper,” Alao said of developers adopting the model.

Cohere’s North Mini Code runs on a single Nvidia H100 GPU and targets agentic software engineering, including terminal work, code review and tool use.

The company has also released Command A+, a 218-billion-parameter mixture-of-experts model with only 25 billion parameters active during each generation step.

Its compressed four-bit version reduces the hardware required for private deployment, while its Apache 2.0 license gives enterprises broad freedom to operate and modify it.

Search becomes part of the agent

Asked about Cohere’s longstanding work on embeddings and enterprise search, Alao said the field is moving beyond retrieving text and inserting it into a model’s context window.

“Today, the state of the art is around multimodal search,” he said. “It’s beyond just the text modality.”

Search across documents, images and other forms of information is becoming “an integral component of your agentic workflow,” Alao added, with the model deciding when and how to use retrieval like any other tool.

Asked what would persuade enterprises to move beyond bundled AI services from existing cloud providers, Alao returned to data control and portability.

“If you’re interested in sovereignty, you want to have more control on your data,” he said. Cohere’s governance layer, he added, lets customers route traffic to appropriate models, “breaking that vendor lock-in concern that a lot of our customers have.”

Organizations need to transform to meet the needs of agentic AI.

Meta VP of Engineering Barak Yagour opened his talk at VB Transform 2026 wearing a pair of Ray-Ban Meta AI glasses, a small sign of how far AI has already worked its way into physical life. His argument went further: enterprise infrastructure was built for humans, not for agents, and it's starting to show.

Yagour, who leads its data infrastructure organization, told the audience that agentic queries hitting Meta's data systems grew 30x in a single half, an inversion that he said is breaking assumptions the company spent two decades building around.

The shift is not confined to Meta. Automated traffic overtook human traffic on the internet last year, reaching 51% of the total, according to Imperva's 2025 Bad Bot Report. That traffic is also growing roughly eight times faster than human traffic, according to HUMAN Security's 2026 State of AI Traffic report. Yagour cited both figures to describe what he called an inflection point already underway inside his own organization.

Yagour framed the shift as an open question for infrastructure teams everywhere. "What happens to the infrastructure we've spent years building when agents and not humans become the main consumers of that," Yagour said. "That's the world we're stepping into."

Capacity, identity and velocity are breaking at once

Yagour said three assumptions are breaking simultaneously inside Meta's infrastructure: capacity, identity and velocity.

On capacity, the math no longer works the way engineering teams are used to. "One engineer used to mean one unit of load," he said. "Now one engineer spawns 10 agents, each spawning subagents. Your 1,000-person org can generate the load of 100,000 users practically overnight."

His answer is not to block agent traffic but to make infrastructure agent-aware, with dynamic controls that understand agent hierarchies, cost attribution that traces consumption back to the use case that spawned it, and throttling that adapts based on priority.

Identity is breaking, too. Yagour said an agent does not fit the categories infrastructure teams built access controls around. It is not a human user, it does not carry a badge and it is not a deployed service, yet it makes decisions on its own.

Velocity is the third assumption under strain. Yagour cited a company-reported figure that GitHub Copilot writes 46% of the average user's code, then noted that faster code generation does not make the rest of the pipeline faster.

"That code still needs to be built, tested, deployed, monitored," he said. "The agent writes the code in seconds, but your CI/CD pipeline doesn't get faster just because the machine is the author."

Trusted data environments keep agents inside guardrails

Data is where Yagour said the pressure from agents is most direct. 

"Data sits at the center of everything," he said, pointing to the decisions, products, recommender systems and next generation models it drives.

Meta is also rethinking how much autonomy to grant agents inside its own data systems. In February, the company shipped what Yagour called agentic data apps. Within three months, 63% of dashboards published across Meta were built using the new tooling, part of the same 30x rise in agentic queries Yagour cited earlier.

That growth raises a governance question. Human analysts have traditionally sat between raw data and business decisions, curating it and serving as an informal check on quality. Yagour said Meta wants to grant agents more independence on harder problems, but was direct about the risk. 

"Autonomy without governance is nothing but chaos," he said. That's why the company built what it calls trusted data environments, to preserve the human check as agents take on more of that work.

"Inside, the agent can explore data freely, but every output is traced back to its source and scrutinized. So you always know that the data shared back is trusted and governed," Yagour said.

Sensitive fields are masked before an agent can reach them, and every access request is evaluated in real time against what the agent is trying to reach, why and whether it is allowed. Yagour summarized the approach as exploring broadly while releasing narrowly.

Reasoning models are rewriting the data layer

Meta's models are also demanding more from data as they shift from correlation to reasoning. 

"Reasoning is data hungry," Yagour said. 

Pattern matching works on sparse, summarized signals. Reasoning demands the full behavioral history, every interaction across every surface over time. Yagour pointed to two shifts already underway inside Meta's infrastructure to keep up.

Real-time streaming is replacing batch ETL for ranking pipelines. A pipeline that takes 24 hours to run is not viable when a model is reasoning about a user's current intent. Yagour said real-time streaming, not batch extract-transform-load processing, is becoming the backbone of Meta's ranking and recommendation systems.

Storage is becoming schema-aware to stop GPU starvation. Meta previously stored user data as opaque blobs with no awareness of what the data contained, which Yagour said led to heavy overfetching and idle GPU capacity. The company is now building storage that understands what it holds, pulling only the columns and time ranges a given query needs. Yagour said Meta is building toward 500 million queries per second and a petabyte per second of throughput for training data reads.

That data feeds directly into how Meta's recommendation systems behave. Yagour said 42% of Instagram users have told the company they want to fundamentally change the algorithm, not adjust a single session or setting. Meta's response is what Yagour called fully conversational recommendations, where a user tells the system what they want more of and it reasons about intent rather than matching on keywords. Yagour said the same search term, soccer, would return different results for a casual fan looking for highlights than for a club athlete seeking training drills, because the system would reason about which one is asking.

Yagour described the three threads of his talk, agents, data and recommendations, as reinforcing each other rather than moving independently. 

"Agents make data more accessible. Better data makes reasoning. Reasoning creates new demands that push agents and infrastructure forward," he said. "This isn't linear; it's a flywheel."

During the Q&A, an audience member asked whether Meta's push toward more intelligent infrastructure signals the end of traditional file systems in favor of newer neural storage approaches, and whether agents will keep using SQL as their interface to data the way humans do. Yagour said Meta is experimenting at every level, including questioning whether SQL is the right interface for agents at all, and that storage at Meta's scale already operates in the multi-digit exabyte range and needs to keep expanding.

Yagour closed his talk with the timeline he believes the industry is working against. "We spent 20 years building infrastructure for humans. We have maybe 20 months to rebuild the whole thing for a world where humans and agents co-create at scale," Yagour said. "The window is open, but it won't stay open for long."