Google keeps cranking out Flash models: the company on Wednesday announced two versions of a new 3.8 Flash. 

The variants include a standard Flash, a “workhorse” model for agentic tasks, software development, and multi-step reasoning, and Flash Cyber optimized for vulnerability detection and mitigation.

Google CEO Sundar Pichai said in an X post that 3.8 Flash delivers “significant leaps” from 3.7 Flash across software engineering, agentic tasks, and multi-step reasoning. For instance, it outperformed many large frontier models on the DeepSWE coding benchmark, at far lower cost. 

Meanwhile, Flash Cyber is the company’s “most capable” cybersecurity model, Pichai said; it also matches frontier-level performance when it comes to discovering vulnerabilities and patching them at scale. The model achieved 86.2% on the CyberGym cybersecurity benchmark and 47.2% on CWE-Bench, which evaluates AI patching abilities. In an internal Google benchmark, the model achieved a more than 70% success rate discovering vulnerabilities across 20 programming languages, Pichai said. 

3.8 is Google’s third Flash release in six weeks and comes quickly on the heels of version 3.7. 

3.8 working "harder" with "greater diligence"

3.8 Flash is available now in Gemini Enterprise; devs can try it out in the Gemini API via Google AI Studio, Google Antigravity, Android Studio, or generate UIs in Stitch. It is priced at $0.75 per million input tokens and $3.75 per million output tokens — the same introductory pricing as Gemini 3.7 Flash — and users can customize and adjust model effort levels based on their needs around quality, cost, and latency.

For instance, when compute efficiency is a priority, they can adjust to lower token overhead, or simply continue working with 3.7 Flash, which is “fully supported for efficiency-first workloads,” Google senior product director Tulsee Doshi and Gemini security lead Raluca Ada Popa wrote in a blog post. 

“3.8 Flash works harder,” exhibiting “greater diligence” with complex tasks like executing extra reasoning steps, although at times it may use more tokens to maximize performance, Doshi and Popa note. The model has a 1M-token input window and a 64K-token output limit, and can ingest text as well as images, audio, video, and PDF files. 

3.8 Flash was evaluated across numerous benchmarks testing coding, multimodal capabilities, computer use, long-context and knowledge work, and scientific reasoning. Google says it also does well in specialized knowledge domains requiring more in-depth analysis and reporting. For instance, the model outperformed its predecessor and other frontier models on benchmarks like Vals Finance Agent V2 for finance, and Harvey's Legal Agent Benchmark for law; it also scored 54.9% on Humanity’s Last Exam (HLE)-Verified, reflecting its ability to take on multi-step reasoning tasks across subjects like math, science, and humanities. 

In one example shared by Google, Gemini 3.8 Flash built a game with a simple prompt using looping techniques in Google’s Antigravity platform. The game uses puzzles, storytelling that changes based on the environment, and images and textures from Nano Banana to create a 3D experience (in this case a wizard navigating a castle). 

In other instances, the model created a fully-functional DOS version of Google Maps featuring interactive locations, directions, and street views; a 3D visualizer that automatically decomposed devices into layers for inspection with a slider capability; and a topographic map of famous geographical sites based on real datasets from the U.S. Geological Survey, complete with real-time cross-sections, 2D projections, and scientific explanations. 

According to Arena.ai, 3.8 Flash landed at No. 14 in Agent Arena, ranking above DeepSeek-V4-Pro, and showed a significant jump over Gemini 3.7 Flash (which sits all the way down at No. 32). It debuted at No. 7 in Text Arena, ahead of Claude Opus 5 and Gemini 3.7 Flash. It improved over 3.7 Flash in several areas: multi-turn requests, writing, literature, and language, longer queries, hard prompts, coding, instruction following, software and IT services, and business, management and financial ops. 

Flash Cyber is already securing Google's code

Flash Cyber is initially being rolled out to “trusted defenders” through Google’s Fairwind Program, which prioritizes government authorities, critical-infrastructure operators, and other partners looking for advanced cyber defense capabilities. Organizations can apply for access. 

Google says the model version has undergone “rigorous training” in the cybersecurity domain and represents a “significant leap in prompt injection robustness.” It is particularly adept at autonomous vulnerability discovery — at least, based on internal Gemini benchmarks — and automated patching. It is also very good at coding, Popa said in a video. 

The goal was to equip defenders with expert-level capabilities to give them a leg up over threat actors (whether malicious, fellow AI agents, or human hackers). “We have invested in vulnerability fixing from the start, and prioritized it over offensive capabilities like exploitation,” Doshi and Popa explain. 

The model ships a more permissive set of mitigations for cybersecurity safeguards — which is why, for now, it is only being shared with limited partners — and safeguards against misuse in cyber offense and areas like chemical, biological, radiological, and nuclear (CBRN).

Google is already using 3.8 Flash Cyber to secure its own code; it produced 2.6 times more correct patches in Chrome vulnerabilities versus much larger commercial models. 

Wiz — which Google acquired earlier this year at a historic $32 billion — reported that 3.8 Flash Cyber had 7.5% to 9.7% higher recall of real-world vulnerabilities on an internal penetration testing benchmark at 2.3 to 5.2 times lower cost than leading frontier models. Similarly, Google’s Cloud Vulnerability Research found a critical foundational vulnerability in less than 2 hours with 3.8 Flash Cyber. Typically, that research and discovery would take months, Google claims. 

AI agents are “incredibly skilled” at finding and exploiting vulnerabilities, Popa said. Scanning large codebases with big AI models is expensive, and defenders are overwhelmed. “In cybersecurity, attackers need only find one significant flaw over millions of lines of code. Defenders have to remove every one of those flaws to be able to defend against attackers.” 

Doug Turner, engineering director for Chrome, described a “vulnerability apocalypse” in recent months due to generative AI. “Simply overnight, we saw a hockey stick increase in the number of software vulnerabilities reported through our vulnerability research program,” he said in a video. 

One interesting vulnerability 3.8 Flash Cyber discovered had been in Chromium and Chrome for 13 years, he explained. It was a “very subtle bug” that dozens, if not hundreds, of engineers looked at but never flagged. “Gemini 3.8 Flash Cyber is going to allow us to create better suggested fixes so that developers’ lives can get a lot easier.” 

Enterprise AI teams have stopped betting on a single orchestration platform. The median enterprise now runs three at once — not by accident, but because none of them fully trusts a single vendor to run the show, according to VB Pulse data.

This is not just to avoid vendor lock-in and retain flexibility (although that’s a big part of it). There’s still a lot of uncertainty, even distrust, in vendors’ security and permissioning capabilities. Enterprises want the ability to impose their own.

Microsoft leads on primary usage today, while Anthropic leads by a wide margin in what enterprises are considering next. But enterprises still struggle with many challenges, notably around token usage and visibility into agent spending.

These findings are from an ongoing analysis of how enterprises are actually deploying and using AI: Their platforms of choice, what guides their decision-making, what they prioritize, their AI expectations, how they control costs, and whether their AI is actually agentic or still a chatbot in an "agent" label.

VB Intelligence is getting feedback from builders actually in the trenches: software and machine learning (ML) engineers, product and program managers, and data/AI/analytics VPs and directors.

Concerns around retaining visibility and control

Across 107 enterprises, agentic orchestration has become decidedly plural. The survey found that the majority of enterprises are not committing themselves to any one model: 85% are using two or more orchestration tools; 64% are using three. Just 15% run a single orchestration platform.

Microsoft AI Foundry/Copilot Studio shows up in 70% of stacks, OpenAI’s Agents SDK in 68%, and Anthropic’s Claude Platform in 47%. Builders surveyed are also to some extent using Google’s Enterprise Agent Platform, LangChain/LangGraph, Salesforce Agentforce, Amazon Bedrock, and LlamaIndex. Augmenting vendor tools, 22% of builders run custom in-house orchestration.

This trend of hybridability is only expected to continue. More than half of respondents (53%) said the primary control plane will be hybrid by the end of 2026. Fourteen percent expect to use a provider-managed service, 13% plan on a custom in-house control plane, and 11% are betting on external platforms that are abstracted away from model providers.

Dovetailing with this, more than two-thirds of respondents plan to change platforms within the year: 15% in the next three months (or sooner), 24% in three to six months, and 28% in six to 12 months. Claude Agent SDK is a top tool under consideration; 43% of builders are exploring the Anthropic-built model. Roughly one-third are looking at Google’s Enterprise Agent Platform, another 31% are focused on custom in-house orchestration, and 25% are investigating OpenAI’s options.

Perhaps learning from the lock-in of the early cloud days, enterprises aren’t choosing one “winner.” They are deliberately building for a future where multiple orchestration platforms, models, and agents work with each other across a hybrid control plane.

Generally speaking, respondents are pleased with the platforms they’ve been running, rating them 4.17 out of 5 for overall satisfaction. But they are less satisfied with ease of implementation (rating it 3.91 out of 5) and value for the money (3.63 out of 5). Keep an eye on these ratings as orchestration platforms and AI roadmaps mature.

Where enterprises are putting their money

Enterprise buying logic is now based on a mix of several factors. Beyond flexibility (cited by 29% of respondents), top considerations include security and permissions (17%), production reliability (15%), and control over agent execution (15%). Just one out of 10 identify model gravity — native alignment with a state-of-the-art base model — as important in purchasing decisions; 8% name ease of development, 4% cite total cost of ownership, and just 2% cite latency and memory performance.

Spending also reflects enterprise priority on visibility, security, and control. Builders are investing the most in agent monitoring and debugging (31%) and security and permissions enforcement (30%). Workflow tooling accounts for another 19%. That's a shift from VentureBeat's prior wave a month earlier, when workflow tooling led orchestration spending outright.

Enterprises are largely optimizing for task completion reliability (30%), multi-step workflow management (27%), developer productivity (23%), and operational stability (13%). Just 7% of respondents name end-user experience as a top priority at this point, indicating that many are still focused on orchestration at this point rather than UX.

Essentially, enterprises are signaling that workflow succeeds when it carries multiple steps to completion. Simplifying development and end-user experiences could become a larger concern when platforms are actually in place.

The visibility problem

Builders’ biggest concerns when choosing platforms center around control and oversight. They don’t want vendors to constrain their ability to see what their agents are doing on a given platform. Factors top of mind include security and permissioning limitations (37%), vendor lock-in (23%), limited visibility and observability (22%) and inflexibility around models and tools (16%).

Meanwhile, in these early days of AI agents, enterprises still struggle to control agent token use; one in five still can’t stop a runaway agent’s spending in real time.

Builders are using various strategies to try to keep agent spending in line: 30% rely on native platform controls (built-in budget caps or throttling) and 25% have built custom gateway plumbing (proxy middleware to intercept runaway agents).

A quarter of respondents use dynamic routing to offload heavy work to low-cost models, and 21% still rely solely on reactive monitoring, such as post-hoc logs; these enterprises have no real-time kill switches.

One interesting finding: unlike the prior wave, organization size makes little difference in fiscal control maturity — 18% of enterprises with 10,000-plus employees exercise only reactive control, compared to 23% of smaller ones.

Clearly, while enterprises recognize the problem with spend, many have not yet instrumented their stacks to rein it in.

Most enterprises still aren't running true multi-step agents

Builders polled were asked to honestly assess their tech stacks; the consensus seems to be that ‘agents’ are slowly but surely progressing beyond chatbots wrapped in that fancier label.

Here’s how the numbers break down: A small number of respondents (2%) report that 76 to 100% of their systems are advanced and largely autonomous; 14% say 51 to 75% of their systems are complex, multi-agent pipelines; and 47% report that 26 to 50% of their systems are true orchestration.

On the other end of the spectrum, 35% say just 1 to 25% of their systems are true orchestration; most deployments remain basic assistants, and 3% are still only deploying chatbots.

This is in line with VB’s June Pulse survey: 71% of respondents said a quarter or fewer of their deployed “agents” can autonomously complete multi-step work, and just one-tenth say they have deployed agents at scale.

There’s no doubt that enterprises are building control planes and infrastructures for agents; but for many of them, the true agentic wave is still off on the horizon.

DeepSeek's V4 Flash has topped model leaderboards and been hailed by developers as a "total monster" since its rollout. But in real-world testing, it completed just 53.8% of a batch of complex agent tasks.

Composio ran the model through eight different agent harnesses, including Claude Code, Codex, and OpenCode, on 30 deliberately difficult, multi-step tasks spanning live tools like Gmail, GitHub, Slack, and Google Sheets. Of 240 total runs, 129 passed — and only six of the 30 workflows were completed successfully by every harness tested.

The gap illustrates why orchestration, not raw model capability, may decide whether the model succeeds in enterprise settings: the same model produced substantially different results depending on the harness, tool configuration, caching behavior, retries, and provider stack it ran on.

DeepSeek said it will be hiking the prices for V4 Flash and Pro, models that have quickly become favorites among developers building coding assistants and agents.

While it seems the move might undercut its very appeal — strikingly capable models at ultra-low pricing that frontier providers simply can't match — it also moves the story beyond the now-clichéd "cheap Chinese model" narrative, as early use cases emerge and enterprises figure out where different models fit into their tech stacks and what workflows they should be aimed at.

"Insane" adoption numbers as DeepSeek flips the cost structure

DeepSeek rolled out V4 Flash to public beta July 31, and made V4 Pro generally available on August 13. The 284-billion-parameter Flash is built for volume and speed, the 1.6 trillion-parameter Pro for more complex workflows.

Both models have flexible reasoning capabilities (low, high, max) and ‘thinking modes’ applying chain-of-thought (CoT) reasoning to improve answer accuracy.

Users were immediately impressed by Flash’s capabilities. It has dominated OpenRouter's usage leaderboard since its rollout, currently the most-used model on the platform by weekly token volume.

“The adoption numbers of the initial DeepSeek V4 Flash were insane,” ML researcher Nathan Lambert posted to X, adding that the new version "scored the same as GLM 5.2," making it a "total monster" that will be used extensively.

DeepSeek switching the cost model adds an interesting dimension.

V4 API rates are going up by as much as 1,100% depending on the model, token type and time of use. The new pricing structure:

• Flash will be 22 cents per million input tokens and 66 cents per million output tokens off-peak; and 44 cents per million input tokens and $1.32 per million output tokens at peak. This represents a 57% to 371% increase.

• Pro will be 66 cents per million input tokens and $1.98 per million output tokens off-peak; and $1.32 per million input tokens and $3.96 per million output tokens at peak. This shows a 51% to 355% jump.

• Cache hits, meanwhile (when models reuse prompts rather than starting from scratch), are going up between 52% and 1,100%.

DeepSeek says offering 50% lower off-peak usage is intended to encourage “more flexible workload scheduling." Seventeen of every 24 hours stay at half price, and the new structure actually prices the company's home market the highest.

"This is not a simple price rise," said Sanchit vir Gogia of Greyhound Research. "It is a pricing architecture that makes the timing of inference an economic variable."

Work that can wait — such as batch evaluation, synthetic-data generation, and overnight development runs — moves into the cheap hours; interactive agents and live operations cannot. Gogia said irritation among developers and enterprises is genuine and vocal, and that DeepSeek's past low pricing doesn't obligate it to stay cheap forever.

At first glance, it does look like a “suicidal move from a platform still looking for credibility against more established AI model vendors,” said tech analyst Carmi Levy. The increases will certainly eat into DeepSeek's price advantage and force customers to weigh concerns around the company's Chinese origins more heavily.

Still, DeepSeek remains far cheaper by all pricing measures relative to competing models from OpenAI, Anthropic, Google, Cohere, xAI, and others, he said.

So, while the move will force DeepSeek to emphasize performance and security over cost, it hardly wipes out its already-notable price-performance advantage, and still gives customers ample wiggle room to justify its use for specific workloads, Levy said. The math will just have to be more tightly calculated.

“The advantage will likely erode over time as DeepSeek inevitably continues to align pricing with market realities, but for now it's still easy to make the business case,” Levy said.

Where can DeepSeek Flash fit into enterprise environments?

Adoption inside enterprises remains an open question due to cost, capability, reliability, data governance, security, and other factors.

One use case is batch processing, Levy said. This kind of work is typically routine and repetitive rather than demanding top-tier intelligence, so it makes sense to use a cheaper, more efficient model. “It’s a high-performance inference engine that enterprises can consider using for point solution workloads rather than as a wholesale replacement for the incumbent offerings,” Levy said.

Partial adoption will likely involve isolated, non-sensitive workloads with clearly-defined success metrics, strict oversight, permissions controls, and fallback models in case of failures, he noted. Broader deployment will require DeepSeek and its hosting partners to demonstrate strong reliability, security, privacy, auditability, and deployment options. As it adjusts price structures based on demand, DeepSeek also must retain a large enough price-performance advantage to justify any risk, he said.

Expect unsanctioned, smaller-scale use in backroom labs and contained test environments as IT teams get familiar with the new model and figure out when and how to bring it to senior leadership for budget approval.

“DeepSeek has built a well-earned reputation as a global disruptor,” he said, “and it’s clear that its march to broader enterprise adoption will continue to gather momentum.”

Testing DeepSeek in multi-tool workflows

While many use cases are still in the experimental phases, Meta software engineer Naman Ahuja offers one that could translate directly into enterprise environments. In a project unrelated to his employer, he built a home-automation agent with DeepSeek V4 Flash to explore how a lower-cost model performs as the reasoning/orchestration layer for a real multi-tool workflow.

When he leaves home, an agent coordinates several actions across otherwise separate systems: Such as setting a thermostat to “away” to reduce unnecessary energy use, arming a Ring security system, closing and locking doors.

“What interested me was not simply whether the model could understand a command, but whether it could translate intent into a sequence of actions across multiple tools where reliability matters,” Ahuja said.

The biggest lesson was that once a model can take actions, reliability matters as much as intelligence. The system needs structured tool outputs, verification that actions actually succeeded, retry/failure handling, and clear boundaries around what the model is allowed to do.

In the case of enterprise, “the architecture is similar.” Home devices change to ticketing systems, databases, CRM platforms, or infrastructure APIs. The most useful agents will likely orchestrate repetitive workflows across multiple systems, with scoped permissions, auditability, observability, and human approval for higher-risk actions.

“Many valuable AI agents will not be chatbots; they will be background agents coordinating APIs, infrastructure, and business systems in response to events,” he said.

Enterprises need tangible use cases

But Flash's API is still in public beta, Gogia pointed out, and there is not yet an evidence trail of settled enterprise adoption, real-world deployments, and named customers.

“The benchmark story is looser than its retelling, the portfolio story is newer than it looks, and the economics have moved into the system around the model,” he said. Developer mainstreaming is proven; enterprise standardization is not. “The model is mainstream by traffic and still unproven by contract.”

DeepSeek's own integration guidance is an important consideration, he said: Its documentation for at least one popular agent environment states that built-in V4 entries are not sufficient for reliable operation without compatibility overrides.

“Which is a vendor telling the market, accurately, that benchmark performance is not a proxy for production readiness,” Gogia said. “A model can score beautifully and still misbehave once tools, credentials, and state enter the room.”

The serving layer behaves no differently: the same open weights run by different hosts show visible differences in throughput and uptime. "Choosing Flash therefore answers one procurement question and opens three more: Who serves it, where it runs, and which controls surround it," Gogia said.

Prepare for a multi-model future

DeepSeek offers a nuanced case for a multi-model future. V4 Flash is being deployed as the high-volume worker inside diverse estates, Gogia noted: It handles routine generation, retrieval, and background automation, while more difficult or sensitive tasks go elsewhere.

“The question is whether its performance is sufficient for the real-world workflows enterprises actually run, not whether it tops every benchmark,” he said. Enterprises must determine which combination of model, harness, and provider completes the work safely at the lowest cost.

Adam Dalloul, CEO and founder of EmpirioLabs AI, pointed out that bigger isn’t always better; workflows should be task-dependent. For example, his team at EmpirioLabs AI — which hosts 100-plus models on one API, including DeepSeek V4 Flash — were recently working on translating its site into different languages, and there was no need for a large model like GPT 5.6 Sol or Opus 5 to complete the task.

“This is where subagents come in handy,” he said.

His recommended approach: Spawn cheaper subagents and adapt per task. For instance, use Flash variants for day-to-day work, and Pro variants when you need something more powerful. “It depends on the nature of your application.”

Many companies are pivoting towards their own internal benchmarks to route models effectively, Dalloul noted. For example, his team has a workflow that puts a model through various gates and instructions. This helps them identify the model with the speed and accuracy required for the task.

In another example, one of his enterprise clients exclusively wanted access to DeepSeek V4 Flash. They had tested a variety of models and V4 Flash was the only one that met their criteria for speed, cost, and an “appropriate intelligence threshold.”

Meta’s Ahuja agreed that smaller, more efficient models can handle frequent, well-defined agentic tasks, while more expensive frontier models can be reserved for “ambiguous, difficult, or higher-risk decisions.” The relevant metric increasingly becomes cost per successfully completed workflow rather than simply cost per token.

The trade-off, however, is that cheap inference does not automatically mean cheap or safe automation, he said. Once an AI system can take actions, reliability, verification, permissions, failure handling, and security become much more important.

“A failed text response is inconvenient; a failed action in an operational workflow can have real consequences,” Ahuja said.

At Kilo Code, engineers are reading or writing code themselves only about 1% of the time now, according to co-founder Emilie Schario — the rest is agents. That shift is forcing new questions onto dev teams: which systems are safe to hand over, who cleans up when models goof up, how to support multi-model architectures, and whether skyrocketing token bills mean real progress or just burned IT budget.

As far as tech leads from Replit, Kilo Code, and Symbotic are concerned, it’s a natural — and welcome — evolution as agentic AI becomes embedded into more and more enterprise workflows.

“Unless something's really broken or debugging, 99% of the time engineers are not reading or writing code anymore,” Emilie Schario, co-founder of Kilo Code, said at VB Transform 2026.

AI good at greenfield, not so great at brownfield

For Jared Go, distinguished engineer for AI and cloud at warehouse automation company Symbotic, the current moment is about directing the focus of AI. "These are my criteria," he said. "Let's look at it from the lens of security, elegance, clean, concise code, water tightness." That way, AI does most of the heavy lifting, and human code review isn't as critical.

Human involvement becomes necessary further down the line, Go noted, because agents don't make strong product decisions. “Greenfield [building brand new codebases] is so easy for agents. Brownfield [writing, updating, or maintaining existing code] we all know is where the actual challenge lies.”

Replit takes a bit of a different tack: While the company has "gone very agentic," they've been more conservative with AI coding, explained Amol Jain, head of product engineering. An agent reviews each pull request (PR) and assigns it a risk score; low-risk PRs are self-merged by their author, while others go to human reviewers who read the code and give feedback.

“The idea was human on the loop, not human in the loop,” Jain said. Replit’s internal tool is essentially self-driving for software engineers; devs give a task to agents, which do end to end planning, implementation, and testing.

“It's a fleet of agents that run in their own cloud virtual machines (VMs) with access controls behind token proxies so they're secure,” Jain said.

He shared one example where an engineer couldn’t repro or solve a “very gnarly bug” deep in its systems. It was sent to an AI manager agent, which told it to go to sleep. The manager agent then spun up a bunch of underlying agents that found the issue; it subsequently spun up a bunch more agents that found the fix. Six hours later, AI had a PR ready for the bug that had puzzled human engineers.

Multi-model is the future

AI providers are also evolving beyond the lock-in model, as customers increasingly demand multi-model choice.

Kilo Code, for its part, supports 500-plus models in its gateway. "Your software that you're using to do agentic engineering should be decoupled from the model that you're using to do it," Schario said.

For instance, Schario said companies often use expensive frontier-tier models to architect a project, then switch to a less expensive open-weight model for the rest of the work.

It’s also important to respect model provider limitations, such as when they need to work in closed or isolated environments or providers in their specific regions. “It's factoring in what's important to you, what limitations you've set, what data retention policies you've established, what keys you've brought in, what commits you might have … into that routing decision,” Schario said.

Replit, similarly, tends to have a better sense of the cost versus capability spectrum than its customers, Jain contended. “We are essentially making the decisions on users' behalf of what model to use when, in what capacity, to minimize cost and maximize capability.”

To tokenmaxx or not to tokenmaxx

Of course, an important consideration as AI adoption increases is runaway costs, which has led to some enterprises tracking and capping AI use through tokenmaxxing.

Concerns come from both sides, Schario said: internally and from customers. From the latter, she's hearing, "I accidentally spent my whole AI budget for the year … so what do I do now?" In response, Schario said Kilo Code points customers to the same workflow: use expensive models for planning, then open-weight models for affordability.

Further, sharing skills, strong guidance, and Model Context Protocol (MCP) will empower models. “Realizing where you can really uplevel your team to help them get the most out of the models they're using is going to make a big difference,” Schario said.

Internally, meanwhile, Schario noted one particular engineer that has a "heavy foot" and is constantly at the top of the usage board. "I regularly have to nudge, 'What are you doing there?'" she said. It's easy to look at a $600 bill for daily work and react, "Wow, that's so much," but looking at the amount of work completed can sometimes justify the cost.

“Cost per pull request is the metric that I'm paying attention to right now,” Schario said. “It feels like the closest proximity for how I can measure value.” Ultimately, AI changes how enterprises are thinking about ROI because spend is not the problem. “The spend with no return on that spend is the problem.”

Symbotic, for its part, has set per-month cost tiers for its employees. The company built a tool that gives managers visibility into PRs and usage trends. They can then move users up or down a tier as they see fit, Go explained. “Having a cap and seeing how many people went up in cap this month makes a big difference when you're trying to corral these costs and make things efficient,” Go said.

When Cursor — which Symbotic uses heavily — ended a legacy discount that had grandfathered the company into a flat per-request rate even for frontier models, and moved everyone to full pricing, it forced a company-wide reckoning on efficiency, Go said. "People were saying, 'You should try this model … This works better for this C# code, this whatever,'" he said.

But the cost problem is increasingly moving out of IT; Replit, for one, broadened agents beyond engineering, and eventually found that a user on the support side had "blown through an insane amount of money," Jain said. When they looked under the hood, they figured out it was because they were running an automation on GPT 5.5 Pro Max.

“At least till that point, the ROI was rather clear,” Jain said. “We could see engineering productivity 3X, so no one had questioned it yet.”

Visibility that isn’t “anti-productive,” model routing, and sensible defaults are critical, he emphasized. “Most tasks do not need the frontier.”

Enterprise AI agents can do the work — but the infrastructure to let them talk to each other, prove they should be trusted, and be audited when something goes wrong is still being built.

Here's a look at how five startups are tackling that gap — around orchestration, observability, connectivity, and security — as shown at VB Transform 2026.

BAND is orchestrating all the agents you have running in the background

In the very near future, agents will be deployed everywhere, and they will do work on our behalf, noted Vlad Luzin, CTO and co-founder of BAND.

As he describes it: They will receive tasks, visit registries, recruit other agents to help them, delegate subtasks to AI peers in a “conversational space,” gather and share results, then return a summary to the human user.

BAND is building a coordination infrastructure layer for multi-agent AI systems to make this a reality.

Why don’t Telegram, Slack, or Discord solve the problem? These platforms were built for humans, Luzin noted. Agents have to be onboarded manually in numerous steps, and they can’t see each other; “they are still alone in a kind of digital solitary confinement.”

Similarly, Claude is stateless, and devs often have multiple sessions open at a time that they toggle between for different tasks — something Luzin said creates real friction.

The challenge is connecting remote processes, which Luzin called a distributed systems problem.

“The transportation layer needs to be solved first, how the agents communicate in real time,” he said. Conversations can’t happen through IPs and URLs; they need to be bumped to the abstraction layer so agents can talk across channels, conversational spaces, and platforms.

“Agents see each other. They understand. They can collaborate together. They discuss issues. They fix issues, and they ask for review from another,” Luzin said.

BAND supports autonomous workflows that can run for eight to 20 hours and is compatible with A2A and MCP protocols, according to Luzin. Importantly, humans can join the conversation as agents converse and discover one another, he said.

“We can record and show you all the tasks that your agent generates in real time,” Luzin said.

Conifers is helping defenders move at machine speed

The biggest challenge defenders face today is that they’re still running at human speed, but adversaries are running at machine speed, said Tom Findling, CEO and co-founder of Conifers.

Attackers are already adopting agents, Findling said, and they only have to be successful once to penetrate an enterprise. Malicious campaigns that used to take months and weeks now take hours, even minutes. Security operations, on the other hand, are fragmented, manual, inefficient, and slow.

Findling said Conifers has taken various components of cyber defense — private intelligence, hunting, detection, engineering, investigation, response — and made them agentic. They then broke down the silos between them, he said. Various agentic systems can communicate with one another to ensure that operational defense and active defense are always on and adapting.

Findling said that Conifers’ system is condensing containment time from 7 hours to 12 minutes, and that the company can turn around complex cyber investigations in four minutes or less.

He emphasized the importance of connecting to an enterprise’s existing security tools, whether that be endpoint detection and response (EDR), security information and event management (SIEM), posture management, or others. Conifers helps customers understand their security posture, pain points, which controls are working and which are not, and the areas to invest for the best ROI.

“The threat landscape is changing, detection stays the same, and threat intelligence is not being operationalized,” Findling said. “This is a job for agents.”

Raindrop AI creates an agent audit log

One of the defining problems of the current era is finding critical issues in AI agents, says Ben Hylak, CTO of Raindrop AI.

It’s what he called a “double whammy”: As agents become more capable, complexity increases, as do timelines; they are running for hours or days in some cases. Secondly, issues become catastrophic in sectors like healthcare or defense.

“This problem is getting a lot worse as models and agents improve,” Hylak said, “and I think there's good reason to believe it will continue to get worse.”

Raindrop AI's platform finds critical issues in agents in production and simulates fixes based on past user behavior, Hylak said. That lets teams confirm a fix works as intended before it's live, without introducing unexpected side effects.

The startup’s reinforcement learning (RL) platform optimizes harnesses and trains models directly from Raindrop data, he said. Its pre-deployment simulation engine helps identify what fixes would actually impact in production; its live A/B testing then shows those changes in action.

Messages, tool calls, retries, and errors are captured in one place, and human users are notified (typically via Slack) when there's an issue, he said. Models are trained for every customer, and signals are powering continual learning across models and harnesses. “It is condensed into something that is actually navigable, easy to understand, easy to verify,” Hylak said.

Arcade gives agents the security clearance they need to take action

AI agents are designed to do all kinds of things for you, but they often hit three major snags: authorization, governance, and reliability.

To act on behalf of real users with real permissions, agents need a new type of security architecture, said Sam Partee, co-founder and CTO of Arcade.dev.

Partee said his company’s secure agent runtime provides this authentication and authorization layer so agents can pass critical security reviews. It also provides observability so human users can watch everything an agent is doing. Actions are attributable to the exact moment in time with the least amount of privileged scopes.

Arcade is available in an installable plugin that can be deployed on-prem in a clean room-like environment; companies can continue to use their own sign-in and security tools, Partee said. Whenever anything is run in Arcade, it's gated by the same role-based access controls (RBACs), intrusion detection and prevention systems (IDPS), policies, entitlements, and other already-established checkpoints.

Arcade is tackling the supply chain attack problem, which has “gotten so rampant; it's unbelievable,” Partee noted. Security and observability have continued to be challenging because “largely, the abstraction has been wrong.”

Omilia is tackling the "not straightforward" CX problem

Solving enterprise customer experience (CX) is “really not straightforward,” said Claudio Rodrigues, CPO of Omilia.

Heuristic-based systems are controlled but slow; agentic systems are fast but unpredictable, Rodrigues said. Omilia built its platform to deliver both control and speed together.

The agentic, self-learning offering is built on a philosophy of observing customer service operations as they actually happen, rather than in the abstract. Omilia's agents observe problems first-hand, listen to every customer and agent interaction, ingest data, API specs, screen recordings, and standard operating procedures (SOP), then map those to use cases for customer support, he said.

Contact centers should be a revenue driver, Rodrigues said, and Omilia’s differentiator is its speech-to-text systems and governance and observability layers.

AI creates insights, suggests improvements, automatically generates conversational agents, pulls information from documents and APIs, and designs dialogue flows. Human experts can then test real and simulated interactions and deploy into production under their supervision. Omilia combines all of this into one enterprise-wide engine that continuously learns over time, Rodrigues said.

Rodrigues said the company handles more than 3 billion calls a year, 1 million-plus voice calls a day in some deployments, and has seen 30 to 45% improvement in time to resolution (TTR). Omilia’s agents generate 21x more upsell revenue versus human agents, he said.

In a mature deployment, automation “easily” reaches 80 to 90%, he said. However, “human in the loop is still very fundamental for us.”

Target SVP Siobhán Mc Feeney says the AI models her company runs aren't what gives Target its edge — everything built around them is.

"There's a lot in it. That to us is the moat," Mc Feeney said at VB Transform 2026. "The models are great, and they're important. They're just not sufficient to be the competitive advantage."

That discipline shows up early in how Target decides whether to build an agent at all. Mc Feeney was blunt, even "controversial" by her own admission, about the current AI moment: every enterprise wants AI agents, but not everything needs one, she said.

Agents earn their autonomy over time rather than getting it by default, she said — a principle that runs through everything Target has built around them.

Mc Feeney said the goal is to make sure agents are aimed at the problems that drive the most value for Target's guests. “We want to make sure we're investing in the right places," she said.

Being deliberate about agents

Agents are becoming part of Target's underlying architecture, increasingly connecting signals, systems, and decisions across supply chain, replenishment, and demand forecasting.

Mc Feeney framed it as retail's oldest promise — the right product, in the right place, at the right time — delivered at scale.

But her team has been deliberate about building AI agents, beginning with the simplest, most obvious question: What is the problem they’re trying to solve? This leads to several follow-on questions: 

• Does that problem need an agent? 

• If it does, what type of agent? An orchestrator? A super agent? A domain-specific agent? 

• Or is what you're calling an "agent" actually just a tool?

“You define that upfront, and this may sound a little process-heavy, then you have to register and certify your agent,” Mc Feeney said. Because a solution may already exist, and you don’t want to duplicate work. 

Agent design kicks off another series of important questions: What triggers an agent to act? Automation? An engineer? A timer? What needs to be put in place to track that? 

"We're trying to make sure we have lineage from the very beginning — the birthing of this agent, all the way through — because at 2 a.m. one morning, when something goes sideways, we want to make sure we understand everything that happened," Mc Feeney said.

Autonomy level is another consideration; new agents typically start with base autonomy and earn more over time. What the agent has access to is a separate question: what data, what systems, what tables, what databases?

Finally, there’s monitoring and observability; agents won’t solve problems, or improve over time, if they’re not continuously evaluated. 

“We measure everything: What it was intended to do, its calibration, its trajectory, not just runtime and latency,” Mc Feeney said. This creates full transparency, and allows agents to be tweaked over time. 

“You're talking about architecture and taxonomy and a data governance layer that absolutely had to be established,” she said.  

There's a lot in these "layers of autonomy" — that foundation is what gives Target the ability to scale and properly invest in the right models for the right problem.

Models have different “gradients” that are better for different jobs; for instance, frontier models excel at complex tasks that require crunching billions of pieces of data (like in heavy merchandising supply chains). But in some scenarios they can be cost-prohibitive. 

“So it’s making sure there's always a cost benefit,” Mc Feeney said. 

Agents must earn their autonomy

A digital-twin simulation predicted men's shorts inventory across three Target stores in Long Beach this summer — and one store came back needing six to seven times more stock than the others, she said. Inventory analysts' first reaction: That can't be right. But the system had found something they hadn't factored in. That store sat less than two miles from the beach; the other two were 10 to 12 miles inland. Analysts let the recommendation stand, and the stock sold through.

"This is science. This is mathematically more significant and more confidence-filling than humans doing it," Mc Feeney said. Results like that are what let Target's agentic systems earn more autonomy over time, she said.

Target looks at AI agent autonomy as "earned" and structures it as a four-level ladder, Mc Feeney said: agents start by making observations without acting, then move to suggesting actions while waiting for approval, then to acting within defined guardrails. At the highest level Target currently operates, agents run end-to-end — but still with a human in the loop.

“The autonomy levels for the agents are super important,” Mc Feeney said. “They earn them, and they can lose them if they don't perform as expected.” Models that drift will be taken out of service. 

As she put it, humans earn autonomy when we prove we can do something over time. Nobody is given a bunch of extra responsibilities just because; they have to have shown they’re able to handle them. 

In a similar way, agents can be scientifically measured and quantified: how accurate they were, how much they drifted, and how close they came to their intended goal. This helps establish guardrails, allowing builders to work faster, and “go fast forever,” because they're not constantly wondering where the guardrails are. 

“If you follow these guardrails, you [follow] security guidelines, you register the agent, and something still goes wrong, we have full lineage all the way through from the start,” Mc Feeney said. “Our ability to recover is much better.”

When it comes down to it, agent success is a confluence of factors, not just one, she said: “It's about your architecture. It's about your taxonomy. It's about the autonomy levels your agents have, and it's about security and observability.” 

A new skill set for new workflows

Even when agent autonomy is high, though, builders must still be held accountable when something goes wrong. Mc Feeney noted that teams are now working at speeds no one could have anticipated, which means evaluation harnesses have to be established and agents registered and tracked.

A lot of it is cultural; the workforce is being reshaped and builders and engineers need new skills to manage human workers and AI systems side by side. These contexts are quite different, but the career evolution is “super exciting.”

“You're a builder. You're observing agents building, and you're also coaching humans observing agents building,” Mc Feeney said. “The level of nuance is pretty special.”

Instacart is posing the provocative question: What if most of the work your engineers do today should, in fact, be done by machines?

At VB Transform 2026, CTO Anirban Kundu argued that dev teams continue to waste their time on draining, repetitive, high-volume work; this should be absorbed by AI agents so that humans can focus on problems that require judgment, intent, and exception handling. 

In fact, in 97% of cases, Instacart’s builders don’t even read code anymore. 

“In the past, the tactical level was the creation of the code,” Kundu said. “In the most tactical level going forward, it's going to be, ‘How do you navigate around the AI system to give you what you want?’”

AI generating code, performing "pretty serious evals" 

That doesn’t mean humans never look at code; agents handle the bulk of code generation and boilerplate, particularly with newer projects where code is generated or regenerated on a weekly basis. 

“The benefit of that is we don't care about tech debt anymore,” Kundu said. “Things that are not active just get dropped out and then it gets rebuilt, kind of like how we used to build assembly code or object code.”

So why not 100%? The remaining 3% is in legacy, compliance, and latency-sensitive systems and workflows, or driven by a “boatload of code” that is dead, not active, or half-active. These cases still need careful human attention. 

Instacart is slowly “smoothing those parts out,” however, breaking systems down in an aptly-named project Atoms, then building them back up in a cleaner, more modular form. Kundu’s team started with the “monoliths” and is shifting to remote procedure call (RPC)-driven architectures. 

But evaluation remains one of the overarching challenges. Code reviews aren’t as relevant when AI is generating code — as Kundu noted, “the lines of code are going to be correct, the syntax is going to meet your expectations” — so the goal is to move to an “intent model.” That is, training devs so they can ask different models the right questions from an intent perspective. 

Evals are then performed independently: Roughly 7,000 automatic evaluations run each month, and the system answers 8,000-plus real-time developer queries with about 99.9% accuracy.

Identifying "hiccups" that human intuition might have missed

Dovetailing with this, Instacart has built an agentic site reliability engineering (SRE) system trained on years of the company’s own incidents and root-cause analyses rather than generic failure data. Instead of teaching a model how production outages work in the abstract, the team fed it the specific ways Instacart’s systems have broken over time, along with the ways humans diagnosed and fixed them.

As a result, the company has seen accuracy in detecting and mitigating production issues jump from roughly 60 to more than 90%. 

Kundu pointed to one example with Instacart’s internal tool Blueberry. The AI SRE colleague watches 200-some-odd Slack channels, monitors signals, and looks for patterns across human conversations and alerts. 

In one incident, a database shard backed by an EBS volume that had a “hiccup” for a period of time. The human team did not immediately suspect AWS disk issues and were “obviously scrambling” to figure out why this particular shard misbehaved. 

But about 20 minutes in, Blueberry posted on Slack, pointing to a specific blip and tying it to a feature-flag-like system called "roulette" that had been inadequate. "It's supposed to be rolling out in this cadence, [but] it had been too much,” Kundu said.

Blueberry figured it out, and the team resolved the incident. “Would have a human been as quick? I think the problem is human intuition would hold us back a little bit,” Kundu said. 

Humans tend to default to patterns we’ve seen before, then resort to debugging; Kundu called this the “first brain-second brain kind of thing.” But Instacart’s agentic SRE is actually “more comprehensive in its ability to look at everything and then be able to decide what does or doesn't matter.”

Redefining the engineer’s job

Looking ahead, the most tactical work for engineers will be navigating AI systems: Designing and supervising evaluation processes; coordinating multiple simultaneous experiments and features; managing constraints like limited top-of-funnel traffic for testing; figuring out when to escalate; identifying edge cases and where things might break.

Domain expertise is also being rethought in the age of AI. Instead of bottlenecking changes through a single “owner” team that touches the code, Instacart is embedding domain knowledge into definitions and specs that any team can use. 

“We’ve lived in this world where this group or this engineering team is the one that can touch the code and make the modification,” said Kundu. “We're trying to move into a world where the code becomes completely democratized across groups.” 

OpenClaw has become one of the most widely adopted agentic frameworks, but it has yet to prove itself at enterprise scale. Agents need real credentials — API keys, OAuth tokens, service accounts — to work effectively, and Brex found that traditional guardrails couldn't contain what those agents were doing with them.

Brex set out to overcome these limitations by building an internal platform it calls CrabTrap. The open-source HTTP/HTTPS proxy intercepts all network traffic, examines policy rules, and uses a LLM-as-a-judge to decide whether agent requests should be approved or denied. 

“What we noticed was that the network layer was an untapped enforcement point,” Brex co-founder and CEO Pedro Franceschi told VentureBeat. “Every request an agent makes is an opportunity to intercept, reason about, and make a policy decision.”

The takeaway Franceschi wants IT leaders to draw: agent governance should shift from SDK-level permissions and model guardrails toward a centralized network control plane that enforces and learns from real in-the-wild agent behavior.

How Brex targeted the transport layer

The “obvious fix” (at least initially) to the agent security gap was guardrails, and much of the early work has centered on scoped tools, per-action permissions, and human-in-the-loop approvals. But as agents evolve, each new capability means there’s another API to tune or surface to audit, Franceschi noted. 

“Any agentic system with multiple tools and access to the open internet creates an immediate tension for builders: The more capable you make an agent, the more dangerous it becomes, and the safer you make it, the less useful it is,” he said. 

Existing solutions to this tradeoff were “weak”: Fine-grained API tokens help at the margins but can still be misused and constrain functionality. Semantic guardrails (such as context, skills, or prompt steering) are easily bypassed by prompt injection, especially for agents connected to the internet.

Agents can be “defanged” when given read-only access or limited toolsets, but then they can't do meaningful work, Franceschi said. On the other hand, granting broad write access and a large tool surface can result in hallucinations and real production consequences.

Model context protocol (MCP) gateways enforce policy at the protocol layer — but only for traffic using MCP. Meanwhile, guardrails from LLM providers are tied to a single model and can be “opaque” to customize with enterprise-specific policies. And powerful tools like Nvidia OpenShell offer more of a “per-sandbox egress control.”

“When we started, we hadn’t found a solution to deploying harnesses like OpenClaw safely,” Franceschi said. “Instead of waiting for the industry to catch up, we decided to own the problem and invent the necessary tools.”

Notably, they needed a platform that sat between every agent and every network request, and could make “nuanced decisions about what to allow,” he said. 

This made the transport layer a core architectural component and natural starting point, he said. 

By operating at this layer, CrabTrap is framework-agnostic, language-agnostic, and API-agnostic. It doesn't require SDK wrappers or per-tool integration. Users set HTTP_PROXY and HTTPS_PROXY in the agent's environment, and every outbound request routes through the proxy before it reaches a destination.

However, Franceschi emphasized, Brex didn't start at the transport layer because it thought it was the only answer; rather, they believe in “security by layers.”

“The transport layer was simply an underinvested one, and we saw an opportunity to add meaningful enforcement there alongside everything else,” he said. 

The LLM-as-a-judge training loop

CrabTrap combines deterministic static rules with an LLM-as-a-judge for requests that fall outside known patterns, Franceschi explained. The judge only “fires on the long tail of unfamiliar endpoints or unusual request shapes,” which for a mature agent is typically fewer than 3% of requests.

The more pressing problem was how to know that a policy is the right one? With static rules, it's “relatively straightforward” to reason about accuracy. But with an LLM judge, the system is nondeterministic, and users need confidence that the policy approves the right requests and blocks the rest.

“Our key insight was to bootstrap policy from observed behavior rather than write it from scratch,” Franceschi said. Beginning with real behavior and editing down based on real-world learnings turned out to be “dramatically more effective than starting from a blank page.”

Brex’s team built a policy builder (itself an agentic loop) that runs underlying agents in shadow mode, analyzes historic network traffic, samples representative calls, and drafts a natural-language policy that matches what the agent actually does. 

From there, they built an eval system that tests policy changes before they go live. CrabTrap compares historical audit entries against a draft policy and reports the exact changes to be made. Users can slice results by method, URL, original decision, and agreement status. 

All of this runs with concurrent judge calls, so replaying thousands of requests “takes minutes, not hours,” Franceschi said. Brex also developed a live feedback loop: Full audit trails are stored in PostgreSQL and queryable through the admin API and dashboard. In cases where a resource is continuously denied, the system can notify a human or an agent to propose a policy update for review. 

“That closes the loop between observed denials and policy refinement,” Franceschi said. 

Core challenges and roadblocks 

Of course, the build wasn’t without its challenges. A big one was latency: “Putting an LLM between an agent and every outbound API request sounds like it would grind things to a halt,” he said. 

However, it didn’t turn out to be as big a problem as expected. This was for two reasons: The LLM judge only activates on a small fraction of requests (the aforementioned 3%). Agents quickly settle into predictable traffic patterns; once observed, high-volume patterns become static rules. Second, by using small, fast models like Claude Haiku meant that, even when the judge did fire, added latency was “negligible.” This can be further reduced with local models and prompt caching, Franceschi said.

The harder and less obvious challenge was prompt injection, he said. The judge receives the full HTTP request and all content is user-controlled, so potentially, a crafted URL, header, or request body could manipulate the judge's decision. 

Brex addressed this by structuring the request as a JSON object before sending it to the model, so all user-controlled content is “escaped rather than interpolated as raw text,” Franceschi said. 

Results, and where CrabTrap might evolve

Brex tracks a few factors to measure CrabTrap’s internal impact: Engagement with agents, network traffic patterns, and net promoter scores (NPS). The most meaningful result of CrabTrap has been “organizational confidence,” Franceschi said. 

Previously, the team had “real hesitation” when it came to deploying autonomous agents broadly across business operations, because the existing guardrail options didn't provide enough assurance. 

“CrabTrap changed that calculus,” Franceschi said. They now have an enforcement layer they trust, increasing confidence around expanding agent deployment into more parts of the business and delegating more agent configuration and management to users. 

Franceschi described the policies derived from traffic as “surprisingly strong.” The team expected the policy builder to produce a “rough starting point” requiring heavy manual editing. In practice, though, pointing the platform at a few days of real traffic produced policies that matched human judgment on the “vast majority of held-out requests.”

Additionally, CrabTrap revealed how much noise agents generate. “The audit trail made this visible for the first time,” Franceschi said. They used denial logs and traffic analysis not only to tune policies, but to tighten agents themselves, remove tools, and cut out entire categories of requests that were wasting both time and tokens.

“The proxy became a discovery tool, not just an enforcement one,” he said. 

Areas for growth (and input from the open-source community)

Brex anticipates CrabTrap to continue to evolve, particularly as they have released it as open-source. “We hope the community helps shape it,” Franceschi said. 

Areas of improvement include deeper authentication functionality such as single-sign on (SSO), fine-grained role-based access control (RBAC); escalation workflows that allow agents to request additional permissions; and policy recommendations based on denial patterns.

Programmatic configuration, or developing API endpoints for “creating, forking, and applying” policies to agents, could allow the whole policy lifecycle to be automated rather than managed manually, Franceschi said. 

As for escalation, if an agent is continuously denied a given resource or endpoint, it should be able to route requests to humans or other AI agents for review and back that up with a rationale for why it needs access. 

“That turns CrabTrap from a hard enforcement boundary into something more like a managed permission system,” Franceschi said. 

Additionally, the policy was built to bootstrap from network traffic, but there is opportunity to incorporate additional signals around agent traces and resource-calling, as well as broader context on what agents are ultimately trying to accomplish. This can help produce more accurate and nuanced policies. 

Finally, there's an “open philosophical question” about the right posture for CrabTrap: Should it be a fully transparent layer that the agent itself is unaware of, or should it operate more like a “well-intentioned manager”? (that is, the agent knows about the layer and can interact with it).

The open-source community can help shape these developments, and CrabTrap will only get better with more users, Franceschi said. Brex’s agents speak to a specific set of APIs; teams using CrabTrap with different agents, services, and policy requirements will surface “edge cases and patterns we can't hit alone.”

“We have ambitious plans for where it could go, and we’d rather build in the open,” Franceschi said. 

What other builders can learn from CrabTrap

The response has been stronger than expected. CrabTrap has more than 700 stars on GitHub. Franceschi said Brex has also heard from OpenAI, Y Combinator CEO Garry Tan, and programmer Pete Steinberger, all expressing interest in deploying similar internal infrastructure.

The broader lesson: “Don't let infrastructure gaps become excuses to wait," Franceschi advised. There are “real blockers” for every enterprise looking to seriously deploy AI agents, including security concerns, lack of tooling, or unclear guardrails. 

“It's tempting to sit on your hands until the industry catches up,” he said. “The lesson from CrabTrap is that you can own those problems directly.”

When Anthropic's Fable 5 was pulled from international use for nearly three weeks, some over-reliant businesses were left scrambling.

But Liberty Mutual easily pivoted to other platforms. That’s because 18 months earlier, they built their "AI backbone" exactly for this kind of scenario.

In this rapidly moving AI landscape, the 114-year-old property and casualty insurance company recognized independence as an operating advantage.

“Things are changing so fast, you need a backbone that's flexible,” Brian Craig, Liberty Mutual’s senior director of architecture, said at a recent VB Impact event. “You can't lock in right now on one vendor or even one framework.”

Enterprises need flexibility to hook into different models and vendors, depending not so much on the "flavor of the day," but “what can you feel confident about for the next six months,” he said.

Runtime versus control plane

The company’s "backbone" (or control plane) is its own, while everything underneath remains swappable.

The architecture consists of roughly 50 components across security, identity, orchestration, tool restriction, and the policies that govern how agents behave. Each is designed to be independently and immediately replaceable to support interoperability.

The agent runtime below this backbone is AWS's Amazon Bedrock AgentCore; this is not the strategic center, but explicitly “just for running the agents,” Craig said. He chose this offering because it (at least currently) supports multiple frameworks and Liberty Mutual’s model-agnostic philosophy.

“We still have flexibility based on what we write,” Craig said. “But if something comes along and is better, we will move to it quite quickly.”

The software factory

This architecture delivers, as proven by Liberty’s “software factory,” an agentic pipeline that automates much of the software delivery process.

They started with a business process with clear pain: Onboarding electronic content management documents for insurance products. This repetitive, manual task typically required engineers to code every change.

Instead, the team built a factory of coordinated agents working “in tandem and in sequence”:

• An Epic agent consumes high-level requirements.

• A Story agent breaks work dictated by the Epic agent into narrow slices within specific application areas. This agent is “constraining the context, because the smaller the context, the better the output.”

• A planning agent defines the technical execution plan.

• A coding/testing agent handles coding, testing, and basic review.

• A triage (critic) agent sits across all other agents, reviewing quality and feeding back improvements.

• Finally, a librarian agent helps others find the “context of the knowledge for their job.”

Craig and his team learned quickly that a single “do everything” agent was a mistake. “You were asking it to do too many things, which meant you had to give it too much information,” Craig said. Splitting into six agents let them dramatically shrink context windows and tighten scope.

Once the factory hit production, the impact was immediate. In the initial deployment, they did “about three months of work” in roughly a week. They realized that “the current software engineering process has a massive amount of handoffs, which means there's a massive amount of wait time,” Craig said.

Human-paced automation

The factory is not a fully autonomous pipeline; it runs at the speed of human overseers.

Liberty’s first model was a “day shift/night shift” rhythm: Engineers set goals and rules and reviewed the previous night’s outputs during the day, then let the factory run overnight. But in practice, there was never enough work to keep the agents busy all night, and the cadence felt unnatural.

They shifted to a more iterative loop. Users decide when to trigger the factory, how far it runs before pausing, and at which points they want to review outputs. “Then the factory kicked in, and it may only run for less than an hour, and then you would look at it again,” Craig said. “It was more controlled at the speed that our users felt comfortable with.”

The orchestration layer lets teams choose whether to review after the Epic stage, after planning, or once coding and testing complete. “That is up to the users of the factory, but it has completely removed a lot of the wait time that we currently have within our processes,” Craig said.

By contrast, early on, “every time the thing ran, they wanted to look at the output,” and that became the feedback loop that trained both humans and agents.

Some of this is, as Craig put it, “just automating agile at speed” and giving iterative feedback; some from humans, some from the triage agent. “But we're seeing that start to bake in, and it starts to become the rules.”

When the same feedback is coming from two different directions (agent and human), it makes sense to go into Liberty’s context repository. The agents will then rely on that context the next time they need to make that decision, "and the next time again, and you start to speed up."

“It's like a flywheel once you start building these and you start to get them flowing,” Craig said. “You realize it listens to what you say.”

Contracts that match the pace of change

Liberty paired that architectural posture with a contract posture, deliberately shifting from five-year enterprise deals to one-year agreements. The logic is simple as Craig sees it: The AI market moves too fast to lock into one vendor or framework for half a decade or more. Shorter terms let them evaluate — and if necessary, swap — models and platforms at the speed the market actually changes. Cost is part of the story. When a premium frontier model like Fable arrives, the sticker shock is real. “You see the price and go, ‘Goodness, it better be really good,’ Craig said. (It was; they got to use it enough to "fall in love with it.”) The backbone’s interoperability lets his team compare models at different price points and route workloads based on price–performance rather than vendor inertia. The same attitude governs how they plan to use agents from major SaaS platforms. Liberty is a customer of Salesforce and Splunk, who will both “bring their agents to the table.” His team has no interest in replicating engineering work, but they do insist on observability. “We just want to harness it as part of our system,” rather than control it, Craig said. “But we want the observability to understand what their agent is doing with our data, with our users.”

Closing the “control gap”

Importantly, Liberty built observability into its backbone. As Craig explained, it’s not just logging what an agent does, but what it accesses, which identity it uses, and which tools it’s allowed to invoke. Identity and access run in part on Microsoft Entra ID, and agents are given only the tools and permissions explicitly assigned to them. Whenever an agent realizes, ‘I don't have the information,’ it asks for it, and it only gets what it needs, rather than giving authority to “use every tool in the box.” “Because too much information given to an agent is worse than no information,” Craig said. “You just overload it, and it gets confused.” For detection, Liberty runs evaluations with MLflow against “golden datasets.” Whenever prompts or models change, they regression-test and immediately see whether results improved or degraded. One of his team’s new mantras is “you need to walk in the footsteps of a new start." If a new start can't find a guiding document, how will an AI agent? One of the key things enterprises need to do, no matter the business process, is “write stuff down, which is not earth-shattering,” Craig acknowledged. Agents obey written standards more reliably than people, and a context repository is a central artifact of Liberty Mutual’s system. Agents have made human judgment more, not less, central, he emphasized. Nothing ships without a human sign-off, consistent with Liberty’s risk posture as an insurer. “The confidence isn't there yet for us to just let it run wild, and I don't think it ever will be for the likes of Liberty,” said Craig. “We have to be rock solid before we let [anything] into production.” Moving fast for product fit and survival might make sense for other companies, but ultimately, “some people will get black eyes, but that's the joy of innovation these days,” he said.

Most verticals aren’t clean, well-oiled SaaS databases; the reality is ugly documents, proprietary schemas, implicit workflows, and long‑running tasks that most general-purpose models struggle with.

This prompted construction project management company Trunk Tools to build a specialized, three-layer architecture — perception, semantics, agents — based on highly-detailed data to support high-accuracy, highly-relevant industry automation.

Their purpose-built stack has shrunk review cycles from months to days, prevented costly field errors, and given autonomous agents the ability to reason over millions of pages of documentation, the company says.

“We really set out to take the data from dispersed systems, pre-process it, structure it, go through our ontology into a knowledge graph, and then train AI models,” said Sarah Buchner, Trunk Tools' founder and CEO and a former carpenter.

For builders in other verticals, the company's approach could serve as a blueprint for transforming data chaos into agent‑ready, industry-specific workflows.

Where general-purpose LLMs break down on industry data

Foundation LLMs, while powerful, are optimized for breadth, not always depth.

“General-purpose LLMs are trained to be okay at everything, so they're weak at anything niche,” said Kriti Faujdar, a senior product manager working in AI infrastructure, agentic AI, security, and LLM platforms. For instance: Rare terms, domain-specific reasoning, the unspoken context that any practitioner “just knows.”

Web, app, and software developer Sébastien De Bollivier agreed that the biggest bottleneck is reliability on data that is “jargon-dense, abbreviation-heavy, and format-specific.”

“A GPT-4-class model can understand a French legal contract, but will fumble the specific article references practitioners need to cite,” he said.

Besides, the most valuable enterprise data never made it into pretraining anyway, Faujdar pointed out. It's sitting in internal systems and proprietary formats. “RAG helps a little,” she said. “But it's just giving better facts to a model that still can't reason properly in the domain.”

Pre-training on domain data is critical; enterprises should then fine-tune on good task examples and build their own evals. “A few thousand examples from real practitioners beats millions of scraped, noisy ones," Faujdar said.

Mixture-of-experts (MoE) can provide specialization without inference costs blowing up. Pairing RAG with fine-tuning also works well; RAG handles the factual long trail while fine-tuning fixes vocabulary and reasoning.

De Bollivier pointed to the advantage of hybrid stacks: A general-purpose model for reasoning and orchestration, a smaller fine-tuned model (or dense retrieval over a curated corpus) for domain-specific extraction. He advised: “Don't fine-tune to make the model 'smarter' about a domain, fine-tune to make it more reliable on the specific output format your workflow requires.”

The trades and construction are certainly industries seeing traction with these techniques, as are legal and healthcare, De Bollivier said. These verticals have “high stakes for errors plus standardized document formats, equaling clear domain-training ROI.”

One honest caveat worth mentioning, Faujdar said: Specialized models can often fall apart outside their domain, so they’re often not useful outside their expertise (unless they’re re-trained).

Perception, semantics, agents: inside Trunk Tools' three-layer stack

In highly-specialized domains like construction, “data dumps” into large language models (LLMs) don’t cut it, said Trunk Tools' CTO Amrish Kapoor. This is because most transformers are probabilistic models: When given an image, they report back that it is “probably” a tree, or “probably” a child playing next to a tree.

This makes them insufficient for high‑precision symbolic interpretation. For instance, in construction documents, a 2-millimeter-wide symbol has a vastly different meaning depending on where it’s placed.

Further, constrained by context limits, probabilistic models struggle with long‑term project memory. “I don't mean a context window of a few tokens,” Kapoor said. “I'm talking about long term memory that stretches across months and years, because this is how long some of these projects are.”

Instead, the company's three-layer system breaks workflows into:

• Perception (reading and extracting data from messy docs like PDFs, drawings, or scans)

• A semantic/graph layer (making sense of that data and understanding their relationships).

• LLMs and agents on top.

Construction drawings are typically symbolic, Buchner said. A door isn't always labeled ‘door.’ Sometimes it's simply an arc on a wall that a trained eye learns to read based on years of practice.

“The perception layer is what teaches AI to read that language,” she said. The semantic layer then gives that information meaning; for instance, connecting the door to the drawing that details it, the spec that governs it, and the trade that installs it. This helps answer project engineers’ critical questions: Not "is there a door here?" but "does this door create a problem down the line?"

Particularly in construction, that shift matters because the cost of a problem compounds with time. “A conflict caught in design is relatively low cost to address,” Buchner said, “whereas the same problem caught in the field might cost tens of thousands of dollars.”

At a high level, the system identifies the document type and begins extracting information based on content (drawing, schedules, paragraph text). This data is then “transformed and augmented” in the platform, which triggers agentic workflows like knowledge graph relationships and end-user workflows.

For instance, an agent might review an architecture bulletin and produce a visual overlay comparing an older version and a newer version (flagging additions and removals), then generate written narratives that describe what those changes are in simple terms. This helps users understand what’s changed and coordinate with trade partners on updated pricing and change orders.

The scale of construction’s data problem

Construction workflows are “ripe with implicit assumptions and connections between data in its myriad of sources,” Buchner said. And the amount of unstructured data is “humanly impossible” to process or make sense of.

Buchner estimated the average high-rise building generates about 3.6 million pages of corresponding documentation. “If you print it into a stack of papers it would be as high as the building itself.”

All three layers of Trunk Tools' stack — perception, semantic, LLM — are trained on “very specific datasets” from customers with “explicit permissions” and auto‑labeling/IP, Kapoor explained. Customers who don’t want Trunk training on their data can opt out.

Data is deidentified and aggregated, and Trunk Tools also collects “tons more” labeled data through other pipelines like 3D building information modeling (BIM).

The company says it only ships agents that achieve around 95% accuracy. The team maintains continuous evaluation pipelines based on ground truth data from customers and experts. They also employ an LLMs-as-a-judge model.

“This notion of an LLM as a judge is to score how well you're doing, both subjectively as well as objectively,” Kapoor said. Objectivity can be an easy ‘right’ or ‘not right,’ but subjectivity requires more nuance.

For instance, when creating an email or narrative or explanation, an LLM as a judge framework can create a composite score, or a numerical value that aggregates different metrics and tests a model's performance or risk.

There can be challenges, though, particularly with latency, Buchner noted; any time the reasoning capacity of underlying models increases, the risk of latency goes up, too. Trunk Tools maintains a set of evaluation criteria to objectively measure latency whenever changes are made to underlying infrastructure, agents, and API calls.

Then, “before we release to customers, we ensure marginal changes to the end-user experience are well worth the performance enhancements,” Buchner said.

From 60 days to 10: the measurable payoff

Trunk Tools' platform powers seven AI agents purpose-built for construction, such as analyzing request for information (RFI) responses, overviewing bids, or reviewing drawings and submittals.

The submittal agent, for instance, flags missing, conflicting, or noncompliant information in product specs and RFIs. While it’s an essential step in the construction process, “it's a super annoying workflow,” Buchner said, because human reviewers have to compare documents “with a bunch of other parts of documents.”

But the agent is able to do this in seconds, and Trunk Tools says it has reduced submittal cycles from 50 to 60 days to 10, “which has massive schedule and financial implications.”

The company is now at a place where these agents are communicating directly with each other, which is “quite exciting,” Buchner said. So, for example, one agent will review an architectural drawing for accuracy, then autonomously hand it over to agents handling RFIs and asking follow-up questions.

“If the drawings have problems, the RFI agent is taking over and is actively reaching out for clarification,” Buchner explained.

Trunk Tools says its customers report savings of 20 to 40 minutes per field question. Buchner said that users in the field know better than anyone how much of a “time suck” it is to go back and forth from office trailers, dig through project documents in scattered systems or printed PDFs, reconcile discrepancies, and return to coordinate with trade partners.

The company says its customers report these additional outcomes:

• Average 8 minute time savings for single-document retrieval (status checks, location lookups, quantity queries).

• Average 20 minute time savings for standard referencing (cross-referencing 2 to 3 spec sections to form an answer.

• Average 40 minute time savings for multi-document research (listing and filtering queries, mapping relationships, analyzing RFIs and submittals across 4 to 6 documents).

• Average 75 minute time savings for complex tasks (creating RFIs and other communication materials, deep cross-referencing across documents, change tracking).

In one instance, the company's drawing review agent flagged that a structural beam had been moved up 8.5 inches. However, this was not documented by the architect. If the change hadn’t been caught, the project manager would likely have had to strip out and reinstall the right size beam, Buchner said. This rework would have added $10,000 or more to the budget, and “certainly there would have been implications on the schedule.”

Buchner also pointed to other examples: an agent flagged $60,000 in exaggerated pricing with no justification from landscaping subcontractors; identified a fireplace that needed to be sealed prior to drywall installation, saving around $100,000 in labor, materials, and delays; and called out that an electric door required a panel that wasn’t included in electrical drawings.

Learnings for other industries

Trunk Tools' approach to building agents is applicable to any vertical working with high volumes of unstructured, industry-specific data. Builders working in specific verticals must understand the industry’s specific data challenges their end users face and build technical infrastructure that can transform unstructured data into something an “LLM can traverse and understand,” Buchner said. “Only then can you build the connections between data points that ultimately feed agentic workflows.” A lot of money is being invested in foundational models, so enterprises should build modular systems that can leverage the strengths of various models as they continue to improve, Buchner advised. Then, “build your technical advantage where the generic models are not investing and not performing well,” she said.