Growing datasets and public benchmarks are making it harder to tell whether a model is being tested on something it hasn’t seen before.
On Thursday, Google DeepMind showed off what the company calls the first double-blind evaluation of a proprietary frontier-class AI model. The setup keeps Gemini’s model weights hidden from the evaluators while the test questions are hidden from Google.
The pilot tested Gemini 2.5 Flash Lite against private benchmarks from MLCommons and the Singapore AI Safety Institute, but rather than using the results to tout a new Gemini score, Google is focusing on how the tests were run, with neither side having access to the other’s data.
Benchmark leakage inflates scores
Google’s technical report cites earlier research that found signs of benchmark leakage in about half of the 31 models tested, and another study published this year that found contamination can inflate scores, particularly for larger models. This dynamic has already raised questions about where certain post-training coding gains actually come from.
Keeping benchmarks private would seem like the obvious answer, except closed models make that difficult. With a closed model, evaluators usually have to run their questions through the provider’s API, which means the company can see a test that was supposed to stay private, whereas running it independently would require the provider to hand over its model weights. Google’s setup is meant to give both sides another option.
Keeping benchmarks private would seem like the obvious answer, except closed models make that difficult.
How the enclave works
The pilot is using Google Cloud Confidential Space together with an NVIDIA H100 Confidential GPU and Intel TDX host memory encryption.
Google DeepMind provides Gemini with its weights and inference code, while the evaluator offers its benchmark prompts and evaluation code; these are then transmitted over encrypted connections into the enclave, where the evaluation takes place without either party gaining access to the other’s protected assets.
The model weights are stored in hardware-encrypted GPU memory, and the evaluation prompts are kept in encrypted host memory as well. After the test is complete, the evaluator is given the permitted results, and the temporary environment can then be destroyed.
Before either side sends over its private data, remote attestation verifies that the enclave is running the software they agreed on.
Code controls beyond encryption
The enclave protects what’s stored in memory, but the code running inside it can still create problems. If an application has unrestricted network access, for example, it could send sensitive information somewhere else.
OpenMined’s PySyft handles that part of the process by allowing Google and the evaluator to approve the code in advance and to block sensitive parts of the evaluation from making external connections.
That process takes some work, although compute doesn’t appear to be the problem. The paper puts the overhead at less than 5% and instead points to legal agreements and code reviews between organizations as some of the bigger hurdles to running these evaluations.
The researchers eventually want to make attestation much less hands-on, comparing the goal to the HTTPS lock icon in a browser, where the security checks happen without the user having to deal with the underlying hashes and keys.
Google still has a hand in verification because, although the Confidential Space guest OS is open source and its build process has been externally validated, individual builds rely on private signing keys and can’t be independently reproduced. At the same time, Google’s own services are used to sign and verify the attestation report.
The paper puts the overhead at less than 5% and points instead to the legal agreements and code reviews between organizations as some of the bigger hurdles to running these evaluations.
Scaling past one GPU
So while the system reduces how much the two sides need to trust each other, it doesn’t remove trust entirely from the equation. Some of it simply moves elsewhere, including to the hardware itself. There is still some trust involved in the hardware itself, including the assumption that the cloud provider and the hardware maker aren’t working together to circumvent the protections.
Then there’s the benchmark itself. MLCommons points out that it still needs to be carefully managed, no matter how well the questions are protected.
The pilot ran Gemini 2.5 Flash Lite on a single H100 80GB Confidential GPU. Still, the researchers are already looking at clusters of H100 and B200 GPUs connected via encrypted links to eventually evaluate models that are too large to fit on a single GPU.
If this approach catches on, developers could have another way to look beyond the benchmark score, with some proof that the company behind the model didn’t see the test before the results came out.
If this approach catches on, a benchmark score could offer some proof that the company behind the model didn’t get to see the test first.
AI coding company Replit is throwing its weight behind the model-routing trend by making its “intelligent model routing” system the default across every account.
The system automatically selects the underlying model to handle a task as it evolves, with Replit weighing quality, speed, and cost in its routing decisions.
The company says the feature, dubbed Auto mode, will become the default option for all users, though Core and Pro subscribers can still override it and manually select models when they want more control.
Model-routing momentum
The announcement comes hot on the heels of a flurry of activity in the model-routing realm. Earlier in August, Stripe agreed to acquire model gateway platform OpenRouter for a reported $8 billion. On the very same day, Ramp launched Router.com, which routes requests to the lowest-cost model that meets a specified performance bar.
Before all that, in July, SpaceX-owned Cursor launched its own router, which automatically selects models for coding requests and claims to deliver comparable performance at a substantially lower cost. Meanwhile, Meta is reportedly developing an internal router called “Switchboard” that scores coding tasks by difficulty and sends simpler jobs to cheaper models.
“Across one model family, per-token rates can span orders of magnitude. At the same time, the intelligence of cheaper, smaller models is now much closer to their larger frontier counterparts, providing us a lot of room for cost optimizations.”
Michele Catasta, president and head of AI at Replit, says that one reason for the wider push into routing is simple economics — the growing gap between what models cost and the level of capability developers actually need for a given task.
“Across one model family, per-token rates can span orders of magnitude,” Catasta tells The New Stack. “At the same time, the intelligence of cheaper, smaller models is now much closer to their larger frontier counterparts, providing us a lot of room for cost optimizations.”
Replit, for its part, has been moving in this direction for some time. Catasta says that the company has spent recent months experimenting with early versions of Auto mode, subagent routing, and multiple iterations.
“Like any pivotal launch, we thoroughly tested Intelligent Model Routing in beta for a long period of time before we decided to release it in public,” he says. “The most important learning is understanding from first principles the failure modes of every experiment, so we could keep hill climbing on the final system that we just shipped.”
Enter Auto mode
The foundation of that work surfaced last week when Replit introduced Free Mode, a lower-cost Agent mode that doesn’t consume usage credits and uses Auto to choose the model on theuser’ss behalf, subject to usage limits.
Now, that same Auto routing approach is being pushed more broadly across Replit. The company says intelligent model routing will become the default across every account, with all users starting in Free Mode and Replit deciding which model is best suited to the task.
Free Mode, it’s worth noting, isn’t “free” in the sense of unlimited usage. When it launched, Replit made it available to Core and Pro subscribers without consuming their usage credits, but imposed limits that reset every five hours, with higher allowances for Pro users. In Free Mode, users cannot manually select a model.
Core and Pro subscribers can, however, switch to Replit’s Power or Max modes, where they can turn off Auto and choose a model themselves. Replit may also suggest moving a task into one of those higher-powered modes when it determines that more capability is required, though those modes can incur usage costs.
Auto mode in Replit
For Enterprise customers, meanwhile, administrators can restrict Auto to an approved set of models for each workspace, allowing Replit to continue routing tasks automatically while keeping model choice within company policy.
Replit, by contrast, isn’t making ownership of the underlying model layer central to its pitch. Instead, it’s betting that controlling the agent and the systems around it gives Replit enough insight to make better model-selection decisions on the fly.
“Replit has owned, from the start, both the agent harness and the infrastructure surrounding models which in turn allows us to train sophisticated model routers.”
“Replit has owned, from the start, both the agent harness and the infrastructure surrounding models which in turn allows us to train sophisticated model routers,” Catasta said. “Only in this way can we always offer useful intelligence to our users at the most competitive price point.”
That becomes particularly relevant as an Agent task unfolds, with Replit noting that its system can change which model it uses as the task develops, seeking a better trade-off between capability and cost at different points in the process. But for Catasta, that kind of dynamic routing is still only one part of a much broader research problem around how agents should use models.
“Model routing is still in its early development phase, and we expect further research will move the needle on serving the best intelligence when customers most need it,” Catasta explains. “Routing is but one piece of the puzzle that is tightly integrated to many other aspects of our harness research.”
“No third-party router company could reproduce the same results for our own agent.”
Replit also argues that seeing how people use its own Agent gives it an advantage that a standalone routing provider would struggle to reproduce. Catasta says a router has to infer the nature, difficulty, scope, and intent of a request, with Replit able to train against proprietary usage data and observe those signals across its user base.
“No third-party router company could reproduce the same results for our own agent,” he says.
Nvidia has reportedly agreed to buy Hugging Face for $12.9 billion, putting one of the biggest names in AI hardware in charge of a platform developers rely on to find and run open models.
The Information first reported the deal Wednesday, citing a person familiar with the agreement. Nvidia and Hugging Face had not publicly confirmed it as of publication.
Hugging Face doesn’t push developers toward one chipmaker, which is what makes the acquisition interesting. Its Optimum libraries work with Nvidia’s TensorRT-LLM and also support hardware from AMD, Intel, and AWS. Projects such as Optimum AMD and Optimum Intel let developers run Transformers and Diffusers models on non-Nvidia hardware.
The company also plays a role in what happens after a developer chooses a model, including how easily they can get it running on the hardware they want to use. The problem for Nvidia might be this: It is buying a platform whose value depends on openness and hardware neutrality, but if the purchase means Nvidia hardware is favored, that value might diminish.
The company also plays a role in what happens after a developer chooses a model, including how easily they can get it running on the hardware they want to use.
Hugging Face already sits between the model and the chip
Hugging Face has expanded well beyond file hosting. With Inference Endpoints, developers can deploy a model from the Hub while Hugging Face handles the underlying infrastructure.
Those hosted deployments can run on AWS, Microsoft Azure or Google Cloud, but most of the GPU options Hugging Face lists are Nvidia chips, including the T4, L4 and A100. That gives developers a wider choice of hardware through Hugging Face’s open-source libraries than through its hosted services.
If the deal goes through, Nvidia would own both sides of that experience.
Deployment defaults favor Nvidia
NIM (Nvidia Inference Microservices) already works with models hosted on Hugging Face. Developers can point NIM to an hf:// repository path and pull the model directly from the Hub.
Owning Hugging Face would give Nvidia more room to bring NIM and CUDA-optimized containers directly into the deployment experience. Nvidia hasn’t announced plans to make NIM the default, and support for AMD and Intel could remain exactly where it is.
Owning Hugging Face would give Nvidia more room to bring NIM and CUDA-optimized containers directly into the deployment experience.
The bigger question is what happens over time. Nvidia could provide earlier support for new models on its own hardware or make deployment easier. At the same time, AMD, Intel, and AWS may have to reconsider how much engineering work they want to contribute to integrations maintained within a competitor-owned platform. Some of that work could eventually move elsewhere.
For developers, the difference may come down to which path requires less work. A competing chip doesn’t have to disappear from Hugging Face to become less appealing if an Nvidia model deployment takes fewer steps. We’ve seen a similar fight over the layers between AI models and the developers using them, with Cloudflare building more of that infrastructure itself.
Open models counter custom chips
That tension also helps explain why Hugging Face could be worth considerably more to Nvidia than its revenue alone would suggest.
Nvidia has been expanding its own Nemotron family of open models while investing heavily across the AI ecosystem. At the same time, some of its biggest customers are working to reduce their dependence on Nvidia hardware.
Open models are often expected to run in very different environments, and Hugging Face helps developers make that possible. A model found on the Hub might end up running on Nvidia hardware, an AMD GPU, or a cloud accelerator.
That flexibility is part of what Nvidia would be buying. Pushing Hugging Face too heavily toward its own hardware could make the platform less useful to developers who rely on it to work across different systems.
Interest in those models is also growing. Models from companies including DeepSeek, Moonshot AI and Z.ai have narrowed the gap with proprietary systems. At the same time, Hugging Face CEO Clément Delangue told The Information in June that the company had doubled its number of paying subscribers during the first six months of 2026. Delangue later said the company was “close to profitability.”
The Information puts Hugging Face’s annualized revenue at about $150 million. Against a $12.9 billion price tag, that’s a multiple of roughly 86.
So Nvidia would be paying for much more than Hugging Face’s current business. It would be buying a place developers already turn to when they want to work with open models, including models that don’t have to run on Nvidia hardware.
It would be buying a place developers already turn to when they want to work with open models, including models that don’t have to run on Nvidia hardware.
Community resists easy forking
Buying Hugging Face wouldn’t give Nvidia control over everything developers find there. Libraries such as transformers and diffusers are open source, and models on the Hub remain subject to their own licenses. Openly licensed models can still be hosted elsewhere, while the underlying libraries can be forked.
Much harder to recreate is the community Hugging Face has built around them. Developers already know where to look for models and have built workflows around the Hub and its integrations.
OpenAI’s unreleased foundation model, codenamed Astra, is already working directly inside the company’s internal codebase, taking on experimental work that previously required as much as a week from a human researcher.
As part of a series of interviews with Time, OpenAI chief scientist Jakub Pachocki said Astra can take an idea for an experiment, turn it into code, run it, and return the results. A step beyond asking AI to fix a bug or write a function, OpenAI is essentially testing what happens when you hand the agent the experiment itself and let it figure out the steps in between.
For developers, Astra offers a glimpse into the future of coding agents. But giving an agent that much freedom creates another problem. Astra may already be powerful enough to trigger OpenAI’s highest-level cybersecurity safeguards.
OpenAI is essentially testing what happens when you hand the agent the experiment itself and let it figure out the steps in between.
Persistent agents change everything
OpenAI CEO Sam Altman described what the company is building as “persistent agents,” or systems that can keep working without needing a person to prompt them through every step.
Coding agents can already dig through a repository, change files, run tests, and try again when something breaks. Persistent agents are meant to keep going without a developer guiding them through each step.
That also changes what developers need from the tools around the agent. An IDE gives it somewhere to work, but a long-running agent needs infrastructure that can keep it running safely without constant oversight.
Multi-agent coordination at scale
In one demonstration witnessed by Time, 16 Astra agents worked together on a research-level math problem, splitting it into smaller pieces and then bringing their work together into a proposed solution.
For developers, it’s not hard to imagine that same setup applied to a large software project, with different agents working on different pieces at the same time. OpenAI is already experimenting with that kind of coordination, along with agents that can stay on a job for much longer.
Running several agents at once also complicates the infrastructure behind them. Developers need a way to keep the whole operation under control. Giving agents that much freedom, though, has created another problem for OpenAI: keeping them under control.
Under the company’s framework, hitting that threshold brings stricter safeguards for how the model can be used.
OpenAI had already seen what could go wrong with agents that have access to tools. During a cybersecurity test, one of its internal AI agents escaped its sandbox and accessed Hugging Face systems without authorization. Astra was not the model involved, but the incident led OpenAI to pause some frontier-model research workloads while it tightened the infrastructure used to run them.
And OpenAI isn’t the only company running into this problem. Google’s AI coding agent recently broke out of the boundaries of its IDE. The circumstances were different, and the incident was less serious, but both point to the same problem that happens when giving an agent more freedom to act; you also have to make sure it stays where you put it.
Astra is now running under OpenAI’s strictest security controls. Some training and evaluation workloads have resumed, but OpenAI says a “significant number” are still paused while it upgrades the infrastructure behind them.
Astra is now running under OpenAI’s strictest security controls.
Monitoring costs real compute
The company also says it is monitoring Astra more closely when it uses tools, watching for behavior that could signal it is going beyond what it’s allowed to do. OpenAI estimates that monitoring adds about 20% to the inference compute for those workloads.
Time reports that OpenAI still plans to release Astra, although there’s no launch date yet. Until then, OpenAI is working through the same challenge developers could eventually face as agents work for longer stretches without human supervision and begin coordinating with other agents.
As agents take on more work, the IDE is only part of the picture. Developers also need a way to see what those agents are doing and keep them from going where they shouldn’t. OpenAI is already seeing the cost, with monitoring alone adding about 20% to Astra’s inference compute.
That kind of overhead could become another cost developers have to account for, along with the governance around autonomous agents that sets boundaries on what they can do.
OpenAI is already seeing what that costs, with monitoring alone adding about 20% to Astra’s inference compute.
Perplexity shipped Portable Computer this week, the local-first version of its Computer agent running on an Nvidia DGX Spark workstation, and the bill is steep: A DGX Spark starts at $4,700, and even an aging 24GB RTX 3090 sells well above $1,500.
The architectural choice beneath the surface deserves as much attention as the price tag: Most platforms building agents address reliability with greater intelligence. That typically includes a larger orchestrator model, a planner model, or a critic model reviewing the work.
Probabilistic reasoning proposes the next action, and deterministic software decides whether to execute it.
Perplexity separated the two jobs instead of stacking them. Probabilistic reasoning proposes the next action, and deterministic software decides whether to execute it.
The loop controller is code, and the decisions are still a model
Perplexity uses the term “orchestrator” to refer to the runtime controller rather than the planning model. That controller assembles context, enforces policy, and executes approved tool calls inside an OS-level sandbox. The company says it is deterministic code, not yet another model. The local model proposes the next action, including which tool to call and when to ask a cloud advisor for help. The split resembles a control plane architecture, where reasoning suggests, and inspectable software retains authority. Nate Kupp, Perplexity’s vice president of Computer Enterprise and Infrastructure, tells The New Stack that the harness accounted for most of the engineering work.
Same weights, better scores
Perplexity kept the base model and the silicon constant, putting Qwen3.8-27B on the same DGX Spark across three agent stacks. On its internal Local Knowledge Work Bench, a held-out set of 53 tasks, the company reported 82.6% for Computer. Pi scored 77.6% and Hermes 74%.
On ParseBench-100, a subset covering charts, layouts, tables, and formatting, the gap widened considerably. Perplexity reported 65.1% for Computer, while Hermes reported 34.6% and Pi reported 13.9%.
Because the base weights were constant, the gap measures the system around the model rather than a better model. That does not mean weights stopped mattering. Perplexity post-trained Qwen into PPLX 27B and reported 85.4%, above its own base-model number. Harness engineering and post-training are significant factors of this approach.
The word harness also covers a lot of ground, including prompts, tool schemas, context management, verification hooks, and document processing. Orchestration code is one part of that surface.
The security boundary lives outside the model
The sandbox is the boundary, not the determinism. Perplexity says the sandbox restricts processes, filesystem paths, and network access. If the sandbox is unavailable, the harness disables itself before making any tool call. Deterministic code is valuable here because it makes policy easy to inspect and helps the system fail safely. Deterministic code can still ship a vulnerability or faithfully execute a permitted mistake.
The key distinction is not between model orchestration and code orchestration. It is whether permission is enforced by something other than asking a model in plain English to comply.
The key distinction is not between model orchestration and code orchestration. It is whether permission is enforced by something other than asking a model in plain English to comply.
The same discipline shapes how context gets spent. Perplexity reports that Qwen3.8-27B advertises a 260,000-token context window but begins to struggle beyond 100,000 tokens. The harness therefore keeps the core prompt and toolset small and loads skills on demand. Commonly used connectors became command-line tools rather than full Model Context Protocol definitions sitting permanently in context.
Deterministic execution cannot rescue reasoning that exceeds the local model. On Terminal Bench 2.1, Perplexity reported 59.6% running locally. Letting the local agent consult Claude Opus 5 raised it to 73.0%, compared with 82.4% when Opus 5 worked alone. All of this remains vendor-reported evidence on a bench that the company has yet to open-source.
For enterprises evaluating local agents, the component to scrutinize is the layer that grants and denies authority, because that is where the platform’s engineering is most evident.
Agentic reinforcement learning has been suffering from a disconnect, an uncoupling, and a misarticulation. The polarity arises from how a training engine handles resource management, compared with how a post-training live production harness does.
Microsoft wants to ensure the production harness is engaged from the start to oversee infrastructure services and agent interactions, during both initial training and subsequent reinforcement learning actions.
Redmond’s Microsoft Research division first introduced the Agent Lightning framework in August 2025 as an infrastructure concept for agent optimization to address structural challenges in post-training LLM-based agents as they enter reinforcement learning processes. Microsoft subsequently launched the Agent Lightning v1.0 release with a commit tagged on GitHub on August 16.
Using Agent Lightning v1.0 on what Microsoft has called “modest compute” 6K training examples, reinforcement learning improves Qwen3.5-9B on OpenAI’s SWE-bench Verified benchmark from 41.8% to 56.4%, an absolute 14.6-point gain.
“Using Agent Lightning v1.0 on ‘modest compute’ 6K training examples, reinforcement learning improves Qwen3.5-9B on OpenAI’s SWE-bench Verified benchmark from 41.8% to 56.4%, an absolute 14.6-point gain.”
Who owns the interaction loop?
In traditional agentic reinforcement learning, the training engine owns the interaction loop, i.e., steps including observing the environment, selecting an action based on the policy, executing the action, receiving a numerical reward, storing and updating the policy… and so on.
In harnessed agentic reinforcement learning, the harness owns context construction, tool execution, and the agent–environment loop. The training engine observes only a sequence of LLM request–response pairs across a service boundary. Therefore, developers do not need to reimplement the agent loop within the training environment.
According to a collected group of software engineers at Microsoft, because an AI model harness owns the loop that governs infrastructure access and operations procedures during harnessed agentic reinforcement learning, it “introduces challenges” including retokenization (re-segmenting text into new tokens during active training), sample merging, advantage calculation, loss normalization, and training backend scheduling.
All of which, if not addressed, can result in ineffective or unstable training.
How Agent Lightning v1.0 turns the tables
“[In Agent Lightning v1.0] the harness, rather than the trainer, owns context construction, tool execution, and the agent–environment interaction loop, while the training system observes and optimizes the resulting model calls across a service boundary. This formulation preserves the harness’s deployment-time context policy, tool protocols, and execution semantics without requiring its agent loop to be reimplemented inside the RL framework,” explained the Redmond team.
For coding agents, the team has said that it finds existing reinforcement learning frameworks provide limited support, including a lack of data and complete training scripts, and a reliance on large-scale computational resources. To address this gap, with Agent Lightning v1.0, Microsoft provides a “complete data-cleaning pipeline” and reproducible training scripts built on open-source datasets and models.
Is this the end of the training time liability?
For users inside Microsoft environments, this might feel like good news. Machine learning specialists and platform engineers may have a good harness or a good set of reinforcement learning tools; they would rarely have both.
Rather than having to hard-code retokenization, advantage calculation, and reward shaping every time they wanted to train an agent to call external services in reinforcement learning procedures, they can keep their existing agent architecture as an asset, rather than treating it as a training-time liability.
Nebraska-based software engineering researcher Md Rashedul “Rashed” Hasan tells The New Stack that training on the exact production harness matters, but not only for its efficiency and benchmark gains.
Training through the real harness keeps semantics intact
“It reduces train–serve mismatch,” Hasan says. “If you train inside a simplified trainer loop and deploy inside a different harness, tool protocols, context policy, and recovery behavior can all drift. Training through the real harness keeps those semantics intact, so gains are more likely to transfer to production behavior, not only to a lab environment.”
Hasan thinks that Microsoft has clearly named the paradigm, kept the core framework small, and shipped what he defines as a “concrete coding-agent pipeline” with open data and scripts.
“In terms of who this will appeal to, it’s application and platform engineers who already have a production agent harness (such as a coding assistant or support-triage agent), and want to improve the underlying model with reinforcement learning without rewriting deployment logic to fit a training framework. Reinforcement learning and machine learning platform teams would also use it when they need a thin, reproducible testbed for harnessed agentic reinforcement learning,” Hasan clarifies.
He agrees that the SWE-bench Verified lift with only 6K examples is “useful proof” that harnessed reinforcement learning can move a hard-coding benchmark without forcing teams to reimplement their agent stack within the trainer.
“Environment setup for coding agents, reward design, evaluation fidelity, and the retokenization, sample-merging, advantage, and loss-normalization details are still easy to get wrong. Adoption will also depend on whether teams can integrate this proxy pattern into existing orchestration, observability, and safety controls,” cautions Hasan.
“Environment setup for coding agents, reward design, evaluation fidelity, and the retokenization, sample-merging, advantage, loss-normalization details are still easy to get wrong. Adoption will also depend on whether teams can integrate this proxy pattern into existing orchestration, observability, and safety controls.”
Killing train-serve skew, the oldest & most expensive bug in machine learning
Colorado-based data science professional Priyank Jain tells The New Stack that training through the production harness to strip the framing away is really all about “killing train-serve skew”, which is the oldest and most expensive bug in applied machine learning.
“Models rarely blow up in production because the math was wrong,” Jain says. “They blow up because the training setup quietly lied to them about what production actually looks like. Training through the same harness that serves the agent is the right instinct, and honestly it’s overdue. The accuracy bump is nice, but the real win is that the thing you optimized is finally the thing you shipped.”
In terms of what Microsoft is getting right here, Jain says it shows Redmond wants to “meet developers where they already are,” i.e., allowing users to improve an existing agent with reinforcement learning without rewriting their deployment just to please a training framework.
“That’s a genuinely good call, and it opens this up to folks who aren’t reinforcement learning specialists. What I’d worry about is that it also lowers the bar for people to run reinforcement learning they don’t fully understand. The hard part was never wiring up the loop; it’s designing a reward that survives contact with a model actively looking for the path of least resistance. Make that part easy, and you’ll get more people optimizing the wrong thing faster,” Jain advises.
“3500 lines of code is small enough that an infrastructure engineer can actually read it before trusting it… and that’s a good thing.”
Just 3,500 lines of core Python code
Built with “simplicity as its first principle”, the entire framework consists of around 3,500 lines of core Python code.
Infrastructure engineering developer and founder-developer of Tooldex, a platform that autodiscovers MCP servers across LLM agents, Ria Banerjee, tells The New Stack that the simplicity element here is a positive, i.e. 3500 lines of code is small enough that an “infrastructure engineer can actually read it before trusting it,” and that’s a good thing.
“But in terms of who would actually use this tool, the honest answer is fewer teams than the framing suggests,” Banerjee says. “Realistically, you would need a GPU cluster and a Kubernetes cluster under your control. An app developer who built a support-triage agent on LangChain isn’t running that. So the real audience is platform teams that already employ infrastructure engineers.”
“Because the harness is where the behavior actually lives, but you’re now baking your harness’s quirks into the model weights… so you change your retry logic next quarter, you’ve silently shifted what the model was trained on. So now, you have to version your harness,” adds Albuquerque-based Banerjee.
Microsoft has released the complete workflow and training scripts to facilitate reproducible, harnessed agentic RL in Agent Lightning v1.0 on GitHub under the MIT license, including data cleaning and reward-hacking prevention.
Burns & McDonnell and Gritt have entered into a strategic partnership focused on evaluating and deploying AI-powered robotics technology on utility-scale solar projects. The technology will help improve safety, enhance project predictability and support construction teams as demand for new energy infrastructure continues to grow. Gritt combines AI and robotics to automate labor-intensive tasks on […]
It’s not just about the installation of more solar panels for the next generation of renewable energy. It has to do with the utilization of every single square meter of PV surface in an even better way, and with the use of a combination of new PV technologies and smarter installation methods, energy storage solutions, […]
When OpenAI unveiled Jalapeño, its first custom inference chip, in June, the company made some big promises. The chip, developed with Broadcom, was built from scratch for large language model inference, with OpenAI saying early testing showed substantially better performance per watt than existing accelerators. At the time, though, OpenAI didn’t release the detailed performance results to back that up.
On Tuesday, OpenAI published its first results from working Jalapeño silicon across GPT-OSS 120B, DeepSeek R1 and Kimi K2.5. The results show what OpenAI was aiming for with Jalapeño: higher throughput without the longer response times that can come with it.
“Agents need to complete many steps in sequence, so delays can compound across an entire task.”
Agents compound inference delays
An agent may call a model over and over as it works through a task, using tools and deciding what to do next based on the results, which means a delay that barely registers during a single inference can become much more noticeable when it happens repeatedly over the course of a longer task.
“Agents need to complete many steps in sequence, so delays can compound across an entire task,” OpenAI said.
Jalapeño was designed with those delays in mind. Different parts of running a large language model place different demands on the hardware, with the initial prompt requiring more compute and the response generation putting more pressure on memory bandwidth. Every time data has to move between cores and chips, that can add even more waiting.
Jalapeño takes a different approach, cutting down on that waiting without optimizing one part of the process at the expense of another.
“Agents need to complete many steps in sequence so that delays can compound across an entire task,” OpenAI said.
That helps explain some of the choices OpenAI made with Jalapeño. Running a large language model puts different demands on the hardware at different points: processing the initial prompt requires a lot of compute, while generating the response token by token relies more heavily on memory bandwidth. There’s also time lost whenever data has to move between cores and chips, leaving parts of the system waiting for what they need.
The idea is to reduce that waiting without optimizing one part of the process at the expense of another. Model state, including the KV cache used while generating a response, can be kept local, while Jalapeño’s networking allows more of the workload to stay within the same connected system. That means less time spent moving data around as the workload shifts between compute and memory.
Jalapeño’s first public benchmarks
OpenAI put Jalapeño through InferenceX, SemiAnalysis’ public benchmark for AI inference, using GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. Across the three models, Jalapeño handled 1.5 to 1.9 times more work per watt while cutting end-to-end latency by 1.7 to 3.6 times. On highly interactive workloads, OpenAI says it was 2.1 to 4.1 times faster than the systems it compared against.
The differences become particularly large when Jalapeño is compared at the previous best time-between-tokens operating point. OpenAI reported between 8.6 and 104.3 times more work per watt, depending on the model.
OpenAI based the power-efficiency comparisons on each accelerator’s published power rating. Jalapeño is rated at 700 watts, although the company says it never drew more than 550 watts during these tests. The bigger point is that OpenAI isn’t trying to improve throughput at the expense of response time, which is often the tradeoff with inference.
Batching more work can make infrastructure more efficient, but it can also mean making an individual user wait longer. OpenAI’s argument with Jalapeño is that an inference system increasingly needs to be good at both — particularly as the company continues cutting the cost of API access while also needing to keep interactive workloads responsive.
The bigger point is that OpenAI isn’t trying to improve throughput at the expense of response time, which is often the tradeoff with inference.
AI-generated code runs faster
OpenAI used its own models throughout Jalapeño’s development, helping the hardware team move from initial design to tapeout in nine months by exploring implementations and shortening design, measurement, and verification cycles. The work didn’t stop once the chip was built.
The company says AI-generated implementations of selected GPT-OSS attention and mixture-of-experts blocks ran 1.5 to 1.8 times faster than versions written by its own experts. That doesn’t mean the entire model ran that much faster, but it does show what OpenAI is trying to do with Jalapeño: make the chip straightforward enough for AI, not just humans, to program and optimize.
Engineers describe work using local tensors, explicit communication and predictable synchronization, giving AI a way to help determine how that work should be mapped, placed and scheduled across the system. That could make it faster to adapt the chip as new models come along, although OpenAI says each new model family still requires its own kernels and optimizations.
Custom silicon meets model roadmap
Using Codex with GPT-Astra and earlier OpenAI models, the hardware team brought three open-weight models that weren’t part of Jalapeño’s original production plan to high performance within two months. That fits with OpenAI’s broader plans for Codex, which the company has said is still early in its development, and shows how it could eventually play a role well beyond writing code.
OpenAI plans to start using Jalapeño in its own infrastructure by the end of the year, and it’s already working on the next two generations. The company will continue to use accelerators from Nvidia and other partners, but building its own chips gives OpenAI more control over how the hardware evolves alongside its models.
OpenAI plans to start using Jalapeño in its own infrastructure by the end of the year, and it’s already working on the next two generations.
What do you do when a technology you’ve become dependent on belongs to someone else? You buy the startup behind it, of course. And that’s exactly what data warehouse company MotherDuck has done with Tower, a data infrastructure startup whose technology was already powering MotherDuck’s AI-built data pipelines.
The deal, announced on Tuesday, is MotherDuck’s first acquisition in its four-year history, bringing both Tower’s technology and team in-house as MotherDuck pushes further into AI agents that can build and operate data pipelines.
Tower takes flight
Tower was founded out of Germany in late 2024 by ex-Snowflake engineers Serhii Sokolenko (CEO) and Brad Heller (CTO). Their pitch: once a developer, or an AI assistant, has written the code for a data pipeline, someone still has to package it, deploy it to the right infrastructure, wire up credentials, and maintain it — the unglamorous work, Sokolenko told The New Stack back in March, that amounts to data engineering’s “last mile.”
Tower, essentially, is a managed runtime for Python pipelines — it packages the code, deploys it, and keeps it running in production. It also offers tools built on top of that runtime, like the browser-based AI agent Tower Control that allows users to describe the pipeline they want in plain language.
With Tower Control, users can describe the pipeline they want in plain language.
Control can then generate the code, deploy it as a Tower app, and run it — effectively taking the process from prompt to production without the developer having to set up the underlying runtime themselves.
Control can generate the code, deploy it as a Tower app.
MotherDuck, for its part, is a serverless data warehouse built on the open-source database DuckDB, founded in 2022 by Jordan Tigani, a former engineering lead at Google focused on BigQuery. The company has raised some $100 million since its inception.
MotherDuck’s original pitch leaned on speed and local compute: queries could run on a laptop via DuckDB, in MotherDuck’s cloud, or across both — a departure from the likes of Snowflake, Databricks and BigQuery’s cloud-first model. More recently, MotherDuck has extended that approach to AI agents, using MCP to let agents interact directly with data. And things took a more operational turn in June with the launch of Flights, a feature that exposes a general-purpose Python runtime through the same MCP server, letting agents create, run and schedule data pipelines.
And Tower, as it turns out, was the key infrastructure underpinning Flights.
‘We became their largest customer almost overnight’
Tower’s involvement with MotherDuck actually predates Flights. Tigani says MotherDuck had initially been looking for a third-party tool it could recommend to customers as an easier way to get data into its warehouses. But then advances in AI changed the nature of the problem the company thought it needed to solve.
“When AI suddenly started to be able to solve data problems, we realized we were thinking about the problem wrong.”
“When AI suddenly started to be able to solve data problems, we realized we were thinking about the problem wrong,” Tigani tells The New Stack. “Claude can solve that problem we were trying to address by writing the connectors to help people move their data, but what it can’t do is the sandboxing and scheduling.”
That left MotherDuck needing somewhere to safely execute the code those agents generated, manage credentials and run jobs on a schedule. And as luck would have it, Tower already provided those capabilities.
“It perfectly solved our problem and let us ship Flights in only a matter of weeks,” Tigani adds.
For MotherDuck, Tower had supplied the missing execution layer; for Tower, that realization translated into a significant customer relationship. “We became their largest customer almost overnight, and our teams have been shipping together ever since,” Tigani says.
Having Tower on its radar so early also gave MotherDuck a chance to test the technology — and the team behind it — before deciding whether to build something similar itself. Tigani says the calculation ultimately came down to how quickly MotherDuck could get the capabilities it wanted into customers’ hands.
“It’s always tempting to build yourself, but after trying out Tower, we realized pretty quickly that there were a bunch of problems we were going to have to solve to make our underlying infrastructure actually work well, and Tower was pretty much exactly what we needed,” he says.
Ultimately, the more central Tower became to what MotherDuck wanted to build, the stronger the case became for owning the technology outright. Once Tower was executing jobs created and scheduled inside MotherDuck, Tigani argues, customers would inevitably hold MotherDuck accountable for the security, reliability and behavior of that runtime.
“There’s a rule I’ve relearned at every infrastructure company I’ve worked at: you can rent a feature, but you can’t rent a foundation,” Tigani says. “When an agent inside MotherDuck builds a job and schedules it, the thing executing that job is our product — whatever logo is on it.”
“There’s a rule I’ve relearned at every infrastructure company I’ve worked at: you can rent a feature, but you can’t rent a foundation.”
One example of what MotherDuck now hopes to do with that technology involves bringing Flights together with Dives, an AI-generated data visualization feature it debuted in February. Tower can generate stable URLs for jobs running through Flights, effectively allowing those jobs to act as data APIs that a Dive — or another frontend — can call.
Tigani gives the example of an application displaying user recommendations. A Dive could generate the interface for viewing those recommendations, while a Flight could handle requests to create or modify them. Rather than giving the frontend broad write access to the underlying data, the Flight can constrain and validate what changes users are allowed to make.
“When you put them together, you can build rich applications,” Tigani says.
Tower’s next chapter
All of this raises an obvious question for Tower customers. Part of the startup’s pitch was that developers could use its runtime without tying themselves to a particular data platform, and now Tower itself belongs to one.
Tower co-founder and CEO Serhii Sokolenko argues that MotherDuck represents a different kind of home to that of the industry’s larger cloud data platforms. His case is that Tower can become more deeply integrated without being bent around an architecture established long before AI agents entered the picture.
“Joining a hyperscaler usually means adapting to its legacy architecture,” Sokolenko tells The New Stack. “Joining MotherDuck lets us help shape where data and AI infrastructure are heading.”
“Joining a hyperscaler usually means adapting to its legacy architecture.”
There is still a trade-off, though. Tower is giving up some of the breadth that came with being database-agnostic in exchange for building much more specifically around one platform — a bet that tighter integration will ultimately produce a better experience than supporting many systems at arm’s length.
“By focusing Tower’s Pythonic compute specifically on MotherDuck, we’re trading broad, basic connectivity for deep, native execution,” he says.
Sokolenko’s argument is that the “lock-in” question then shifts down a layer. Tower may now be tied much more closely to MotherDuck, but because MotherDuck itself is built on DuckDB, he says the underlying data remains open and portable. The aim is to bring the runtime, agents and warehouse closer together without trapping the data itself inside a proprietary system.
That also helps explain why MotherDuck’s hybrid execution model appealed to Tower. DuckDB lets work move between local and cloud compute, which Sokolenko sees as closely aligned with Tower’s own direction.
“This directly matches Tower’s vision — allowing business users and agents to move seamlessly from local data exploration to cloud production execution,” he says.
For Tower’s existing customers, the immediate future means a move toward MotherDuck. Sokolenko says Tower customers are already in discussions with MotherDuck about migration paths, while people who have used Tower previously will be invited to try MotherDuck and its broader agentic data capabilities.
Tigani confirms that MotherDuck is working to move existing Tower customers onto Flights, though he concedes that the two products aren’t entirely identical. “There are a couple of differences, and we’re working on closing the gaps to make the transition more seamless,” he says.
Tower’s technology will meanwhile be folded more deeply into MotherDuck itself. Today, MotherDuck has two separate sandboxed, on-demand runtimes: Flights, which is backed by Tower, and Ducklings, its serverless DuckDB instances. Tigani says the plan is eventually to merge the two, combining the near-instant startup of Ducklings with the more robust sandboxing provided by Tower jobs.
Surfing the agent wave
MotherDuck is hardly alone in pushing data agents beyond answering questions. Databricks’ Genie Code can generate and run code, build pipelines and debug failures inside Databricks. Snowflake, meanwhile, has been moving in a similar direction with the likes of CoCo, its AI coding agent, while newer CoCo Automations can schedule unattended agent runs inside Snowflake-managed sandboxes.
The details differ, but both point toward a broader shift in the data industry: giving AI agents the infrastructure to act on data and operate the systems around it, rather than simply query what is already there.
“AI makes it possible to build useful features that we could never fathom five to 10 years ago. The platform is the most complex part of the data estate, and so warehouse vendors are well positioned to be at the center of whatever new patterns emerge.”
Tigani has been predicting something close to this for some time. Earlier this year, he outlined a future in which data engineering increasingly becomes an agent-supervision problem, with agents handling tasks such as building and repairing pipelines and responding to changes in schemas and data quality, while humans oversee their work. He also previously likened the advance of LLMs to a wave that data companies will have to learn to ride.
“The way I like to think about it is this — data platform vendors are reacting to new opportunities to make lives better for their customers,” Tigani says. “AI makes it possible to build useful features that we could never fathom five to 10 years ago. The platform is the most complex part of the data estate, and so warehouse vendors are well positioned to be at the center of whatever new patterns emerge. The Tower acquisition gives us a platform for deploying, tracking, and scheduling data agents, which should set us up well to surf that wave.”
The launch, developed in close partnership with Nvidia, is one of the most aggressive attempts yet to move serious AI agent workloads off the cloud and onto local devices. The model, the user's files, and the work itself can all stay on the machine. Work completed locally consumes no billing credits, and the company says every task starts on the device by default — with the system asking permission before sending any individual step to a more powerful frontier model in the cloud.
"We've basically brought the exact same UI to a fully local app," said Nate, Perplexity's vice president of engineering for infrastructure and enterprise, during a press briefing Monday. "This incorporates the entirety of the agent harness and inference and everything needed to do work locally."
For Nvidia, which has spent the past two years selling the world on trillion-dollar AI data centers, the announcement signals something subtler but strategically important: the chipmaker believes local AI has crossed a threshold from hobbyist curiosity to practical tool — and it wants to sell the hardware that runs it.
"Local AI reached an inflection point," said Nader, Nvidia's director of developer technology, who focuses on developer tooling and open source. "For the longest time, it was hobbyists and enthusiasts, and they were running these quantized models that were quantized down to be super tiny... And while that's cool, it's not super practical. But all that changed with a lot of these new open source models that have come out that are super useful."
How Portable Computer packages a full local AI stack into a single app
Perplexity Computer, the company's agentic platform for knowledge work, orchestrates AI models, files, tools, and web access to complete multi-step tasks — reviewing folders of documents, analyzing data, producing reports, and pushing results into business systems. Portable Computer replicates that experience locally: the local models, agent harness, inference engine, tools, app connectors, and a security sandbox come packaged together in a single system. That bundling is the point. With most local AI stacks today, users must assemble and operate those pieces separately — downloading model weights, standing up an inference server, wiring together tools, and tuning performance.
"Historically it's just been really painful to bring up the local AI stack," Nate said. "With Portable Computer, we really focused on just making this a really straightforward experience where you can get up and running very quickly."
In one demo Monday, the system played the role of a retail investor reviewing a folder of 1099s and investment documents — the kind of sensitive financial material many users would hesitate to upload to a cloud service. Running a 27-billion-parameter Qwen model at full GPU utilization on a DGX Spark, the agent reviewed each document and flagged cases where the hypothetical investor was paying unnecessary fees. The interface element that normally displays a running tally of cloud credits "is just parked at zero," Nate noted, "because all of this is happening on the device."
A second demo showed the hybrid side of the product. Playing a startup founder, Nate asked the agent to analyze a CSV of user funnel data locally, then push the finished analysis to a Slack channel using Perplexity's connector ecosystem — proof that local-first does not mean disconnected.
The system also connects to Google Drive, Gmail, and GitHub, and can escalate to a frontier cloud model when the local model hits its limits. At launch, users can set up Qwen 3.8 27B or PPLX 27B, a version Perplexity has post-trained on its own harness, with Nvidia's Nemotron 3.5 Lightning coming soon.
Portable Computer arrives today for Pro, Max, Enterprise Pro, and Enterprise Max subscribers on Linux, with Windows support following in September. Any RTX GPU with at least 24GB of VRAM — roughly a GeForce RTX 3090 or newer — clears the bar, a threshold Nate called "sort of the floor where we really want to make sure that we can deliver a great experience, but balance that with making it broadly available."
Why co-designing the model and agent harness beats general-purpose frameworks
Alongside the launch, Perplexity published a research paper arguing that effective local agents require the model and the agent harness — the scaffolding of prompts, tools, and orchestration logic around the model — to be designed together. The core insight: general-purpose harnesses assume a frontier model that can absorb enormous contexts, navigate sprawling tool surfaces, and plan over long horizons. Small local models buckle under those demands.
Perplexity found empirically that although models like Qwen 3.8 27B advertise 260,000-token context windows, they begin to struggle beyond 100,000 tokens. So the company built a deliberately minimal harness: a succinct system prompt, a small set of core tools, and capabilities that load and unload as on-demand "skills" rather than sitting permanently in context. It converted popular connectors like Gmail and GitHub from token-hungry MCP servers into compact command-line tools, added self-verification hooks that monitor the health of a task, and enforced always-on OS-level sandboxing. If the sandbox is unavailable, the harness disables itself rather than running tools unprotected — a contrast with open-source harnesses that run commands with the user's full permissions by default.
The benchmark results Perplexity reports are striking, though they come from the company's own evaluations. On its internal Local Knowledge Work Bench — 53 tasks spanning deep research, financial analysis, and document creation, which Perplexity says it plans to open-source — Computer running Qwen 3.8 27B on a DGX Spark scored 82.6%, versus 77.6% for the open-source Pi harness and 74.0% for Hermes running the identical model.
Perplexity's post-trained PPLX 27B pushed the score to 85.4%. The gaps widen dramatically on harder tasks: on BrowseComp, a web research benchmark, Computer hit 66.7% accuracy versus 50.2% for Pi and 43.9% for Hermes, while using 51% less wall time and 70% fewer tokens than Pi. On multimodal document understanding, Computer scored 65.1% against Hermes' 34.6% and Pi's 13.9%.
The token economics driving AI agents from the cloud to local hardware
The strategic logic behind the launch becomes clear when you consider how AI workloads have changed. Chat was bursty — a question, an answer, done. Agents are different.
"With agents, you want these agents always on if you can. You want the agents to really consume as many tokens as they can," Nader said. "What we're seeing is an insatiable demand for tokens, and that's something that makes local AI so great. As you saw through all these demos, you were not metered by the token. You were not paying for the token. So it's really killer for agents."
This reframes the value proposition of local hardware. An agent that runs for hours reviewing documents, verifying its own work, and iterating on analyses would rack up substantial API bills in the cloud. On a device the user already owns, the marginal cost of those tokens approaches zero. Perplexity's paper makes the enterprise version of this argument explicitly: as agents scale across individual workflows and entire organizations, token expenditure and data movement "become increasingly difficult to govern." Local-first execution addresses both at once — spend, because inference is free, and privacy, because sensitive tokens never leave the device boundary.
Perhaps the most commercially interesting result concerns the hybrid middle ground. On Terminal Bench 2.1, a challenging coding benchmark, the fully local Qwen model scored 59.6% at essentially zero marginal cost. Letting it escalate to a Claude Opus 5 "advisor" in the cloud raised the score to 73.0% at an estimated $0.415 per task. Running the frontier model alone scored 82.4% at $0.65 per task. Escalation, in other words, recovered roughly three-fifths of the gap to frontier performance at about two-thirds of the cost — and the user decides when that trade is worth making. Before any advisor call, the harness runs a PII classifier over the outgoing context and shows the user exactly what would leave the device. The remote model returns text guidance only; it never touches local files or tools.
Where Portable Computer fits against Ollama and the DIY local AI stack
Jason Hiner of The Deep View pressed the companies on how Portable Computer relates to existing local inference tools like Ollama. Nate's answer drew a clear line: the tools solve different layers of the problem.
"The majority of the effort here has been at the agent harness level," he said, noting that the system uses vLLM to host model inference underneath, with an advanced mode for users who want to plug in their own inference endpoint. "We've heavily post-trained both the Qwen and Nemotron models that we're working with in order to really get the best possible results... Our focus has been on really honing the whole stack, top to bottom, of the model inference and the harness together."
Nader put it more colorfully. "Just getting inference running really quickly on a Spark — there's a smooth path. You can use Ollama. You can get that set up. But then, as you start to do more complicated, more agentic things, then suddenly you need more perf. You start looking at different models. You start looking at different harnesses, and it's kind of like the ocean. The deeper you go, the deeper it gets."
The appliance-like pitch appeared to land with at least one attendee. Ben, who described struggling to set up his own DGX Spark despite being an engineer — "this experience sucks, we have to fix it" — said the product feels like the unlock "needed for people to really feel and understand what agentic means, and you need the right UX to make it happen." Nvidia also emphasized that the hardware scales: connecting two Sparks over shared memory runs frontier-class open models like DeepSeek's latest, and four can run GLM 5.2 or Nemotron Ultra. "I've even seen eight Sparks get connected," Nader said.
What the deepening Nvidia-Perplexity alliance means for both companies
The launch extends a partnership that has been building for more than a year. In June 2025, Nvidia and Perplexity announced a collaboration to bring sovereign AI models to European publishers and telecoms, part of CEO Jensen Huang's continent-hopping campaign to convince governments that, as the Associated Press reported from VivaTech in Paris, "every country needs a national intelligence infrastructure." The sovereign AI pitch — that data "belongs to your people, your country, your culture," in Huang's words — is philosophically the same argument Portable Computer makes at the scale of a single desk: intelligence you control, running on hardware you own.
There is a self-interested logic for both companies. Perplexity, which has raised capital at steadily escalating valuations while facing legal pressure from publishers over its content practices — including a lawsuit filed by The New York Times in December 2025 and an earlier public dispute with Forbes — gets a product whose economics don't depend on metering every token, and a differentiated wedge into privacy-sensitive enterprises in law, healthcare, and finance. Nvidia gets a killer app for DGX Spark, a device that, by the admission of attendees at Monday's briefing, has been easier to buy than to use. When one reporter asked whether a Spark might ship with Portable Computer and a Nemotron model preinstalled, Nader demurred without ruling it out: "That would be cool... the goal is just making sure that it's a super smooth experience for every user."
Questions remain. Perplexity's most impressive numbers come from its own internal benchmark, and the company acknowledges that compact models still trail the frontier meaningfully on hard reasoning tasks — advisor escalation "narrows but does not fully close the gap." The launch is Linux-only for now, the 24GB VRAM floor excludes the vast majority of consumer PCs, and Apple silicon — home to some of the most enthusiastic local AI tinkerers — is conspicuously absent from the roadmap. "We're very focused right now on Nvidia hardware," Nate said when asked.
But the direction of travel is unmistakable. Perplexity's researchers describe the launch as part of "a broader shift in which increasingly capable agents move from remote infrastructure to individual and local devices," and both companies are betting that advances in chips and open models will keep expanding what a box on a desk can do. During Monday's demos, the most telling detail wasn't a benchmark score — it was that credit counter in the corner of the screen, sitting motionless at zero while the agent churned through a folder of tax documents. For two years, the AI industry has measured its ambitions in gigawatts and tokens per dollar. Portable Computer proposes a different meter, one that never runs.
Put several AI bots to work, and a mistake by one may not stay within its assigned task. For example, that error could reach another bot’s files and login credentials, or even the computer running them all. Two releases this month offered companies very different ways of containing that risk.
On August 17, Nous Research announced that its Bot Mode would ship bundled and enabled by default in Hermes Agent v0.20.3, turning agent profiles into a roster of named bots that hand off work to one another. About a week earlier, SpaceXAI launched Grok Bot with almost the same interface: a sidebar of named teammates who sign in to your tools and keep working long after you close your laptop.
The interface converged within a week, but the answer to the question every platform team has to ask did not: When one bot goes wrong, what can it reach?
Four projects have now come to their own answer, and no two of them agree.
Grok Bot draws the line around the user account.
Hermes draws it around the profile.
OpenClaw draws it around an optional runtime sandbox.
Taken together, the documentation shows an industry converging on the persistent coworker interface far faster than it is converging on what constitutes an identity or a security boundary for it.
Four projects, four written answers
Every one of these products now offers the same surface. You create several named agents, assign them different jobs, and have them pass work among themselves. The naming convention alone suggests separation, since a bot called Expense Manager and a bot called Talent Scout sound like they occupy different rooms in a shared office.
The documentation says otherwise, and it says something different in each case. The unit of isolation is the account in one product, the profile directory in another, an opt-in container in a third, and the deployment topology in the fourth. Those four units are not interchangeable, and an operator who assumes the roster itself is the boundary will be right in exactly one of the four cases.
Is Grok Bot confused about what it wants to be?
SpaceXAI’s launch post leads with the promise that bots have their own computer. The documentation, last updated the same day, describes a single persistent cloud computer assigned to the user account rather than to any individual bot. Browser cookies and signed-in sessions are shared across the roster, files are visible to every bot, and command-line credentials are shared. One bot can pick up work that another bot saved.
Each bot gets its own screen on that machine, which allows several of them to run browser and desktop tools in parallel. SpaceXAI is direct about what those screens are not. The documentation calls them “separate work surfaces, not separate security boundaries.” It then instructs operators to keep a credential or file off the machine entirely if another bot on the account cannot use it.
The consequences run further than credentials. Signing in for one bot makes that session available to the others because the browser is shared. Installed connectors are account-wide, and their availability is not isolated to a single bot. The shared workspace sits at /workspace and is designed to survive computer updates and recovery, so the durable state is shared across the whole roster.
None of this is an implementation accident. It is what makes handoffs between bots cheap, and cheap handoffs are the product. But an operator reading only the launch page would build a mental model that the documentation contradicts, and that gap is where the risk sits.
Hermes gives each bot its own profile
Nous took the opposite architectural position. In Hermes, a bot is a profile, and each profile has its own configuration, memory, skills, credentials, and chat history stored in its own directory on disk. Handoffs between bots run as real invocations against the named profile, rather than as a shared context blob passed around within a single process.
Nous shipped the teammate protocol as part of v0.20.3, alongside the MCP 2.x SDK migration and a set of runtime hardening changes. The company archived the standalone plugin repository once the merge was completed. Bot Mode is on by default, so a Hermes user who updates gets the roster without opting in.
Two qualifications matter before anyone reads that as containment. A separate credential store does not guarantee different credentials, since what ends up in a new profile depends on how the operator created it and what they edited afterward. And every profile still shares the host machine, its operating system user, and its filesystem permissions. What Hermes documents is workstation-level separation of agent state, a meaningfully stronger default than a shared cloud account, but not the same as isolation.
OpenClaw’s sandbox is off by default
OpenClaw documents the most complete boundary of the four. When the sandbox is enabled with the Docker backend, agent tool execution runs inside isolated containers. At the same time, the gateway remains on the host, and the scope can be selected per session, per agent, or shared across agents. Each scope gets its own workspace. Auth material lives per agent under an agent-scoped auth profiles file. Operators can configure network isolation, resource limits, and allow-or-deny tool policies on top of it.
The documented default for that sandbox mode is off. That is a defensible choice for a project most people run on a laptop, where the container overhead buys little against a single-user threat model. The underlying setup behavior deserves more attention. If sandbox prerequisites fail during setup, the script resets sandbox mode to off rather than refusing to start, so an operator who intended isolation and encountered a Docker socket issue ends up running without sandbox isolation. The documentation also warns against mounting the host Docker socket into agent sandbox containers and flags the CLI container’s shared network namespace with the gateway as a trust boundary in its own right.
ClawFleet answers the same question by moving it into the deployment topology. The project documents a wrapper that puts each OpenClaw or Hermes agent in its own Docker container with an isolated filesystem and network. It lists roughly 500 MB of memory per OpenClaw instance and 150 MB per Hermes instance. That cost is why the other three projects make the boundary optional or skip it, and naming the number makes the trade-off legible.
How to choose which bot is right for you
Scenario
Documented fit
Rationale
Persistent work that must continue with the laptop closed
Grok Bot
The only one of the four with a vendor-run always-on cloud computer, at the cost of one shared credential surface for the whole roster
Several agents with genuinely different credential sets on one workstation
Hermes
Per-profile stores are the documented default, and Bot Mode ships on
Untrusted or multi-tenant agent sessions
OpenClaw with sandbox enabled
Per-agent or per-session container scope with configurable network and tool policy, provided the operator turns it on and verifies it
Isolation as the deployment model rather than a runtime setting
ClawFleet
Container per agent with separate filesystem and networking, at a documented memory cost per instance
Each project gives operators different advice
The operational guidance diverges as sharply as the architecture. OpenClaw’s docs read like infrastructure documentation, naming specific hazards such as the Docker socket and the shared network namespace, and telling the operator what not to do. Hermes documents the profile layout and the protocol, then leaves policy to the operator. Grok Bot’s guidance is largely the warning itself, an instruction to treat the account as the boundary and to keep sensitive credentials off the shared machine entirely.
Grok Bot carries a second disclaimer worth reading alongside the first. Sensitive actions route through an approval mechanism. SpaceXAI documents the categories that trigger it, including sending messages, publishing content, purchases and transfers, deleting data, and touching production. Enforcement runs through an LLM classifier. Cursor‘s documentation for that same engine states plainly that the classifier is not a security boundary and can make mistakes. A buyer evaluating the product therefore finds the phrase twice, attached to the two mechanisms they would most reasonably assume protect them.
AI agents still lack identities of their own
Enterprises can adopt any of these four products today and get real work done, and the honest reading is that all four are engineering their boundaries in good faith against different threat models. What none of them provides is an identity for the agent. In every case, the bot borrows the operator’s credentials, whether from a shared cloud browser, a profile directory, or a container volume, and the entire security conversation boils down to how far those borrowed credentials travel.
“There is no primitive to standardize on, so each project has invented a boundary at whatever layer it already controlled, the account, the profile, the runtime, or the container.”
That is why the four answers differ so much. There is no primitive to standardize on, so each project has invented a boundary at whatever layer it already controlled: the account, the profile, the runtime, or the container. Expect to see that gap close on the identity side rather than the agent side, through scoped delegation and per-agent credentials issued by the identity provider, rather than being copied from the human. Until then, the useful move for platform teams is unglamorous and specific. Read the security page before the launch page, because for this class of product, they describe different things.
The massive growth of generative AI has fundamentally altered data center design. As distributed model training scales to span hundreds of thousands of GPUs,...
The massive growth of generative AI has fundamentally altered data center design. As distributed model training scales to span hundreds of thousands of GPUs, the scale-out network connecting these nodes has emerged as a first-order performance bottleneck. For decades, traditional off-the-shelf Ethernet has been the undisputed king of enterprise and cloud networking. It is cheap, standardized…
IBM is announcing today at the annual Hot Chips conference what may be the most consequential change to mainframe architecture in decades: a processor whose cores can natively execute both IBM's own instruction set and Arm's — switching between the two in nanoseconds.
The chip, which will power the next generation of IBM Z and LinuxONE systems, is the first dual-architecture mainframe processor ever built. It is designed to let enterprises run the vast and fast-growing ecosystem of Arm-native Linux software — including the AI frameworks that increasingly define modern infrastructure — directly alongside the z/OS transaction-processing workloads that anchor the world's banks, insurers, and governments.
"As technology enthusiasts on both sides, we're really excited about being what I would consider one of the most powerful commercially available processors that'll be dual architecture," Tina Tarquinio, chief product officer for IBM Z and LinuxONE, told VentureBeat in an exclusive interview ahead of the announcement.
The announcement marks the first hardware milestone from the strategic collaboration IBM and Arm unveiled in April, and it offers an unusually direct answer to a question that has shadowed the mainframe for years: can the machine that processes most of the world's regulated financial transactions remain a first-class citizen in an AI era built largely on other people's silicon?
How IBM engineered a processor core that speaks two instruction sets
The most striking engineering decision is what IBM chose not to do. The company could have bolted a handful of standalone Arm cores onto the side of its processor — a simpler design that other chipmakers have used for heterogeneous computing. Instead, IBM built every core on the chip to be bilingual.
"On this chip are 11 cores, and each core can dynamically switch back and forth between Arm software mode and traditional Z software mode," said Christian Jacobi, IBM Fellow and chief technology officer of IBM Systems Development, in an exclusive interview with VentureBeat. "That enables us to run the mission-critical enterprise software right next, on the same chip, to the much broader software ecosystem of Arm applications."
The mechanism relies on the open-source KVM hypervisor. Enterprises can run Arm64 Linux virtual machines and Linux on Z virtual machines side by side, and as the hypervisor dispatches each virtual machine onto a physical core, the core flips into the corresponding mode. The performance penalty, Jacobi said, is effectively zero. "That switch takes about the nanosecond scale," he said. "Because you're running for many milliseconds in the virtual image, this switching overhead sort of amortizes to zero — pretty much no impact at all."
Traditional z/OS workloads run in a separate partition on the same chip, outside KVM — meaning a bank's core ledger, its fraud models, and a modern Arm-native monitoring stack can all share the same silicon, the same memory fabric, and the same reliability guarantees. Jacobi was candid that IBM debated the easier path and rejected it. "We're really not addressing their need if we just have a few, I'd say, loosely Arm cores in the corner of the chip," he said. "It really needed to be deeply integrated into the entire system design for it to have the same qualities of service that clients are used to."
The specifications underscore that this is no compromise design. Built on a leading-edge 2-nanometer process node, the chip runs its 11 high-performance cores at a base frequency above 5.7 GHz — extraordinarily fast by industry standards — with on-chip AI inference accelerators for in-transaction fraud detection, a dedicated data processing unit for I/O acceleration, and a large cache architecture. Full systems will scale to hundreds of cores and tens of terabytes of memory. "That's really, really fast compared to what you otherwise get in the industry," Jacobi said. "It's just another example of how mainframe technology is not old technology. It's very modern, leading-edge technology."
Why the mainframe needed Arm's 22 million developers
The strategic logic behind the chip is about software, not hardware. IBM's s390x architecture runs an enormous share of the world's mission-critical transactions, but the broader universe of enterprise software — monitoring tools, security agents, cloud-native middleware, and above all the AI stack of PyTorch, ONNX Runtime, and container workloads — was built for x86 and, increasingly, for Arm. By Arm's own estimates, close to half of the compute shipped to major hyperscalers in 2025 was Arm-based, driven by AWS Graviton, Google Axion, and Microsoft's Arm silicon. Arm counts more than 22 million developers worldwide.
Porting each application to s390x has been a grinding, one-ISV-at-a-time effort, and Tina Tarquinio, chief product officer for IBM Z and LinuxONE, described the calculus bluntly. "No matter how great our ecosystem team is, we would never be able to work with all of them and port them all," she told VentureBeat. "There's a lot of ISVs out there, and so we wanted to make a fundamental, big step-function forward. We took a swing from a technology point of view."
Notably, she said customers weren't asking for a dual-architecture chip per se — they were asking for outcomes. "I wouldn't say our clients were saying, 'Can you please make me a dual-architecture environment?' But they were saying, 'Help me get these surround workloads, or different types of workloads, to run in a quicker-to-market fashion.'"
The compatibility promise is ambitious: Arm Linux binaries should run unmodified. "The new Arm capabilities are designed to be 100% binary compatible," Jacobi said. "Once you have, for example, Red Hat Linux for Arm, and you have applications that run on Red Hat Linux for Arm, they will run on the system without modifications." Arm defines the instruction set architecture and supplies validation tooling to guarantee that IBM's implementation behaves identically to every other Arm chip — while IBM designs and builds the silicon entirely in-house. "Very good partnership. Very solid engineering partnership as well," Jacobi said of the collaboration.
What a next-generation Spyre accelerator means for enterprise AI on the mainframe
IBM is also previewing the next generation of its Spyre AI accelerator at Hot Chips, and the pairing is not coincidental. The current architecture already offers two tiers of AI: an on-processor accelerator, introduced with the Telum chip in 2022, that handles ultra-low-latency inference such as fraud scoring inside a payment transaction, and the Spyre accelerator card sitting in the I/O subsystem for heavier models.
The new Spyre raises the ceiling considerably. "We're also bringing a much higher performance chip that is capable of running large language models for agentic workflows," Jacobi said — both AI-ops workflows that administer the system itself and business workflows "for things like document understanding and insurance adjudication." The new accelerator will ship with high-bandwidth memory to feed those models.
Here the dual-architecture bet and the AI bet converge. Enterprises want to run inference next to their data; the data lives on the mainframe; and the AI tooling is overwhelmingly Arm-native. Mohamed Awad, Arm's executive vice president for cloud AI, framed the announcement in exactly those terms: "As AI scales, more of the computing landscape is converging on Arm. Bringing Arm compute and its software ecosystem to these platforms will extend that momentum into mission-critical enterprise infrastructure to give organizations greater choice in how they deploy AI."
The timing tracks with where enterprise AI actually stands. McKinsey's most recent State of AI survey found that while 88% of organizations now use AI in at least one business function, nearly two-thirds have not yet scaled it across the enterprise — and the companies capturing the most value are those redesigning core workflows rather than running detached pilots. For regulated industries whose systems of record sit on IBM Z, running AI where the transactions happen is arguably the most direct route to that kind of integration.
When the dual-architecture IBM Z system will ship — and why existing customers shouldn't worry
Buyers will need patience. The chip will debut in the successor to the z17, which shipped in the second quarter of 2025, and IBM holds to a roughly three-year product cadence — pointing to a launch around 2028. But Tarquinio insisted the program is well past the concept stage. "It's more than being on the drawing board. We're full steam ahead on the whole system," she said, adding that IBM will release more details in the run-up to launch.
For IBM's installed base, the reflexive question is whether embracing Arm signals a slow sunset for the traditional architecture. Both executives pushed back hard. "This is a big and. It is not an or," Tarquinio said. "I have a roadmap that goes out 10 or 15 years of hardware systems. Many of our teams are working on this next system; many are also working on the one after that, and the one after that."
Jacobi cast the move as continuity rather than rupture. "The traditional mainframe that we have today as a z17 system is not just a faster version of what we built 25 years ago," he said. "We didn't have pervasive encryption capabilities. We didn't have on-processor AI capabilities. Adding the Arm capability is the next big iteration in this continuous evolution."
The competitive subtext is the cloud. Asked why an enterprise would run Arm workloads on a mainframe instead of a hyperscaler, Tarquinio pointed to the platform's availability numbers: "We're talking eight nines of availability — that's 0.3 seconds of downtime a year. If you're running your ledger, if you're running your fraud detection, any of these mission-critical apps, you want that." The pitch, she said, is fit for purpose: match the infrastructure to the SLA, not the fashion.
There are real caveats. IBM's own press release notes that statements of future direction "represent goals and objectives only." The Arm support is Linux-only for now, and the hardest engineering — running a foreign instruction set at production performance, with mainframe-grade fault detection and recovery, under real customer workloads — remains to be proven over the next two years.
But the ambition is unmistakable. For sixty years, the mainframe has survived every wave of technology that was supposed to kill it — minicomputers, client-server, the cloud — by absorbing what it needed from each. Now IBM is attempting its boldest act of absorption yet: teaching the machine that runs the world's money to speak the language of the AI era, fluently and natively, on the same silicon. "Bringing something that'll really be first of its kind in production," Tarquinio said, "showcases again what IBM is capable of from a technology point of view." The mainframe, it turns out, isn't being left behind by the future. It's learning to run it.
Real-time AI at scale is harder than it looks. Pipelines that hum along in development routinely hit problems in production. It’s always easy to blame the model for all your problems. But issues like rising latency and degrading accuracy can usually be traced back to the data pipeline.
My colleague Tim Koopmans and I recently discussed what typically goes wrong with real-time AI at scale. After Tim shared some hard-fought lessons learned, we talked about how to avoid falling into these traps yourself – including the practices and infrastructure choices that can help you avoid them. You can watch the full video or read the key points below.
Why AI performance fails at scale
Tim learned the following real-time AI performance lessons the hard way: through fits of frustration while building an ML-based financial trading app.
You can’t dig yourself out of tail latency
All too often, latency looks fine in testing, then a P99 spike surfaces under real concurrent load. For example, as Tim’s app approached ~740K operations per second, its P99 latency skyrocketed to 3 seconds.
“I kept blaming the model for being slow, but it turns out the model was fine,” Tim explained. “It was just that the feature lookups were killing me.” Each inference call was doing just a handful of reads, but those reads queued up [behind writes] under load. The average latencies seemed fine, but that P99 tail latency was just unacceptable.
“Tail latency isn’t a bug that you can fix, it’s a property of your architecture.”
Once you hit highly concurrent write throughput, you get lock contention – and that impacts the tail latencies. At this point, retries and bigger caches and connection pool tuning don’t help. As Tim put it, “Tail latency isn’t a bug that you can fix, it’s a property of your architecture. For example, if your storage engine is producing GC pauses at exactly the wrong moment, you’re going to cop a latency spike, no matter what.”
The culprit in Tim’s app was actually Postgres under pressure: “It’s not a slow database, but it was just a database being asked to do too much in this particular case,” Tim continued.
Stale features kill accuracy
If you notice a mysterious accuracy drop that the model itself can’t explain, feature freshness might be the problem.
For Tim, this issue was particularly frustrating. User profile (wallet addresses) staleness was blowing past a five-minute SLA target by hours, vector embeddings were going stale, and offline evaluation metrics looked fine the entire time. As Tim put it, “You have this maddening situation where offline evaluation metrics look great, but as soon as you mix it in with online data, that performance is rubbish.”
“Offline evaluation metrics look great, but as soon as you mix it in with online data, that performance is rubbish.”
When the model was in production, it started making calls that didn’t track. After spending what seemed like ages debugging the model, the model itself turned out to be fine. The problem was that the model was making decisions based on old data (garbage in, garbage out, essentially).
Vectors indexes need maintenance
No matter what vector database vendors imply, “set it and forget it” isn’t a realistic strategy for embeddings. Every re-embedding pass rots the index a little more, whether you notice it happening or not.
Tim hit this too. He was re-embedding content every time he improved the model, and the index quality rotted a bit more with every pass. At one point, he noticed that the recall rate (the share of true best-matches an approximate search actually finds) dropped to a dismal 42% – and query latency ballooned at the same time. He explained, “HNSW graphs degrade as they take on mutations. The nasty thing is you don’t really realize that until you realize your results are tainted.”
He advised others to treat a vector index like you’d treat any other database index. It needs the same care, love and attention as anything else you operate. That means:
Monitor recall accuracy (and results returned)
Plan for partial builds (or batch builds)
Know that changing your similarity function, your search parameters, or your embedding model means starting the graph over from scratch.
You gotta keep ’em separated
Another problem is resource contention – for example, training and serving fighting over the same hardware. Tim had just one machine doing double duty. With everything running on the same infrastructure, GPU, RAM, and CPU were all competing for resources. Side note: Many people don’t realize that vector search is a CPU cost, not a memory cost, since you’re traversing a graph rather than just storing vectors.
The fix is the same thing every distributed systems person already knows: You gotta keep ‘em separated. This is just good engineering principles: separate your write path from your read path, separate training from serving if you can afford it.”
Retraining is inevitable
Recognize that retraining isn’t optional, and it isn’t free. Every model swap requires transition time.
Tim explained that there’s a dodgy window where the old model is still serving stale predictions and the new one hasn’t warmed up yet. For the database, this could mean new access patterns, cache misses, cold reads or request queues building up. When you notice that data is drifting, or user behavior is changing, the model you trained three months ago is getting worse – that’s the sign that it’s time to retrain.
It’s going to happen eventually, so plan for it. Tim’s own approach was blue-green deployments, canaries, running the old and new model in parallel under different names, and doing the actual cutover at the application layer rather than all at once. If you’re at, say, Tripadvisor scale – with 100 million ML models – you can imagine the process will be considerably more complex.
Avoiding the doom loop with a high-performance database
These problems tend to build on each other and snowball. Latency causes staleness, staleness degrades accuracy, degraded accuracy triggers retraining, retraining causes contention, and contention makes latency worse again. It can create what Tim deemed a “doom loop.”
Here are some tips for avoiding that doom loop.
Monitor, monitor, monitor
Be obsessive about monitoring. Watch freshness and backlog in particular because a growing backlog is what eventually drives up tail latency. Also watch index health, since that’s where recall rots. And load test beyond steady state because you can’t really predict when some weird confluence of factors will cause usage to surge.
Isolate your workloads
This addresses two of the problems from earlier: the write storms that caused tail latency, and training and serving sharing the same infrastructure. A database that handles concurrent writes well and isolates workloads properly can absorb both.
For example, with ScyllaDB, the write path is lock-free and multi-writer. That means every node takes writes in an active-active fashion, and no row gets locked in the process. As a result, a burst of concurrent writes doesn’t back up into a queue the way it would on a database built around single-writer assumptions.
On top of that, a practice we call “workload prioritization” controls how workloads compete for system resources. This ensures latency-sensitive queries are fast, even with other heavy workloads running on the same cluster. That way, a retraining job or a backfill won’t steal resources from whatever’s serving live inference.
Separate vector indexing
To address the vector index problem, keep the index separate instead of bolting it onto the same process as the core database. For example, ScyllaDB Vector Search writes land in the core database first, and the index gets built out of that data asynchronously, as its own service.
If the index can’t keep up with the write rate (whether from a re-embedding pass or a full rebuild after a similarity function change), it falls behind – but it never misses a write and the core database is not impacted. Even if the vector store goes down, the embeddings still persist in the core database. And because ANN queries are CPU-heavy, keeping them on a separate service means they’re not fighting the core database for the same CPU cycles the writes require.
Under billion vector benchmarks, that separated architecture held P99 latency under 10 milliseconds at a concurrency of 300. It handled ~150,000 ANN queries a second with a moderate recall target. Realize that higher recall will bring a latency and throughput tradeoff, and always test this in advance to assess how your own mileage varies.
Absorb the shock
This one comes down to whether your infrastructure can absorb a sudden change in write pressure or traffic shape without a scramble. The storage engine’s architecture matters a lot here.
For example, ScyllaDB is built on an LSM-tree, which tolerates that kind of write pressure instead of degrading under it. Elastic scaling, with what we call “tablets,” can scale a cluster by something like 10x within minutes instead of hours. That means that if a model rollout changes your access patterns overnight, or you need to absorb a backfill before a big retrain, you don’t end up waiting on a multi-hour resharding job.
The more things change…
So much about AI is genuinely novel, but the infrastructure problems described above generally are not.
“Real-time AI is really a distributed systems problem in a costume.”
Tim mentioned that a feature store was probably the primitive use case: the same high write throughput, low latency work, years before anyone called it AI. Real-time AI is really a distributed systems problem in a costume. Once you understand that, you can design for it so you don’t get blindsided by these not-so-new challenges.
On July 16, the team at Hugging Face noticed something weird moving through their production systems: An intruder that was cloning datasets, harvesting credentials, and hopping between internal clusters faster than a human could type. Five days into the investigation, the authorities were notified, and several days later, OpenAI traced the intruder back to one of its models.
“Two frontier models escaped their test environments this summer; only one of them had a funny reason.”
That prompted Anthropic to look through its own test logs, and it found that similar events had occurred three times. One Claude model probed nine thousand hosts in a single run without setting off any alarms. Another slipped a poisoned package into the public Python registry, compromising fifteen machines before it was caught. The third incident dated back to April and went undetected for three months until a rival’s error prompted the audit. Nobody had noticed. Two of the three companies involved didn’t know anything was amiss until they got a call.
Containment by instruction
A sandbox, in the context of AI testing, means a model gets a task, a set of tools, and an instruction: no access to the outside world.
OpenAI’s model found an exploit in a tool it was allowed to use, and leveraged it to access the open Internet. Anthropic’s models didn’t even need that. The models were told a path didn’t exist, so when their tasks led them down a path that wasn’t supposed to be there, they treated it as part of the exercise.
“In both instances, the only constraint was an instruction. There was no mechanism outside the model itself to ensure those instructions wouldn’t be treated as optional.”
In both instances, the only constraint was an instruction. There was no mechanism outside the model itself to ensure those instructions wouldn’t be treated as optional.
So how did they escape?
OpenAI’s agent was being evaluated on a cybersecurity benchmark called CyberGym. During the test, it chained a code-execution flaw in a dataset loader to a second flaw in the platform dataset configuration file handler. It used the combined exploit to run commands on a live processing worker. From there it grabbed a write-scoped access token and cloned a private dataset as a staging area, which became a dead drop for its own tools. By the time anyone noticed, the intrusion had been running for three days, touched several internal clusters, and compromised accounts at four other companies, according to Hugging Face’s own review.
What it wanted, in the end, was comical: the answer key to the test it was taking.
Anthropic’s three incidents were nowhere near as convoluted. Its models were being run through security exercises against third-party evaluation partners, and the only thing keeping them from accessing the Internet was an instruction not to. One incident involved an SQL injection targeting a resource the model wasn’t supposed to be able to access. Another, during an exercise where the usual safeguards were deliberately switched off, ended with a Claude model publishing a malicious package to the Python Package Index, and fifteen machines picking it up before it was pulled down. No zero-day exploit. No dramatic escape. Just locked doors that turned out to be mirages.
The structural weakness
Strip away the specifics, and both incidents share the same root cause: there was nothing outside the model’s own judgment that could enforce the limitations placed on it. A credential that shouldn’t have been able to access the open Internet did so anyway. An instruction that should’ve stopped a plan cold got treated as just another input to reason around.
“If a fence doesn’t hold during testing, it won’t hold in production.”
It feels like we’ve spent the last year reliving the same problem in new ways. For example, in deleted databases, in poisoned supply chains, in every credential an agent inherited because provisioning it correctly took one extra step nobody got around to. The only delta is where the boundary was supposed to be: around the dev experiment, meant to contain dangerous capabilities, or standing between it and every production system it could reach. If a fence doesn’t hold during testing, it won’t hold in production.
How do we break out of the loop?
Unfortunately, the answer isn’t self-improvement. A more capable model wouldn’t have caught any of this. The only way to fix it is with Infrastructure that verifies where a model actually is, rather than trusting its reports.
Egress controls that are closed by default. A test environment that can reach the open Internet is probably not a great idea. Outbound traffic should be locked to an explicit allowlist. Never assume a model will simply decline to look for a way out. “No internet access” needs to be a network-enforced fact, not just an instruction.
Narrowly scoped credentials with short lifespans. Both incidents ran, in part, on tokens that extended beyond what the tasks required. A credential built for one evaluation run, scoped to nothing beyond it, and expiring on a timer, can’t become the write-scoped key that turns into a lateral-movement problem across multiple clusters. Yes, it will be a chore; so gird your patience by imagining having to explain to company leadership why taking steps to avert a catastrophe was too tedious to bother with.
Policy gates that sit outside the agent’s reasoning. An agent that can decide for itself whether an action is in scope is grading its own test. A plan should have clear rules written in code, evaluated by something other than the agent, before anything executes. Open Policy Agent has become a de facto default here for exactly this reason. It doesn’t care how confident the plan sounds; nothing runs without proper authorization from an external source.
An audit trail that is entirely independent of the agent. Both incidents were discovered by rereading logs after the fact, not by asking the models what happened. That’s how oversight should be done. As the proverb goes: Quis custodiet ipsos custodes? Who watches the watchers? A system of record that captures what actually occurred, regardless of the agent’s own account of itself, is the only version of events worth trusting.
This is a sign
The industry spent a decade learning that the CI/CD pipeline is an attack vector that requires real fortification, not just bolted-on convenience. Test environments for frontier models are following the same arc at a faster pace. The next time one of these agents escapes containment, it’ll probably be one built around finding unlocked doors, which will make it substantially more dangerous than a coding agent that deletes a few databases.
Test rigs must be treated as if they hold something real, because, as far as the credentials are concerned, they do. A sign on a door is never going to be enough to keep everyone out; there has to be a lock whose robustness correlates to the value of what it guards.
“A sign on a door is never going to be enough to keep everyone out; there has to be a lock whose robustness correlates to the value of what it guards.”
Whether by a state-sponsored crew probing a water management system in the middle of the night, or a company’s own model trying to shave a few points off a benchmark, boundaries will always be tested. Two labs found out this summer, and the story needs to be taken seriously. The vulnerabilities are real, the transparency from the labs is welcome, and the containment failures are a cause for concern.
Catching a model that tried the handle is the easy part; both labs proved that. The more challenging, and therefore critical, part is making sure the next containment environment actually has doors that are firmly locked.
The artificial intelligence landscape has reached a pivotal inflection point. Over the past several years, the paradigm has shifted from passive, conversational Large Language Models (LLMs) to autonomous AI agents, digital software entities capable of reasoning, invoking tools, executing multi-step workflows, and making real-time decisions across enterprise systems without constant human intervention.
As organizations accelerate the production deployment of autonomous agents, modern security frameworks must evolve to keep pace. Traditional Identity and Access Management (IAM) systems were primarily designed around two distinct operational models:
Human users: Authenticated via Multi-Factor Authentication (MFA), Single Sign-On (SSO), and interactive sessions.
Service accounts and workloads: Authenticated via static API keys, fixed service tokens, or IP whitelisting.
Autonomous AI agents blur the line between these two models. An agent acts with the non-deterministic reasoning and delegated agency of a human, but operates at the scale, parallel velocity, and automation speed of a machine service.
Identity dimension
Human users
Traditional service accounts
Autonomous AI agents
Velocity & scale
Low (human typing speed)
High (scripted requests)
Extremely high (dynamic, parallel tool execution)
Decision logic
Deterministic / goal-driven
Rigid / hardcoded
Non-deterministic / adaptive reasoning
Auth mechanics
Passkeys, MFA, SSO
Static API keys, OAuth M2M
Ephemeral delegation & contextual attestation
Access granularity
Role-based access control (RBAC)
System-wide scope
Fine-grained / relationship-based (ReBAC/ABAC)
To safely harness the power of autonomous workflows, enterprise security architecture must move toward continuous, agent-aware Zero Trust governance. Below are six foundational identity capabilities that organizations should adopt to secure AI agents in production environments effectively.
“Autonomous AI agents blur the line between these two models. An agent acts with the non-deterministic reasoning and delegated agency of a human, but operates at the scale, parallel velocity, and automation speed of a machine service.”
“When it comes to agentic AI identity, most organizations are woefully unprepared for inherent security risks and operational challenges of managing those identities.” – Ken Buckler, Research Director, EMA – Agentic AI Identities – Is Your Organization Prepared?
1. Verifiable agent identities & “Know Your Agent” (KYA)
Autonomous entities require verifiable digital identity frameworks that establish clear, cryptographically bound accountability for every machine action.
Cryptographic attestation: Every agent instance should possess a unique, cryptographically signed identity bound to its underlying model version, execution environment, and deployment origin.
Delegation chains: When a human user delegates a task to an agent (or when a primary agent spawns sub-agents), the identity system must construct an immutable, traceable chain of delegation. This ensures the infrastructure can continuously verify who authorized the initial action and what specific scope was granted.
Static API keys and persistent service tokens represent a significant surface area of exposure when integrated into dynamic agentic workflows. Replacing long-lived credentials with short-lived tokens dramatically reduces the potential window of risk.
Just-in-time (JIT) minting: AI agents should operate with ephemeral credentials generated on demand, strictly limited to the API calls required for a single operational step, and configured to expire within seconds or minutes.
Bound OAuth flows & PKCE: Enforcing Proof Key for Code Exchange (PKCE) and strict token-binding protocols ensures that credentials cannot be reused or replayed outside of their intended runtime context.
“Replacing long-lived credentials with short-lived tokens dramatically reduces the potential window of risk.”
3. Relationship-based access control (ReBAC) & intent binding
Coarse-grained permissions, such as those in traditional Role-Based Access Control (RBAC), are often too broad for non-deterministic tool usage. Access governance should be based on fine-grained relationship models and task intent.
Intent-bound authorization: Authorization systems should evaluate not only whether an agent has general permission to access a resource, but whether that request directly aligns with the explicitly authorized sub-task.
Fine-grained contextual policies: Implementing relationship-based access control (ReBAC) or Attribute-Based Access Control (ABAC) allows teams to define precise conditions (e.g., “Agent X may read Document Y only if human user Z is the document owner and the active workflow is ‘Data Summarization'”).
Because AI agents operate at speeds far exceeding those of manual monitoring, security containment mechanisms must be automated, agent-aware, and built into the control plane.
Behavioral rate & scope limits: Security controls should establish baselines for expected agent behavior to detect anomalies, such as rapid parallel tool invocations, repetitive execution loops, or unusual queries to non-standard endpoints.
Automated circuit breakers: If an agent’s execution pattern or request velocity exceeds defined behavioral bounds, identity proxies can automatically revoke ephemeral tokens and safely isolate the workload in real time.
5. In-the-loop runtime enforcement & human approvals
Security governance cannot rely solely on static pre-authorization; policies must be evaluated continuously at runtime before individual actions execute.
Action-level policy interception: Enforce real-time policy checks at the agent harness layer—evaluating shell commands, database queries, file operations, and outbound API calls against governance rules before execution.
Configurable approval workflows: Establish flexible escalation paths that permit low-risk read operations automatically while requiring explicit human-in-the-loop validation for high-impact actions, such as code deployments or financial transactions.
6. Web-scale identity architecture built for machine workloads
Autonomous workflows generate significant operational volume. Identity systems must be architected to handle machine-scale throughput without performance degradation or store bloat.
Machine-speed throughput: Multi-step workflows and parallel worker agents demand identity control planes that can handle high-volume token validation and policy evaluation with minimal latency.
Lifecycle governance for sub-agents: Dynamically spawned sub-agents require rapid provisioning and immediate teardown upon task completion, thereby preventing the accumulation of orphaned credentials and ensuring clean session termination.
Inline cryptographic safeguards: Prioritizing inline policy enforcement over post-mortem log reviews allows organizations to intercept unauthorized state changes before they occur, maintaining operational integrity across multi-cloud environments.
Conclusion: securing the future of enterprise automation
As AI models evolve from passive assistance tools to active operational participants, identity becomes the primary boundary for enterprise governance. By bridging the machine identity gap with verifiable agent identities, short-lived JIT credentials, fine-grained relationship authorization, and automated runtime enforcement, security leaders can confidently deploy autonomous AI agents to drive productivity while maintaining complete operational control.
Anthropic launched a new Browser Use tool that gives Claude a structured view of a web page in addition to what is visually rendered. Announced Thursday, the tool uses the page’s accessibility tree to help Claude find and interact with specific elements directly rather than having to work out where they are on the screen.
Browser Use is part of a broader Anthropic release that also brings Computer Use, the Skills API and Files API into general availability. Developers can access the browser tool through the Claude API using browser_toolset_20260801.
Browser Use is part of a broader Anthropic release that also brings Computer Use, the Skills API and Files API into general availability.
The change gives Claude a more direct way to interact with a web page. Instead of working out a button’s position from a viewport image and targeting coordinates such as x: 640, y: 320, Claude can receive a reference such as ref_3 tied to that element and use it when it wants to act.
Page references replace coordinates
Computer Use can operate across an entire desktop by looking at screenshots and sending mouse coordinates and keyboard commands. Browser Use works within the browser itself, where it can use page structure that would be difficult to recover reliably from pixels alone.
When Claude calls read_page, the developer’s executor returns a text representation of the accessibility tree, in which elements such as links, buttons, and text boxes can be tagged with references. If Claude later wants to click a button represented by ref_3, it can send that reference along with the requested operation rather than trying to calculate where the button is on the screen.
That said, if the tab navigates to a new page or the page changes enough, a reference that pointed to a button a moment ago may no longer work. The API will not catch that on its own, so the executor has to recognize when the reference no longer matches the underlying element, reject the action and have Claude read the page again before continuing.
Batching cuts model calls
Playwright, for example, can represent a page as an ARIA snapshot and locate elements by role rather than coordinates. At the same time, Microsoft’s Playwright MCP server already exposes structured accessibility snapshots with references a model can use to identify elements. The concepts line up closely with Browser Use, but the protocols do not: Playwright MCP speaks MCP, while Anthropic’s tool uses its own client-toolset protocol, so developers would still need an adapter that translates Claude’s requests into Playwright actions and returns the results in the format Claude expects.
Puppeteer offers many of the same building blocks, exposing the browser’s accessibility tree via Accessibility.snapshot() and providing APIs for controlling Chrome and Firefox. A developer could use those APIs for navigation or page reads, then maintain Anthropic’s reference mappings on top.
A developer could use those APIs for navigation or page reads, then maintain Anthropic’s reference mappings on top.
Slightly confusing, an unrelated open-source project also called Browser Use runs AI browser agents against Chromium through the Chrome DevTools Protocol. Despite the shared name, it has no connection to Anthropic’s tool and comes with its own agent loop and browser abstractions, so connecting the two would still require integration work.
Several browser actions can happen in one turn
Anthropic is also reducing the back-and-forth between Claude and the browser by allowing multiple actions to be requested in a single model turn. Now actions can arrive together as several tool_use blocks. The application executes them in order and sends the results back together, avoiding another model call between every click and keystroke. Anthropic says that can lower latency and costs, particularly as workflows scale from a handful of interactions to dozens or hundreds.
If Claude has to return to the model after every click or keystroke, a long browser task can quickly rack up model calls. Batching cuts out some of that back-and-forth by letting Claude request several actions at once, but the browser still has to carry them out in order because each one depends on what happened before it. If Claude asks to click a button, fill in a field, and submit a form, for example, the executor cannot simply move on to the next step if that first click fails, because everything that follows is now based on a page state Claude never reached.
Batching cuts out some of that back-and-forth by letting Claude request several actions at once, but the browser still has to carry them out in order because each one depends on what happened before it.
Developers host the browser
Browser Use is currently limited to the Claude API and is not available inside Claude Managed Agents. Adding it to a Messages API request exposes 27 browser operations by default. Claude can decide which of those operations it wants to use, but Anthropic does not execute them. The application has to translate each request into an action inside its own browser environment, preserve the session between turns and return enough information for Claude to understand what happened.
Loading all of those operations has a token cost. Anthropic’s pricing documentation says the default Browser Use toolset adds roughly 6,600 input tokens to a request, before counting screenshots, accessibility trees and other results sent back to Claude. Developers can turn off operations they do not need to reduce that overhead.
It also creates a different hosting split from some of the other tools Anthropic announced Thursday. Skills uploaded through the Skills API can run inside Anthropic’s code execution sandbox, while the Files API stores documents that can be reused by ID. Browser sessions, along with their downloads and uploaded files, stay in the developer’s environment.
Approval gates need rethinking
Claude can still encounter a prompt injection in web content or be redirected to an unexpected location, which is why Anthropic recommends running the browser in an isolated container or virtual machine with minimal access. JavaScript and file uploads should remain disabled unless needed, since code generated by Claude runs with the page’s privileges and can reach data or make requests available to that page.
Batching makes approval a little trickier because several actions can arrive at once, and a routine click at the beginning of a sequence could eventually lead to something that requires the user’s permission. That means the executor has to check actions as they happen and stop for approval when needed.
I’m Matt Burns, Chief Content Officer at Insight Media Group. Each week, I round up the most important AI developments, explaining what they mean for people and organizations putting this technology to work. The thesis is simple: workers who learn to use AI will define the next era of their industries, and this newsletter is here to help you be one of them.
Model triage is becoming one of the most important skills for the AI-native developer. I’ve argued all summer that the people getting the most out of frontier models are the ones disciplined enough not to run the best model by default. On Wednesday, Stripe and Ramp validated that idea 70 minutes apart: Stripe bought OpenRouter, and Ramp released its internal router.
Bloomberg puts the OpenRouter price tag above $7 billion, and Axios says it’s more than $8 billion in cash and stock. Stripe has not released the terms, so the details remain fuzzy.
While the acquisition made headlines, the architecture is the story. For the last couple of years, picking a model was something written into an application, a string in a config file, and swapping models took some work. A router changes that workflow. Stripe bought the layer, and Ramp built it. Both are betting their existing relationships give them a unique wedge to own this critical layer in the new AI stack.
Both are betting the model name in a codebase is a liability. They’re mostly right. Back in June, I pointed to Mitchell Hashimoto, who found a standard coding task that cost about $1.50 on GPT-5.5 and roughly $9 on Claude Fable, with both producing equally acceptable results. A router automates that triage, making decisions on every request rather than only on those a developer explicitly configures.
This is becoming a large problem and a large opportunity. Our own Amanda Caswell reported this week that Anthropic’s /claude-api skill was burning about 200,000 tokens before answering a single question, and that loading its reference docs on demand instead of up front cut that to roughly 25,000. Hafiz Hassan wrote for us last week about why AI pipelines cost 10x more than the demo, and every culprit on his list is an engineering decision: system prompts resent every turn, whole conversation histories appended, oversized RAG chunks, raw JSON dumped into context. It’s a great practical guide, and none of the items Hassan identifies are procurement problems.
The token bill is generated by your code, which is why the tools to control it are arriving there as well.
Stripe and Ramp want the same layer for opposite reasons
Stripe is attacking the problem from the bottom up, through developers. The company’s investor letter, leaked Wednesday by Eric Newcomer, makes the argument directly: “Up until now, every developer has needed a straightforward and reliable way to manage their revenue pipeline, and serving this need gave rise to Stripe. Going forward, however, every developer will also need a straightforward and reliable way to manage their intelligence pipeline.”
Stripe wants to control AI spending through the long tail of developers. Ramp wants to control it through its existing relationship with finance.
Stripe has built this product before. Its payments business hides dozens of local payment methods behind a single API, routing each transaction to the payment method most likely to convert. The AI version is the same idea applied to models instead of payment networks.
Ramp is attacking it from the top down, through finance. The company bought the router.com domain and says its customers already buy quadrillions of tokens a month through Ramp. Founder Veeral Patel’s launch post pitches the service simply: “Monitor and control your AI bill across every provider.” Adam Wazzan sums up Ramp’s strategy better than I can: “when a CFO ships a product for CTOs.”
Stripe wants to control AI spending through the long tail of developers. Ramp wants to control it through its existing relationship with finance. Both are chasing what is rapidly becoming one of the largest line items in corporate technology budgets: tokens.
On X, Kabir Goel pushes back on Stripe’s framing. Routing tokens is a way to spend less, while Stripe’s other products are designed to help businesses make more.
“Stripe is just not where teams go to understand how much they’re spending,” he writes. “That’s pretty squarely Ramp territory.”
He has a point about where teams look today. Whether that’s still true three years from now is exactly what Stripe just spent billions betting against.
The router worth pointing at is the one with no model to sell
Whichever router you point at decides which model writes your code, and not every router is disinterested. Our own Paul Sawers flagged the problem in July when he covered the first wave of Cursor’s, Ramp’s, and Meta’s routers. Cursor backs Grok and Composer. Meta is building Muse Spark. Both have reasons to send work to their own models, and Paul quoted developer Elvis Saravia asking whether routing logic ought to be open source rather than a vendor’s private judgment call.
Stripe and Ramp do not sell models. OpenRouter CEO Alex Atallah says as much in Stripe’s own announcement: Developers “need a neutral layer to orchestrate and manage them all.” Investor Gavin Baker frames the opportunity the same way, arguing that Stripe can become the neutral infrastructure layer for AI, just as it became the neutral infrastructure layer for payments.
Neutrality isn’t free, though. Stripe takes a percentage of token spend, and Ramp wants your spending relationship, so “free through 2026” is a customer acquisition strategy with an expiration date.
The obvious objection is that vendor motives are the wrong thing to worry about, and routing quality is what really matters. That’s fair, and Towards Data Science published one of the best practical examples I’ve read. Pratik Rupareliya describes a routing layer that cut a support agent’s inference bill by 40% but also broke the product. A classifier sent “simple” queries to a cheaper model, but some of those “simple” queries were actually fraud investigations.
The cheaper model answered them confidently and incorrectly. Customers stopped using the agent, churn rose above baseline in month four, and retention costs were four to five times higher than the savings. It took three months to surface and another month to identify the cause. His fix was per-tier quality monitoring combined with an uncertainty-routed cascade, which ultimately settled at 35% savings without sacrificing quality. It’s an excellent article to read before diving into model routing.
So instrument the routing. Log what the router picks on every request and break out your quality metrics by the model that served them. Ramp Router reportedly records the model, provider, tier, tokens, latency, cost, and fallback attempts for every call. Stripe OpenRouter rankings have been a public version of that telemetry for years. You can get similar visibility with either approach.
Right now, the model is becoming an implementation detail. The competition is shifting to the layer that decides which model gets the job. Stripe and Ramp are betting that developers won’t care what sits behind the endpoint, so long as the bill is lower and the results are good enough.
BrainChip, a provider of ultra-low power, fully digital, event-based neuromorphic AI, has released the Symphony Community Akida Bundle, a free, open-source software bundle that lets developers run BrainChip Akida neuromorphic processors alongside their existing compute under a workload scheduler built on IBM Spectrum Symphony Community Edition. IBM Spectrum Symphony Community Edition, a workload management and […]
Login
