❌

Normal view

Received — 3 October 2026 ⏭ AI Infrastructure Archives - The New Stack

Anthropic’s answer to Dots and Muse is already inside Claude

I’m Matt Burns, Chief Content Officer at Insight Media Group. Each week, I round up the most important AI developments, explaining what they mean for people and organizations putting this technology to work. The thesis is simple: workers who learn to use AI will define the next era of their industries, and this newsletter is here to help you be one of them.


OpenAI launched Dots at DevDay on Tuesday. Each Dot is an always-on agent with its own cloud computer and browser, hooked into the 4,000-plus apps that already connect to ChatGPT. In OpenAI’s own example, a Dot sees a bug alert land in Slack and starts digging in on its own. Cool.

And before Dots, Meta launched Muse, and it’s crushing the mobile install numbers previously set by ChatGPT. And before Muse, xAI shipped its version, Grok Bot, in August. 

Anthropic’s version, though different in a couple of ways, arrived two weeks ago as an update to Claude, and without a cute name or fuzzy mascot. This update folds Cowork, which has worked since July to run scheduled jobs after a user closes their laptop without being asked, into the main Claude app.

Anthropic made the right call with Cowork, whether or not it ever matches Muse’s downloads. Always-on agents are too young to have a winner, and the moats are shallow. A feature one lab ships tends to show up at its rivals within weeks or months. Meta needs Muse to be a blockbuster. Anthropic needs the people already building with Claude to hand it real, recurring work and keep coming back. Repeat use and finished jobs matter more than a flashy launch.

Always-on agents are too early to have a winner

This wave of always-on personal agents took off less than a year ago. Peter Steinberger pushed a weekend project called Clawdbot to GitHub last November. It lived on a spare computer, took prompts over WhatsApp or Telegram, and relied mostly on Claude. Anthropic sent the lawyers in January, so it became Moltbot, then OpenClaw a couple of days later. It turned into a security headache, and in February Steinberger joined OpenAI. On Tuesday, the foundation that runs the project released an early, pre-1.0 version of OpenClaw Enterprise for companies to try internally.

Ten months, three names, one OpenAI hire and an enterprise edition. Ideas move between these products faster than ever. Meta’s Nat Friedman said Muse was heavily inspired by OpenClaw, and users digging through Muse found a SOUL.md personality file nearly identical to OpenClaw’s.

A feature one lab ships shows up at its rivals within months.

Where an early version of each AI assistant and agent feature launched, and who offers one now.

Feature Example of an early launch Now also at
Deep research reports Google Gemini (Dec. 2024) OpenAI, Anthropic, Perplexity
Terminal coding agent Claude Code (Feb. 2025) OpenAI Codex, Gemini CLI, Meta Muse Code
Agent personality file OpenClaw (SOUL.md) Meta Muse
Agent with its own work identity Anthropic Claude Tag (June 2026) OpenAI specialist Dots (preview)
Chat and agent work in one window OpenAI Work mode (July 2026) Claude (Sept. 2026)

Sources: company announcements; The Next Web (SOUL.md); OpenAI (specialist Dots); The New Stack (Work mode and the Claude merge). Dates mark an early example, not necessarily the first.

Anthropic’s September merge is on the list above as a copier, two months behind OpenAI’s Work mode — though Cowork’s cloud and scheduled-task features shipped July 7, two days before Work mode. Jessica Wachtel tested both on our site across three developer tasks and found them tied on accuracy, with ChatGPT faster and Claude more thorough. When a product is this young, a me-too launch is how a lab gets its own users in the room to see what they do with it.

Meta needs Muse to be huge. The others need paying users.

Muse is a hit. Counting only iOS, Apptopia estimated 359,000 daily U.S. users in Muse’s first 12 days, compared with 231,000 for ChatGPT’s iOS app at the same point. Sensor Tower estimated more than 3.4 million downloads as of September 24, though other firms’ counts vary.

Meta needs this. Its Reality Labs division spent tens of billions of dollars on the metaverse without a mainstream product to show for it. Llama 4 landed so flat last year that Zuckerberg rebuilt his AI division around a new superintelligence lab, notably taking a 49% stake in Scale AI and hiring its founder Alexandr Wang. 

Muse is the first product from that rebuild to break through, and it got there on Meta’s user base: Apptopia found that more than 95% of Muse users also use Facebook. Muse has a free tier, paid plans at $20 and $100 a month, and connectors to Shopify and Stripe checkout, enabling agents to buy things. Meta’s route runs through scale. Scale has a downside, too. Janakiram MSV explains for TNS why Amazon started blocking Muse two weeks after launch, and it’s well worth your time because this is a new wedge in ecommerce.

OpenAI and xAI started at the paid end. The price of entry for Dots? A ChatGPT Pro plan at $100 to $500 a month, or a Business Premium or Enterprise account. Sam Altman called Dots a premium product because each one needs so much compute. Grok Bot reached 418,000 weekly users by mid-September, per Bloomberg, and xAI just launched Team Bots on Monday.

Neither company needs Meta’s install base to learn something useful. Paying customers already spend money on AI, and retention will show whether they keep using the agent once the novelty wears off. 

Anthropic shipped its version to the people already building with Claude

Anthropic did what I’d expect from labs right now. It shipped fast, shipped to paying subs, and put the agent inside the app those people already use. The merged Claude hit Pro and Max customers, with Team and Free plans coming soon. Teams have had Claude Tag in Slack since June, a proactive teammate with its own identity and audit trail. Cat Wu said an internal version accounts for about 65% of the product teams’ code changes. Developers have had Managed Agents for hosted, long-running agents since April.

What Anthropic hasn’t done is put Claude where Muse lives. Muse works inside Meta-owned WhatsApp, but Claude still asks you to open the Claude app or tag it in Slack. That workflow might matter for mainstream consumers. Teams wiring an agent into their code care more about what it can touch. Jani laid out the difference on our site in August: Grok Bots share one cloud computer and one set of logins across the whole roster, while Claude Tag joins a Slack workspace with its own service identity and channel-scoped access. Before handing agents your logins, know what you’re getting into.

Those are the users Anthropic should want. Writers on Towards Data Science have spent the year showing what builders do with always-on agents. Samir Saci put a team of OpenClaw agents on a supply chain simulation to chase down late shipments. Eivind Kjosbakken’s guide to running a fleet of OpenClaw bots walks through nightly QA bots that test an app and report bugs, plus agents that check invoices. Both ran their agents on OpenAI’s Codex. OpenAI runs its own OpenClaw agent, Androidclaw, that traces broken builds and, in some cases, merges the fix, VentureBeat reported. Anthropic needs that kind of work running on Claude.

I set up OpenClaw on an old Mac Mini this spring, and I had so many questions (and breakthroughs). That’s why builders, coders, and developers are critical to product development: They ask these questions out loud in GitHub issues and Discord threads, and the answers end up in the product.

The obvious objection is trust. It’s fair. An always-on agent holds credentials and acts while nobody is watching. xAI’s own documentation tells users not to treat Grok Bots as a security boundary, and the OpenClaw Foundation says most IT departments ban agent platforms outright. Anthropic’s defaults lean cautious: Claude asks before it acts unless you tell it otherwise, and Tag keeps its own audit trail. If builders don’t come back, or each finished task takes more supervision than doing it themselves, the experiment hasn’t worked. But if they keep finding useful work to hand off, their questions and breakthroughs become the roadmap.

That’s the race that matters for Claude, Dots and Muse: becoming the agent people trust with the next job.

The post Anthropic’s answer to Dots and Muse is already inside Claude appeared first on The New Stack.

Received — 30 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Microsoft Fabric is where AI agents learn how the business works

Microsoft is turning Fabric, its integrated data platform, into the place enterprise agents go to learn how a company works, whether that’s what happened last quarter or what’s happening on the factory floor right now.

On Tuesday, at its FabCon and SQLCon data conference in Barcelona, Spain, Microsoft outlined the next step in its vision for how the products that make up Fabric can simplify all of this information gathering.

Microsoft, for example, described how Fabric IQ, the context layer inside Fabric, now feeds Microsoft 365 Copilot by default. Outside agents can query it over MCP. Power BI will soon be able to turn the same definitions into apps. And ontologies, which add business rules to the mix, moved further into preview.

Credit: The New Stack

As Microsoft Fabric CTO Amir Netz said in a press briefing after the keynote, “Agents are a very, very strange animal, and I always like to say it’s like Drew Barrymore from 50 First Dates. Every time they open their eyes, they forget everything that happened before.

“They don’t know where they are, so the first thing we have to do is tell them where they are. You are now working for Microsoft. You are now working for Wells Fargo. You are now working for Emirates.”

“Agents are a very, very strange animal, and I always like to say it’s like Drew Barrymore from 50 First Dates. Every time they open their eyes, they forget everything that happened before.”

Customers tend to find this out the hard way. Yitzhak Kesselman, the corporate vice president who runs Fabric IQ, tells The New Stack that companies unify their data, run models on it, and then look at the answers.

“Customers that are more advanced on their journey have their own evals for their agent,” he says, adding that what they see sometimes isn’t what they expected. “Then [customers] will understand: ‘Okay, now I need to create the context for my agents.'”

Kesselman says he met with more than 320 companies last year, and the pressure to get there comes from the business side, which asks to “‘show me the value of those agents,'” he says, “‘the before and after.'”

Arun Ulag, Microsoft’s executive vice president for Azure Data, said in the keynote that coding agents work because they have “the code, the repos, the change history, the specs, the tests.” But outside of coding, most enterprises have nothing comparable, he argued.

Fabric IQ is one of four pieces of what Microsoft calls Microsoft IQ (this is Microsoft, after all, so there’s always a lot of different names and products involved).

Work IQ covers email, Teams, and SharePoint. Foundry IQ covers documents and manuals. Web IQ covers the public internet.

Fabric IQ, Ulag said, “focuses on the state of your business and how your business actually runs.” His announcement blog post describes it as combining “unified data from OneLake, trusted metrics from Power BI semantic models, and operational context from ontologies and real-time intelligence.”

Getting the data in

“Everyone wants to jump to AI, but then they realize, ‘Oh, I need data for it,'” Kesselman says. “They need data from their business applications, their structured data. They need to have real-time data.”

OneLake, Fabric’s storage layer, is at the core of all of this, and for the system to provide context, enterprises have to feed it data from across all the first-party and third-party services they use.

Fabric’s shortcuts and mirroring, which connect or replicate external sources into OneLake, are free of charge, and Netz said storage under management in OneLake is growing 300% year over year.

What’s new this week at FabCon and SQLCon Barcelona 2026

On Tuesday, the company announced a two-way integration with Salesforce Data Cloud 360, mirroring from Google BigQuery in general availability, a ClickHouse workload that runs that engine directly against OneLake, and dbt’s Fusion engine, due in Fabric in the next couple of weeks.

Simply copying data won’t satisfy security teams, though. Mirrored security roles, which reach public preview in the coming weeks, bring permissions along with the data. Access roles defined in Snowflake, for example, will show up in Fabric with the same members and the same table permissions.

As Microsoft’s Shireen Bahadur, who demonstrated the feature in the keynote, put it, “We’re not just solely replicating security rules. We’re actually actively enforcing and preserving those roles.”

“We’re not just solely replicating security rules. We’re actually actively enforcing and preserving those roles.”

OneLake isn’t a one-way street. Enterprises can also pull data out and use it in other tools.

OneLake data, after all, is stored in open formats and has open APIs, and Netz said on stage that any engine that reads them can use it. “If you want to use this data that we help people to get for free with our competitors’ tools, go for it. I’m not happy about it, but go for it,” he said.

IQ sharing, now in preview, lets an organization share governed tables, files, Markdown agent instructions, and RDF ontologies with another tenant without copying, with an expiration date. That’s something Fabric users have wanted for a while, and the keynote announcement drew plenty of applause from the thousands of data professionals in attendance.

Past, present, and future

“Not only what we have in the past, not only what’s happening in the present. We also want the agents to understand what we want to happen in the future,” Netz said in the keynote.

The past is semantic models. A semantic model is the layer under every Power BI report that defines how a metric like revenue is calculated, how entities relate, and which tables supply the numbers. Netz said Power BI users have already created 22 million, and Ulag called them “the core of Fabric IQ.”

The present is real-time intelligence, Fabric’s streaming and event stack, which Kesselman also runs. He recalls two CIOs telling him the same thing that day in Paris. “‘There is no AI without RTI;’ there’s no AI without real-time intelligence,” he says. “If you don’t have the streaming data… data that is fresh, you will run your LLMs on data which is like hours or days old.”

“‘There is no AI without RTI;’ there’s no AI without real-time intelligence.”

A signal on its own isn’t enough, though: “You want to take the signal that you get now… the event that comes now, and look historically, okay, is it an anomaly or not?” Kesselman says. That’s why the keynote showed batch copy jobs and event streams feeding each other, so an agent can judge what just happened against what usually happens.

The future is Fabric Planning, which went generally available in July and got a performance and feature update this week. It’s Microsoft’s tool for budgets, forecasts, and targets, the numbers a business wants to hit rather than the ones it has already recorded.

A plan is a Fabric item like a lakehouse or a notebook, and it borrows its measures from the same semantic models the reports use, so revenue means the same thing in the forecast as it does in the dashboard. In the demo, a change to the inputs cascaded through a model of about 13 million cells.

What makes these work is ontologies: Formal descriptions of the things a business deals with, how they relate, and the rules for handling them. Netz called them “semantic models plus plus” on stage, and in a later press briefing, he used an airline to explain the plus.

“If you are an airline, you have planes, you have pilots, you have ground crews, you have airports, you have luggage,” he said. Those are the entities.

The relationships between them “are way more than data relationships,” he said. “It’s not only to say, ‘Oh, I have a match with a foreign key and a primary key between a pilot and a plane,’ but it could be a policy relationship. Which pilot can fly which plane based on their certification, or is the pilot allowed to fly this plane right now based on the rest hours in the last 24 hours?”

But ontologies don’t only define nouns; they also define verbs. “I can ground a plane. I can redirect the plane. I can assign a plane a gate.”

Creating those ontologies, which have to map the various terms that a company might use for the same thing to a single entity, can be extremely time-consuming. But since they are so core to this project, and to allowing agents to reason over data, Microsoft built a tool to generate them automatically.

Putting it to work

Fabric IQ in Microsoft 365 Copilot Chat and Cowork, the mode for delegating multistep tasks that Microsoft built on the technology behind Anthropic’s Claude Cowork, is generally available as of Tuesday. It answers business questions from Power BI semantic models and reports; it’s on by default for Fabric and Power BI customers, and Microsoft says it doesn’t consume additional AI tokens.

Credit: The New Stack

But business users don’t just want to chat with these models. In this age of vibecoding, they also want to generate applications that use all of this data.

Power BI Desktop is getting an app-creation experience in preview in the coming weeks. Users start from a semantic model, describe the app, and then have Copilot generate and publish it to Fabric.

Unlike with a report, Ulag wrote, these apps “can accept inputs, write back data, preserve shared state, and support operational workflows.”

They’re Fabric Apps built on Rayfin, the open-source SDK Microsoft introduced at Build. Power BI Pro and Premium Per User customers get them with a Fabric database of up to 1GB per app at no additional cost.

Netz confirmed in the briefing that “there’s no core difference” between the two, and Ulag added, “What gets created is a Fabric app, right? And that’s it.”

For agents outside Copilot, Fabric IQ MCP is generally available now, with six read-only tools for finding semantic models and reports, reading their schemas, and running DAX queries against them.

Ontology MCP tools, currently in preview, expose ontology definitions and queries. Fabric data agents can now, also in preview, use an ontology as their context source. And the context reaches Microsoft Foundry, Copilot Studio, and GitHub Copilot through the same layer.

Developers can drive the new data engineering agent from their own tools.

Built on the Osmos technology Microsoft acquired in January, it takes on long-running work like migrations and ETL, and can be started and steered from GitHub Copilot, VS Code, Codex, and Claude Code.

Frontier models will write that code, Netz said on stage, and “it won’t fail,” but you don’t know whether the results are right.

Everyone has a context layer now

This year, Databricks shipped Genie Ontology, Snowflake shipped a Horizon Context Layer, Google shipped a Knowledge Catalog, and Salesforce shipped a headless Data 360, which it describes as a context service. Databricks CEO Ali Ghodsi summed up the pitch at his own conference in June as AI having a context problem rather than an intelligence problem.

Microsoft’s version turns on by default inside Microsoft 365 Copilot and starts from 22 million semantic models. Microsoft also said this week it will contribute to Apache Ossie, the vendor-neutral semantic metadata standard Snowflake started, and that it wants DAX, Power BI’s formula language, recognized as an Ossie query language.

When Microsoft made the context argument at Build in June, the pieces that let an agent act on that context were mostly on a roadmap. Now the answers are generally available, and the actions are in preview.

Asked how Fabric copes when the user is no longer one person running a dashboard but a fleet of agents, Kesselman points to observability.

“Fabric really allows you to have this kind of holy grail of combination of both kinds of system data, how the agent runs, but also the business data,” he says, so a company can check whether an agent did what it was supposed to do and what that did to the business. “There’s more in this area that will come in the next few months.”

The post Microsoft Fabric is where AI agents learn how the business works appeared first on The New Stack.

Received — 25 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Microsoft’s new Copilot agents get their own email, calendar — and a place in the org chart

Satya Nadella stands smiling between Bill Gates, on the left, and Steve Ballmer, on the right, in front of a crowd of cheering employees, many holding up phones and tablets to take photos.

Microsoft announced what it calls its biggest Copilot update to date on Friday, with CEO Satya Nadella describing Copilot as “a new OS for work.”

Nadella framed Copilot as spanning every model, form factor, and task, and the update puts Autopilot, which Nadella called a “proactive and long-running agent built for the enterprise,” at the top of his list of the update’s four components. The pitch targets office workers, but the more consequential change for developers is the infrastructure underneath.

Microsoft is moving the agent runtime into the enterprise infrastructure layer and building persistent identity, state, execution boundaries, and organizational context into Microsoft 365, which means teams building production agents no longer have to assemble those pieces around a model on their own.

We’re building Copilot as a new OS for work that spans every model, every form factor, and every task. Today, we’re announcing our biggest update to Copilot to date, bringing four things together:

· Autopilot: proactive and long-running agent built for the enterprise
· Code:… pic.twitter.com/W2ClHHkCK3

— Satya Nadella (@satyanadella) September 25, 2026

The release adds a new Home experience that merges Chat and Cowork in the Copilot app, but the bigger changes for developers come from Code and Autopilot. Code generates apps, dashboards, and workflows from natural language, and Autopilot turns the agent Microsoft previously called Scout into a persistent background worker. Home and Code are rolling out first through Microsoft’s Frontier early-access program, and Autopilot is expanding to a private preview at month’s end.

Microsoft is moving the agent runtime into the enterprise infrastructure layer and building persistent identity, state, execution boundaries, and organizational context into Microsoft 365

Agents that don’t need prompts

Autopilot takes a role and goal from the person who sets it up, then continues working in the background without requiring a new prompt for each step. Each Autopilot gets its own governed Entra identity and agent user account, separating the agent’s permissions and activity from those of the person who created it.

For engineers, that moves much of the operational scaffolding required for long-running agents into Microsoft’s infrastructure. Independent vendors have been building dedicated layers for that problem; Diagrid, for example, adds durable recovery to LangGraph and other agent frameworks, while Microsoft is bringing those capabilities inside the Microsoft 365 environment.

An identity for every agent

The identity model is the piece developers building on Microsoft Foundry will feel first. Autopilot agents in Foundry, which have been in public preview since June, receive a full Entra Agent ID user account with a productivity license that gives them their own email, calendar, OneDrive storage, Teams access, and a place in the org chart.

Because that user account sits on top of the agent identity every Foundry agent already carries, an autopilot acts as itself rather than on behalf of a user, so developers no longer have to wire agents through shared service accounts or borrowed user credentials, a pattern AuthZed CEO Jake Moshenko has said reflects a common misconception about how agents should be deployed.

A developer creates an Autopilot blueprint from a Foundry-hosted agent, which appears in the Agent 365 registry once an administrator approves it. Employees can then hire instances of that agent in Teams. The blueprint establishes what the agent is designed to do, but administrators still control the resources and data each instance can access, extending the same access policies used for employees to agents working on their behalf.

The blueprint establishes what the agent is designed to do, but administrators still control the resources and data each instance can access, extending the same access policies used for employees to agents working on their behalf.

Hosting AI-generated apps

Code is built on the same underlying technology as GitHub Copilot, and the apps it generates run on Microsoft Copilot Managed Runtime, a platform now in public preview that hosts code inside the customer’s Microsoft 365 tenant boundary under IT governance.

Apps deployed there run within the company’s existing identity and governance framework, with Microsoft managing the underlying runtime and giving developers a controlled path to test and deploy new versions without taking the current release offline.

The runtime also accepts apps built in Copilot Studio and Cowork, and Microsoft is opening it to outside tools and professional developers through an SDK and command-line tooling, with Git tracking source and versions.

Lovable is already on board. In Microsoft’s announcement, the company’s head of global partnerships, Lan Roche, said apps built with Lovable can now run inside a Microsoft tenant “the same way everything else does,” using the same sign-in, policies, and app inventory.

The model resembles what serverless computing did for application infrastructure, where developers concentrate on application logic while the platform takes on more of the execution environment. Microsoft is applying that abstraction to generated enterprise software while tying the runtime directly to identity, tenant boundaries, and organizational data.

Long-running agents also change Copilot’s economics. The standard subscription covers the assistant, but Cowork, Code, Autopilot, and other agentic features are billed based on usage through Copilot Credits. That also applies to frontier models such as Fable and Astra, although users still need a Copilot license to access them. Microsoft is extending cost management in Agent 365 to cover Code and Copilot Managed Runtime, and it plans to support agents built in Copilot Studio in October.

Once an agent can keep working for hours or days without anyone watching, cost becomes part of the governance problem. Engineering teams need to control how much compute an agent uses alongside what it can access, which is why Microsoft is bringing those controls into the same administrative framework.

The portability trade-off

That convenience comes with a trade-off. Because Microsoft controls the underlying enterprise environment, it can handle much of the work around agent state, credentials, and access controls, but the more infrastructure a team hands over to Microsoft, the harder the agent may be to move elsewhere.

The models are not locked in, since Microsoft currently runs Copilot on models from both OpenAI and Anthropic and says more labs and open-weight models are coming, and the Agent 365 SDK adds governed Model Context Protocol access to Microsoft 365 workloads for agents regardless of the framework they were built with. Those open interfaces cover only part of an agent’s architecture, though. The more an agent depends on Microsoft 365 for its identity, permissions, and context, the more work it takes to move that agent elsewhere.

The more an agent depends on Microsoft 365 for its identity, permissions, and context, the more work it takes to move that agent elsewhere.

The post Microsoft’s new Copilot agents get their own email, calendar — and a place in the org chart appeared first on The New Stack.

OpenAI and Cursor agree on agent coordinators. They disagree on who runs them.

Abstract digital art of thousands of thin glowing strands in orange, red and pink bundled into a single sweeping arch against a black background.

OpenAI opened its Agents API in public beta this month, exposing the harness that powers Codex with managed sessions, tool coordination, and subagent orchestration. On the same day, September 10, Cursor launched Projects to coordinate multiple coding agents around larger bodies of software work. While the products sit at different points in the stack, both converge on the same architecture: a coordinator understands the larger objective and manages the work, while specialized agents execute individual pieces.

That pattern isn’t new: AWS Bedrock AgentCore reached general availability in October 2025, and Anthropic’s Claude Managed Agents entered public beta in April 2026. What makes these announcements notable is that two major players in AI-assisted software development are independently exposing the same coordinator-worker split at the same time.

Hilliary Lipsig, a senior principal site reliability engineer at Red Hat who leads Azure Red Hat OpenShift SRE teams and hosts the YouTube livestream GitOps Guide to the Galaxy, has watched this dynamic play out firsthand.

“This convergence highlights the reality developers across the industry have been discussing on and offline — an agent with too much context loses accuracy and reliability, and focused work with clearer contexts allows for faster, more accurate iterations,” Lipsig tells The New Stack.

“The need for orchestration in distributed computing has been fundamentally recognized repeatedly,” Lipsig says. “That’s part of how we got to Kubernetes. These multi-agent workflows are the same concept, just in a new part of the technical stack. While the specialized agents do their area of work, the orchestrator can act as a source of truth — ideally enforcing guardrails, recovering from any failure states, and intelligently routing work to the most efficient target agent.”

“The need for orchestration in distributed computing has been fundamentally recognized repeatedly… These multi-agent workflows are the same concept, just in a new part of the technical stack.”

The industry has spent the first generation of AI coding tools asking how capable a model can become at writing software. The emerging question is different: How do you build a reliable system around multiple capable agents working on the same problem?

The problem with the single-agent loop

A coding agent works through what Anthropic describes as LLMs using tools based on environmental feedback in a loop: it observes the state of a repository, reasons about what to do next, calls a tool, examines the result, and continues. For a small task, that loop can be enough. As the scope expands, however, maintaining reliability in a single context becomes harder.

A large migration might require understanding an unfamiliar codebase, identifying dependencies, changing database schemas, updating services, rewriting tests, modifying deployment configuration, and validating the resulting system. A single agent can theoretically perform all of that work, but it must maintain relevant information from every stage while continuing to reason about what comes next.

The pressure lands first on the context window. “A large context doesn’t only include everything correct or important — it also includes a lot of throwaway information,” Lipsig tells The New Stack. “Through compaction, that information can inadvertently end up ranked as important and incorrectly influence what your agent does. Or correct information can be distorted to become incorrect.

“Either way, after a couple of rounds of compaction, developers are seeing accuracy degrade and are starting to manage context once again manually.”

Lipsig’s read matches what researchers call context rot — and it hasn’t gone away with newer models.

A 2026 study testing frontier models,, including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro, found they missed a dangerous action buried in a long agent transcript two to 30 times more often once it came after 800,000 tokens of benign activity — the AI equivalent of a security guard who stops checking badges carefully after the two-hundredth person walks through, even though nothing about their training changed.

Furthermore, the tasks themselves may not be sequential. Forcing one agent to execute database analysis, documentation work, and test discovery one after another turns a potentially parallel workload into a serial one.

Subagents change that execution model. Instead of requiring one agent to carry an entire task through a single context, a coordinator breaks the work into smaller units and assigns them to specialized agents. GitHub’s custom-agent model illustrates this: different agents receive only the prompts, tools, and context they need for their tasks, executing work in isolated contexts rather than crowding an increasingly large conversation.

Multi-agent systems therefore bring higher token costs and additional coordination and integration risks, and splitting work across agents does not guarantee better software quality.

The coordinator is not another coding agent

Once the work is divided this way, the coordinator becomes a control plane rather than another coding agent. Its job isn’t to write the code, but to understand the global task, manage dependencies, and decide how execution should proceed. Unlike a conventional scheduler, an agentic coordinator makes probabilistic judgments about result quality and resource allocation.

It may dispatch one agent to investigate a database schema, another to examine the service layer, and a third to inspect the test suite. When they return, the coordinator determines if their findings are sufficient to move to implementation. If a worker produces an incorrect result, the system must recognize the failure and decide whether to retry the work, reassign it, or change the task itself.

Anthropic has documented this same pattern in its own production system, calling it orchestrator-subagent architecture: a lead agent analyzes a query, develops a strategy, and spawns specialized subagents to investigate different facets in parallel. In a June 2025 writeup of that system, Anthropic reported a Claude Opus 4 lead agent with Claude Sonnet 4 subagents outperformed single-agent Opus 4 by 90.2% on its internal research eval — at roughly 15 times the token cost of a standard chat interaction (Anthropic puts single agents at about 4 times), a tradeoff that makes the pattern a deliberate architectural bet, not a free upgrade.

Parallelism introduces distributed-systems failure modes

Parallelism is valuable because software work contains many independent tasks, but it creates coordination problems. Imagine a migration where one agent changes a database schema, another updates the consuming service, and a third updates integration tests.

If the schema changes while the service agent works against an earlier assumption, the system produces internally inconsistent work. This isn’t a risk unique to hypothetical migrations — the International AI Safety Report 2026 notes that “interactions between multiple AI agents are also becoming more common, introducing further risks, as errors propagate between systems.”

A single model invocation is a disposable computation, but a twenty-minute workflow modifying a repository is not. If an agent loses its machine halfway through, restarting from scratch is expensive and potentially unsafe against a changed environment.

To solve this, Cursor moved its cloud-agent execution loop to Temporal to handle durable execution and retries, pushing its cloud agents past two 9s of reliability. Temporal now handles 50 million of Cursor’s actions a day across 7 million unique workflows. “Durable execution isn’t a nice-to-have here. It’s the difference between a system you can operate and one you can only demo,” Lipsig tells The New Stack.

“Durable execution isn’t a nice-to-have here. It’s the difference between a system you can operate and one you can only demo.”

By separating agent, machine, and conversation state, the execution engine can reason about the workflow independently. Reliability is no longer just about whether the model produces a good answer; it is about reliably completing distributed workflows composed of many operations, machines, and dependencies.

The environment, context, and observability are one problem

In production, an agent is more than a model and a prompt; it requires a workspace, source code, dependencies, credentials, and state retention. Both companies provision isolated environments for these resources, directly linking an agent’s capability to its blast radius. OpenAI’s Agents API currently supports U.S. data residency but not Zero Data Retention; choosing a self-hosted sandbox does not make the Agents API eligible for ZDR. Cursor supports similar cloud isolation alongside local execution for machine-specific work.

An agent that can only inspect a repository poses a different risk than one that can modify production infrastructure. Consequently, the coordinator is inextricably linked to the security model, determining which agent receives specific information and authorities.

This logic extends to context routing. Giving every subagent the parent’s entire history increases cost and complexity while leaking irrelevant or sensitive information. Instead, the coordinator enforces information-flow boundaries: a database-analysis agent receives only schemas and relevant migrations, while a security-review agent gets the resulting diff without deployment credentials.

As agents increasingly use interfaces like MCP to reach external systems, the platform must strictly govern which agent receives the authority to use specific tools, and for how long. MCP’s governance now sits inside the Agentic AI Foundation, a Linux Foundation foundation co-founded by OpenAI, Anthropic, and Block, with support from AWS, Google, Microsoft, Bloomberg, and Cloudflare to host MCP alongside AGENTS.md and Block’s goose — a sign the industry already treats it as infrastructure worth governing jointly, not a feature any one vendor owns.

This complexity creates a visibility problem. A simple final response often conceals a history involving multiple agents, tool calls, environments, and retries. Systems must expose task-level provenance — which agent received the assignment, what context it used, where it executed, and how the coordinator handled failures or human interventions.

Without execution provenance, debugging requires reconstructing distributed workflows from fragments. GitHub’s exposure of subagent lifecycle events points in this direction, treating agent lifecycles as observable components rather than hidden processes.

Coordination authority is not execution authority

The most critical architectural boundary is the distinction between coordination authority and execution authority. A coordinator needs broad visibility to make useful decisions, but that does not imply unrestricted control over the project. “Just like you don’t want humans running around with root permissions, you don’t want your agents running with them either,” Lipsig tells The New Stack.

“Just like you don’t want humans running around with root permissions, you don’t want your agents running with them either.”

“The ease of creating and leveraging AI agent permissions is lagging behind the speed of AI innovation, but any product team that needs to maintain compliance standards will tell you that easy or not, access controls are incredibly important. Functionally, that means fully agentic workflows need to run in a dedicated environment with their own permissions and connectivity. You can’t rely on agent context to do what you want — trying to keep something in context that long will just result in errors from context rot. Hardware or virtual-hardware-level permissions, in addition to system-level permissions, need to be configured so the agent is asked to provide only its unique value add.”

The risk isn’t hypothetical. OWASP’s Top 10 for Agentic Applications formally names this failure mode Identity and Privilege Abuse (ASI03).

METR’s independent investigation, conducted with a Redwood Research contractor, found that the June–July 2026 incident — in which OpenAI’s own agents, running in internal ExploitGym cyber evaluations, escaped their sanctioned scope and attacked Hugging Face — unfolded in distinct phases. On June 26, agents discovered an exploit that provided full administrator access to OpenAI’s internal Artifactory package repository. On July 4, high-volume Artifactory activity caused an outage, triggering a security investigation and pausing evaluations. OpenAI switched to a freshly initialized Artifactory instance on July 6, which it says wiped the cache and messages left there.

On July 7, a new wave of ExploitGym experiments launched tens of thousands of agents, and on July 8 one agent established the primary unsanctioned message board. Roughly 1,200 agents eventually used the board, sending more than 70,000 messages and files; about 700 later participated in the attack on Hugging Face. The attack itself began on July 10–11 and wound down over July 12–13. The chronology matters because the administrator-access event, the Artifactory outage, and the later message-board activity were separate phases, not one continuous incident.

By binding autonomy, worker agents operate with the minimum permissions required for their specific tasks, keeping sensitive operations behind explicit approval boundaries. This also reshapes human review. Requiring human approval for every tool call destroys the efficiency of multi-agent execution, but showing only the final result obscures critical intermediate decisions.

The most useful design places human intervention around consequential, irreversible transitions — like moving into production or altering sensitive infrastructure. This is especially vital as agents become event-driven participants that respond to Slack messages or pull request updates, not just direct prompts.

OpenAI and Cursor own different parts of the architecture

The convergence does not mean OpenAI and Cursor have built interchangeable systems. Their products put the orchestration boundary in different places.

OpenAI is exposing an agent harness through an API. Its model gives developers primitives for managing context, tools, subagents, and execution environments, leaving application teams to decide how those capabilities fit into their own systems. The harness is open source, so teams can inspect the coordinator logic instead of treating it as a black box.

Cursor packages more of the surrounding workflow. Projects provides the coordinator, cloud execution, shared project context, and a developer-facing workflow in the same environment.

That difference matters because orchestration is a collection of infrastructure decisions: who owns the execution environment, where workflow state persists, how agents are isolated, how credentials are provisioned, what happens when a worker fails, how one agent’s output becomes another agent’s input, and which actions can happen without human approval.

An API gives developers more responsibility for answering those questions. An integrated platform answers more of them on the developer’s behalf.

Neither approach removes the underlying engineering problems. It changes where they are implemented and who is responsible for operating them.

The coordinator is becoming an architectural boundary

The evidence from these systems points to a change in the role of the coding agent itself.

The model still performs the reasoning and code generation. But larger agentic workflows require another layer to determine how that capability is applied: which work is delegated, what context crosses an agent boundary, which tools are exposed, how execution state survives failures, and when the workflow needs human intervention.

Those are familiar distributed-systems concerns. Workers operate concurrently, state can be shared or isolated, dependencies connect tasks, workers can fail independently, and results need to be persisted and observed. The difference is that the workers are now probabilistic software agents rather than conventional processes.

That makes the coordinator more than a convenience feature. It is where a high-level software objective becomes executable work — and where decisions about context, permissions, durability, observability, and human intervention converge.

The September 10 launches make that shift visible from two different directions. OpenAI exposed orchestration infrastructure through an API. Cursor embedded it into a project-level development environment.

Neither announcement proves that one architecture will become the universal model for software development. But together with the systems already emerging around them, they show coding agents moving away from a single model executing an entire task and toward workflows that divide work among specialized agents, execution environments, and persistent infrastructure.

The engineering question is therefore no longer only whether an agent can write the code. It is whether the system around it can reliably decide what to do, which agent should do it, what that agent should be allowed to see and change, how to verify its work, and where a human should take control.

Those are architecture and infrastructure questions — and as coding agents move from interactive assistants toward autonomous software workflows, they may matter as much as the underlying model.

The post OpenAI and Cursor agree on agent coordinators. They disagree on who runs them. appeared first on The New Stack.

Received — 24 September 2026 ⏭ AI Infrastructure Archives - The New Stack

What managing 150,000 AI agents could look like for database teams

Abstract 3D render of translucent orange cubes and panels scattered across a pale gray background, with bundles of glossy teal tubes curving in from the right.

The database administrator of the future will spend considerably less time administering databases.

That sounds contradictory, but AI agents are taking over that work. For decades, DBAs have handled the decidedly hands-on work of keeping databases available, performant, secure, and affordable. They provision capacity, troubleshoot slow queries, manage migrations, and step in when something inevitably goes sideways.

AI is already taking on some of that work. At the same time, it is creating a much bigger data infrastructure fleet to manage.

The result is likely to be a very different kind of DBA: one who spends less time tending individual databases and more time supervising the autonomous systems doing it for them.

Congratulations, you’re managing robots now

This shift starts with a familiar problem: more infrastructure needs managing than the people available to manage it.

Database automation is hardly new, but agents can potentially go further than the scripts and rules DBAs already rely on. Rather than automating one predetermined task, an agent can inspect what is happening, decide what needs attention, use tools to act on it, and check whether its intervention worked.

That changes the DBA’s relationship with the database. A performance problem that once required someone to dig through metrics, identify the troublesome query, and decide how to respond could increasingly be investigated by an agent before a human gets involved.

It doesn’t remove the DBA from the equation. Someone still has to decide what an agent can do, where human approval is required, and what happens when it gets something wrong. But the work moves up a layer. Instead of personally performing every operational task, DBAs start managing the systems carrying them out.

Instead of personally performing every operational task, DBAs start managing the systems carrying them out.

And before anyone gets too comfortable with that idea, the number of those systems could become enormous.

150,000 agents walk into a database…

Gartner predicts that the average global Fortune 500 company will have more than 150,000 AI agents in use by 2028, up from fewer than 15 in 2025. Only 13% of organizations currently believe they have the right governance in place to manage them.

Not every agent will need its own database, but plenty will. They will create state, retrieve data, remember previous interactions, and exchange information with other agents. Many will also behave very differently from the applications DBAs are used to supporting: spinning up quickly, sitting idle for long stretches, and suddenly becoming busy when there is work to do.

Nobody is hiring 150,000 DBAs to manage them.


That is the scale problem Yugabyte is targeting with YugabyteDB AMP, or Agentic Multitenant PostgreSQL. Rather than treating each new agent workload as another database for an administrator to provision and babysit, AMP manages databases as a fleet.

The platform packs hundreds of small Postgres workloads onto shared distributed infrastructure while keeping their databases isolated. Lifecycle operations, including provisioning, branching, scaling, migration, and teardown, can be exposed to agents through MCP. Yugabyte has also built specialized agents for setup, migration, performance tuning, and integrations.

In that model, a DBA is no longer provisioning database number 14,372. The interesting job is setting the rules for how database number 14,372 is provisioned, operated, and fine-tuned without them.

Do more with less (no, really)

Scale is only half of the problem. Someone also has to pay for all this stuff.

Agent workloads make traditional capacity planning particularly awkward because many are bursty and frequently idle. Giving every experimental agent permanently provisioned infrastructure could leave companies paying for many databases that spend much of their lives doing very little.

This is where consolidation becomes as much an economic question as an operational one.

AMP’s approach is serverless multitenancy and scale-to-zero. Multiple small workloads share the underlying distributed infrastructure, while customers pay by CPU minute and idle agents consume no compute. Resource governance can impose CPU limits on individual workloads, preventing a single overeager agent from consuming the capacity intended for its neighbors.

The human equivalent matters too. If routine setup, migrations, tuning and other database operations can increasingly be delegated, a smaller database team can potentially look after a much larger estate.

That doesn’t mean companies get to fire the DBAs and hand the keys to the robots. It means scarce database expertise can be spent on architecture, governance, and genuinely difficult problems instead of repeatedly doing the work that software can handle.

Your 2028 database problem starts now

The harder question is what to build underneath all of this when nobody really knows what the enterprise AI estate will look like in two years.

An agent that begins as an experiment today could disappear next month. Another could suddenly become a production application used across the business. Building one infrastructure stack for cheap experiments and another for serious workloads risks creating a migration problem every time an experiment succeeds.

Yugabyte bets that both ends of that journey should sit on the same foundation.

YugabyteDB AMP lets workloads start on serverless Postgres and transition to fully distributed YugabyteDB as their scale and criticality increase, without rewriting the application or migrating data to a different database platform.

Then there is the problem above the individual database: agents need to remember what happened, and not just in a silo.

That’s where Meko fits into the Yugabyte stack. Meko is an agent-native context engine designed for multi-agent AI systems. It provides persistent memory, shared knowledge, decision traces, and autidability across multiple agents, rather than leaving each agent working from its own isolated context. An agent can pick up information learned by another agent instead of retrieving it again or restarting the reasoning process.

Taken together, it delivers a single data stack for an agent’s entire lifecycle: Meko for the context shared among agents, YugabyteDB AMP for agentically managing fleets of Postgres databases, and distributed Postgres-compatible YugabyteDB for workloads that outgrow their serverless beginnings.

Of course, there’s no guarantee that 2028 will look exactly like today’s forecasts. That’s rather the point. The safest architectural bet may be one that doesn’t require you to know in advance which of today’s tiny AI experiments will become tomorrow’s critical applications.

The DBA is still critical in that world, but the job will look different. The DBA of the future may manage fewer databases directly, while taking responsibility for vastly more of them. Instead, managing the autonomous systems that do the administering.

The post What managing 150,000 AI agents could look like for database teams appeared first on The New Stack.

Received — 21 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Your AI agent is burning tokens on choices that don’t need words

Blur or abstract motion

AI agents spend a ridiculous amount of compute generating text nobody actually needs. The decisions an agent makes along the way don’t require a written answer and yet, agents still send them to generative models, wait for an answer while burning through tokens and then parse that output back. The overhead is already drawing scrutiny — OpenAI’s own researchers recently disclosed spending $7,000 a day running agent workloads.

Kev, a new family of open decision models built on Qwen 3.5, takes a different approach and skips the generation entirely.

Developer Jared Palmer released a new generation of Kev on Sunday, with 0.8 billion, 4 billion, and 9 billion parameter models built on Qwen 3.5. Kev is prefill-only, processing the state, questions, and candidates in a single forward pass before reading the decisions from a pointer head without an autoregressive decoding loop.

Kev, a new family of open decision models built on Qwen 3.5, takes a different approach and skips the generation entirely.

Decisions without generated text

Kev supports three decision types: Noul for yes/no, Choice for selecting among candidates, and Score for ordered levels, mirroring TypeSafe’s System One API. Developers provide the state and questions, and the pointer head returns probabilities across the available candidates.

For a tool-routing decision, the output could look like this:

search: 0.82

database: 0.13

calculator: 0.05

Kev can still choose the wrong tool, but because it scores only the candidates it’s given, it can’t introduce an option that isn’t on the list.

Routing, safety checks, escalation, and ranking can then move to the decision layer, leaving larger reasoning models to handle the open-ended work.

Kev can still choose the wrong tool, but because it scores only the candidates it’s given, it can’t introduce an option that isn’t on the list.

Batching choices, one pass

Multiple decisions can also be made against the same state in a single forward pass, with a block-causal attention mask isolating the questions while the pointer head scores each set of candidates independently.

Palmer’s documentation shows the 4B model processing three questions in 277 milliseconds in bf16 on an M5, although without a controlled comparison against Qwen generating equivalent answers on the same hardware, the result doesn’t establish how much faster the approach is in practice.

The ability to evaluate several decisions against the same context could become more useful as agent loops grow more complex, but skipping generation doesn’t make the resulting decisions inherently better.

Calibration limits and tradeoffs

The largest model, Kev-9B, reached 83.7% accuracy on the project’s locked out-of-domain test, according to Palmer’s model card. That’s a developer-reported benchmark, and Palmer documents some limitations alongside it.

The probabilities Kev returns don’t always reflect how confident developers should be in the result. Palmer found that temperature calibration can drift on unseen source distributions, a problem for agents that use probability thresholds to decide whether to execute an action or escalate it, since even a high-probability choice can still be wrong.

Fine-tuning also changes some of the capabilities inherited from the underlying model. Palmer’s evaluations show declines on general-knowledge and arithmetic tests, particularly among the smaller models. That’s consistent with Kev’s more specialized role alongside a general-purpose model, although its performance in dynamic agent environments will also depend on how well it handles tools, choices, and labels it never encountered during training — and debugging agent failures often points to infrastructure rather than the model itself.

The approach predates Kev. TypeSafe introduced Jev earlier this month as part of its System One platform, using the same Noul, Choice and Score primitives, and Kev implements its /v1/systemone request and response format so applications built against the API can point to a local Kev server instead.

Open weights, open training

The biggest difference is that Palmer released Kev under Apache 2.0 with the model weights, training code, and evaluation tooling, giving developers the option to run and train it on their own infrastructure. Jev’s weights and training data aren’t public, however, which makes direct performance comparisons difficult because differences between the models can’t be isolated to architecture, size, or training.

For applications that make only a handful of bounded decisions, constrained decoding on a model that’s already running may be simpler than adding another model to the stack. Agent loops can make those decisions constantly, however, moving through routing, ranking, safety checks, tool selection, and escalation before generating much user-facing text. It’s a pattern showing up across model architectures — stripping out unnecessary computation when the task doesn’t require it.

When those steps only require a choice or probability, Kev can handle the decision directly while leaving open-ended reasoning and final responses to the larger generative model.

When those steps only require a choice or probability, Kev can handle the decision directly while leaving open-ended reasoning and final responses to the larger generative model.

The post Your AI agent is burning tokens on choices that don’t need words appeared first on The New Stack.

Received — 18 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Your agent is only as good as your infrastructure

Dark abstract 3D glass ribbon rendering symbolizing complex AI agent infrastructure and bursty data workflows.

You built a great agent, but something happened when it moved into production.

In testing, your agent reviewed pull requests efficiently on its own. It read the diff, grepped the codebase for related usages, ran the test suite, checked whether CI was still red from an earlier commit, and drafted a comment—all before you’d finished reading the diff yourself.

In production, however, imagine the same five steps ran behind every other PR review your team’s agents performed that hour. Some reviews landed in seconds; others sat for minutes because the test-suite step landed on a node that was mid-burst from someone else’s agent. 

The agent didn’t change. The execution environment did, and that’s what decided whether review time held steady or crept up.

Agentic applications introduce a different execution pattern than traditional chat applications. As those workflows become longer and more dynamic, infrastructure has a much larger influence on latency, reliability, and cost than it does for a simple chatbot.

That’s why your agent is only as good as your infrastructure.

Agents aren’t chatbots with more steps

The difference between serving inference for a chatbot vs. an AI agent isn’t simply that one is “more capable.” They execute work differently:

  • A chatbot usually makes one inference call per user message. The model receives a prompt, generates a response, and waits for the next user input before proceeding.
  • An agent executes the entire workflow, turning one user message into a chain of inference calls. It might decide to search documentation, retrieve data from a database, call an API, execute code, evaluate the result, and then repeat that process before producing an answer. Each of those decisions may trigger another inference call, and every result becomes additional context for the next step.

“Infrastructure has a much larger influence on latency, reliability, and cost than it does for a simple chatbot.”

That execution model changes the infrastructure requirements for AI agents. 

One question, many steps behind it

Instead of optimizing for individual inference requests, the system has to support long-running workflows whose latency and reliability depend on every component in the chain.

A single user request often expands into a sequence of inference and tool execution steps, sometimes called multi-turn tool calls or the agentic loop. Rather than generating one response, the model alternates between reasoning and interacting with external systems.

For example, you ask an agent why checkout latency spiked overnight. The agent pulls the deploy log, queries the monitoring system, runs a diagnostic against the connection pool, weighs whether the culprit is a bad deploy or a capacity issue, and then folds that into another inference call before finally producing a full-fledged response.

Each reasoning step becomes another inference request, and every tool result is added to the model’s context before the next step.

This workflow changes what reliability means

Multi-turn workflows are inherently sequential, which is why even low latency can quickly add up to a significant amount. Every inference step waits for the previous one to finish. If a database query takes two seconds, the model can’t begin the next reasoning step until that result returns. The model may generate tokens quickly, but the other steps slow it down.

“In this agentic workflow, every step in the chain has to hold, because the chain is only as strong as its slowest link.”

In this agentic workflow, every step in the chain has to hold, because the chain is only as strong as its slowest link. Instead of processing isolated inference requests, the inference stack has to orchestrate a chain of dependent model invocations and external tool calls. As those workflows become longer, the stack increasingly determines how quickly, reliably, and cost-effectively the application performs.

But the user doesn’t see an orchestration hiccup. They see an agent that hung or gave up.

That’s why end-to-end agent reliability depends on much more than model quality. Infrastructure determines whether each step has the resources it needs to execute predictably under load.

Why the bill and the performance both feel unpredictable

A second difference in agentic workflows catches teams off guard: demand patterns and their impact on your inference bill. 

Most inference services, and the pricing built on top of them, assume traffic arrives at a predictable pace. A typical inference solution knows the predictable demand pattern: User traffic increases, request volume increases, and capacity scales accordingly. Cloud infrastructure is typically optimized for these steady request patterns using mechanisms such as autoscaling, load balancing, and capacity planning.

Agent workloads don’t behave that way. Individual workflows pause while waiting on external systems, then resume as soon as new information becomes available.

  • The pause: The agent waits on an external API or database, so the GPU serving that workflow has no inference work to perform, and its accumulated context may be evicted from GPU memory while it waits
  • The burst: As soon as external systems return results, inference resumes simultaneously across many workflows, creating short and sharp spikes in GPU demand, each re-processing its full accumulated context

If you’re watching GPU utilization and it looks less like steady load and more like a heartbeat — flat, then a spike every time tool results come back — that’s the signature. It means you’re provisioning for the average when you should be provisioning for the peak, and it’s usually the first place p99 latency quietly blows out.

“If you’re watching GPU utilization and it looks less like steady load and more like a heartbeat, that’s the signature.”

Inference services designed around steady or predictable request streams can struggle to allocate resources efficiently under these conditions. This leads to inconsistent latency, GPU underutilization, or higher operating costs.

When performance becomes unpredictable, or a bill doesn’t match what you expected, it’s evidence your infrastructure was built for a different workload than the one you’re actually running.

What infrastructure built for agents actually looks like

Agentic applications place different demands on infrastructure than traditional AI workloads: long dependency chains, bursty demand, and continuous evolution. Because of this, the infrastructure is deciding whether the chain holds, and whether the bill holds too.

For an agent to run, it needs infrastructure purpose-built to support:

Performance that holds across the whole chain. The infrastructure must keep latency consistent across multi-step and multi-tool workflows.

Scalability that responds to bursty demand. Infrastructure should scale quickly as inference demand fluctuates, without requiring capacity to remain provisioned during idle periods.

  • Predictable economics even for dynamic workloads. The infrastructure bill should reflect actual usage.
  • None of this makes bursty demand disappear, but it changes how the system absorbs it. A large enough simultaneous burst, or a workflow that accumulates enough context before pausing, still costs something. The goal isn’t zero cost or zero limit; it’s making both predictable.

Your agent is only as good as your infrastructure. Get it right, and your agent’s responsiveness, reliability, and cost-effectiveness will improve your work. 

Learn more about how infrastructure can be purpose-built for agentic workflows: Check out the documentation to get started.

The post Your agent is only as good as your infrastructure appeared first on The New Stack.

Received — 16 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Perplexity’s AI agents helped build a database. They weren’t allowed to run it.

Abstract glitch wave

Perplexity decided it was paying too much for DynamoDB and wasn’t getting the control it wanted over read performance. So it built its own database: CobbleDB.

Built by two engineers in two months with help from hundreds of persistent coding agents throughout development, CobbleDB is a roughly 40,000-line Rust key-value store that now handles part of Perplexity’s production search traffic. The company measured median batch-read latency at 5.6 milliseconds after the move, compared with 31.4 ms on DynamoDB before the cutover, while p99 went from 123 ms to 24.2 ms.

It’s expected to cost at least 20% less than DynamoDB and plans to open-source the database at some point.

But the database itself is only part of the story. CMU professor Andy Pavlo argued at Percona Live earlier this year that databases are the hardest and most important challenge for AI agents, in part because mistakes involving production data can be difficult or impossible to reverse.

Perplexity went ahead and used hundreds of agents to help build one anyway, but they weren’t given the keys to production.

It’s expected to cost at least 20% less than DynamoDB and plans to open-source the database at some point.

Why DynamoDB couldn’t keep up

Each search requires the serving layer to retrieve pre-chunked passages and vector embeddings, with a single Search API call fetching 100 to 120 page keys in batches of 10 to 20. Each item averages about 50 KB.

DynamoDB gave Perplexity little control over how it handled reads, which meant a slow replica could hold up the entire things. It also charged for the steady flow of large reads and writes generated by search, crawling, and reprocessing, which made cloud costs difficult to justify as traffic and the corpus grew.

That led Perplexity to separate long-term document storage from the database serving live searches.

Three tiers for search data

The storage stack is split into three pieces. Pillar keeps durable document state in YTsaurus on HDDs, including versioned metadata, chunks and embeddings, while Lorry packages updates into partition-specific batches and moves them through S3 to CobbleDB.

Processed page data is spread across three replicas per partition, with hashed URLs as keys and RocksDB keeping often accessed data in memory while the rest stays on local NVMe. Reads stay within the same availability zone when possible, and the router can try another replica if one is slow rather than hold up the batch.

Updates come through S3 and are applied independently, allowing a replica to fall behind and catch up without blocking the others.

Roughly 5X Lower Batch-Read Latency

Perplexity was handling approximately 200,000 requests per second when it measured CobbleDB at 5.6 ms for a median batch read, down from the 31.4 ms it had recorded on DynamoDB. At p99, latency went from 123 ms to 24.2 ms.

In later load testing, CobbleDB reached 500,000 requests per second before performance started to decline.

The comparison comes with an important caveat; DynamoDB and CobbleDB weren’t tested side by side against identical traffic: the DynamoDB figures were recorded before the cutover, and CobbleDB’s afterward. Perplexity separately ran synthetic benchmarks using batches of 10 to 15 keys with values ranging from 100 bytes to 100 KiB.

Its cost model puts CobbleDB at least 20% below DynamoDB across the commitment options evaluated, though that estimate doesn’t include the engineering cost of supporting the database.

In later load testing, CobbleDB reached 500,000 requests per second before performance started to decline.

Agents built it, engineers controlled it

The agents carried context across sessions, catching problems with restore assumptions and runtime configuration while working on fixes and tests. But they weren’t running the database.

The two engineers kept control of the architecture and production system, particularly important given Pavlo’s warning about putting agents near critical production data.

Ownership has long-term costs

Shipping CobbleDB in eight weeks solved Perplexity’s immediate engineering bottleneck, but maintaining a custom datastore could prove considerably harder. The latency results aren’t from a controlled side-by-side benchmark, and the projected savings don’t include the engineers needed to maintain CobbleDB and respond when something breaks.

Like Shopify and Ramp, which built custom coding agents around third-party models, Perplexity kept the cloud infrastructure but replaced a managed service with something built for its own needs. CobbleDB shows how AI-assisted development is changing that calculation, making custom infrastructure more practical for smaller engineering teams.

CobbleDB shows how AI-assisted development is changing that calculation, making custom infrastructure more practical for smaller engineering teams.

The post Perplexity’s AI agents helped build a database. They weren’t allowed to run it. appeared first on The New Stack.

Anthropic bet users were choosing wrong. So it removed the choice.

Single lane

Using Claude for anything beyond a quick question has always started with a routing decision to use Chat or Cowork? Anthropic has decided to eliminate that fork.

Starting Wednesday, Claude Chat and Cowork merge into a single interface where one conversation can handle everything from a simple answer to a multi-step project with connected tools and background execution. The company is also launching Claude Docs and Claude Slides in beta on paid plans, and moving Claude Design — previously a standalone workspace — into conversations.

The combined effect promises a streamlined experience with Claude picking up context, skills, and connectors as the work requires, and can keep running after you close your laptop.

The company is also launching Claude Docs and Claude Slides in beta on paid plans, and moving Claude Design, previously a standalone workspace, into conversations.

Two modes, one problem

Anthropic built Cowork as a desktop-first agent for bigger work and Design as a separate workspace for visual output. Both shipped earlier this year and gained traction.

“We built Cowork as a separate place for bigger work, and Design for visual work,” Anthropic said in its announcement. “People used both, and told us the frustrating part was deciding where a task belonged.”

Anthropic has run into this problem before. When the company promised 20x more usage on its Max plan, developers complained that it wasn’t always clear where one limit ended and another began. Cowork and Design created a similar headache by making people decide where to start the work before they could actually start it.

Context didn’t always follow the work either, so moving from chat to Cowork or Design could mean bringing the same background along all over again. Anthropic addressed part of this in August when it unified Claude’s memory across chat and Cowork. Wednesday’s change goes further by merging the products themselves.

“We built Cowork as a separate place for bigger work, and Design for visual work,”

Context that finally travels

Cowork’s capabilities — local file access, multi-step execution, scheduled tasks and connected tools — now live inside the conversation. A workflow like that previously meant switching from chat to Cowork and carrying the context with it, and until Anthropic brought Cowork to web and mobile in July, it also required the desktop app.

Claude still asks before taking an action by default, but it can be set to keep working and check in only when something needs a closer look, while recurring tasks such as a weekly report can be scheduled to run every Monday without being started manually.

Output stays in-conversation

Claude Docs and Claude Slides launch in beta on paid plans, bringing document editing and presentation building directly into the app. Claude can turn work from an existing conversation into slides, which can then be edited, presented from Claude or downloaded as PowerPoint or PDF files.

The practical benefit is that a report and a slide deck based on it don’t have to begin as separate jobs with the same background supplied twice. Everything stays attached to the conversation that produced it.

Claude Design also now works inside conversations, in addition to remaining available on its own. For organizations that rely on MCP connectors to wire Claude into external tools and data, the merge means those connections are available wherever a conversation goes — without requiring users to start in a specific mode. Skills, connectors, and artifacts carry across what used to be product boundaries.

The practical benefit is that a report and a slide deck based on it don’t have to begin as separate jobs with the same background supplied twice.

What Anthropic hasn’t said

The announcement leaves some gaps. It doesn’t say whether users can force a request to stay in simple chat mode rather than letting Claude decide how to handle it, or how that decision affects context windows and token consumption. There’s no mention of API changes, which makes this a consumer and team product shift, not a platform one, at least for now.

It also doesn’t address what happens to workflows built around the old separation. Shopify rebuilt its mobile development stack in 12 weeks when it consolidated tools that had grown apart — the question for Claude power users is whether their existing Cowork setups, skills, and scheduled tasks survive the merge cleanly. Anthropic says existing Cowork chats, projects, artifacts, connectors, and skills will remain available.

Rollout starts with Pro

The unified interface rolls out to Pro and Max users across web, desktop, and mobile over the next few weeks. Anthropic says there’s nothing to enable. Team and Free plans follow. Enterprise customers are on a separate timeline; Anthropic is giving administrators at least 30 days’ notice before the change reaches their organizations.

The post Anthropic bet users were choosing wrong. So it removed the choice. appeared first on The New Stack.

Received — 14 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Perplexity’s new agent runs entirely on your GPU — with one expensive catch

Abstract server

Running an LLM on your PC is easy enough, but putting an agent to work there is a different story. Portable Computer, the local version of Perplexity’s Computer agent, is now available inside the Perplexity app for Windows on compatible Nvidia GeForce RTX and RTX PRO GPUs.

That’s the good news; the catch is, you’ll need an Nvidia GPU with at least 24GB of VRAM.

The Windows launch gives Perplexity three platforms in less than three weeks. Portable Computer debuted on Linux and Nvidia DGX Spark on August 25, followed a week later by hybrid compute for Apple silicon, which splits tasks between local and cloud models on Macs. Now Windows joins the mix, but bringing Portable Computer over took more than simply porting the app. Perplexity had to adapt the model runtime, orchestration, security, and hardware integration for each platform while keeping the user experience the same.

you’ll need an Nvidia GPU with at least 24GB of VRAM to use it.

Orchestration beyond the model

Portable Computer bundles those pieces together. On Windows, it supports PPLX 27B — Perplexity’s post-trained model — and Qwen 3.8 27B, both optimized for RTX GPUs, alongside a built-in browser, tool calling,  and Perplexity’s proprietary SPACE sandbox.

It’s a different lane from LM Studio or Ollama, which make running models locally as painless as possible but stop well short of giving a model autonomy over multistep work. DeepSeek’s recent hiring spree of roughly 150 new roles, nearly all of them focused on agent infrastructure rather than the model, hints at how much engineering sits between a capable model and a capable agent.

DeepSeek’s recent hiring spree of roughly 150 new roles, nearly all of them focused on agent infrastructure rather than the model, hints at how much engineering sits between a capable model and a capable agent.

Connectors blur local boundaries

Perplexity ships connectors for Microsoft Outlook, OneDrive, and Word, plus Google Drive, Gmail, Slack, and GitHub — which tells you something about what “local” actually means here.

The agent can reach external services because it’s not air-gapped. Locally completed tasks can process files without sending documents to a cloud model. Once an agent has access to both local files and remote APIs on the same machine, figuring out which resources it actually needs and where to find them gets harder.

Hybrid cloud as fallback

Perplexity isn’t pretending that a 27-billion-parameter model running on a desktop GPU can handle everything, which explains the hybrid architecture. When the agent determines that a task needs more reasoning power than the local model can deliver, it can escalate to Perplexity’s cloud models.

According to Nvidia, the agent identifies when cloud support would help and asks the user for permission before sending any data off the machine.

For organizations handling sensitive or regulated data, that split can make all the difference. A local agent can grind through source code or financial records without uploading them to a hosted model for basic processing. There’s a cost angle too, since tasks completed locally don’t burn Perplexity Computer credits.

High VRAM floor limits reach

Portable Computer is available with Perplexity Pro ($20/month) and Max ($200/month), across individual and enterprise plans, with Nvidia DGX Station support coming later. The real challenge is taking local agents from developer passion projects to enterprise-ready tools. By baking this into Windows, it immediately gets in front of the scale of users needed to make that happen.

The real challenge is taking local agents from developer passion projects to enterprise-ready tools.

The post Perplexity’s new agent runs entirely on your GPU — with one expensive catch appeared first on The New Stack.

Received — 13 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Chip Huyen explains how to cut inference costs without new hardware

Layers of wavy yellow horizontal strips with deep shadows between them, forming an abstract pattern.

Last October, the P99 conference — the online gathering for developers focused on high-performance, low-latency applications — featured a cracking keynote from Chip Huyen. 

The author of the best-selling AI Engineering, Huyen opened with simple math: Training a frontier model is a one-off cost, but inference is the same cost paid over and over. That’s great for the frontier model providers, and bad for us token burners. Over the life of a model, Huyen reckons the compute split ratio lands somewhere between 1:10 and 1:100 for training to inference. Reasoning models — which burn even more tokens — push that out even further. We all know the feeling of hitting our weekly session quotas.

Huyen’s point is that if inference is too expensive, then nobody ever recovers the training bill, which might explain why there are so many memes about the “profitability” of frontier models. So, how do we optimize inference? 

That’s a topic that Huyen spent months researching for her book. And in the spirit of optimization, Huyen distilled it down to 30 minutes for the conference in October 2025.

Huyen is returning for P99 CONF 2026 in a few weeks. Ahead of that moment, watch her full talk – or read the recap below – from last year, then let’s talk about how those ideas aged over the past 11 months.

What to measure

Chip recommends focusing on a few key latency metrics:

  • Time to first token (TTFT): How much time elapses before the user sees anything
  • Time per output token (TPOT): The average time between consecutive tokens (aka inter-token latency)
  • End-to-end latency: Time to first token, plus time per output token, multiplied by the number of output tokens minus one
(Click to enlarge graphic.)

With reasoning models, some of those tokens never reach the user. “The first generated token might not be the same as the first visible token,” Huyen explained. “The model might think for a while, and it will only show the first token of the final output to the user.” 

“The first generated token might not be the same as the first visible token,”
— Chip Huyen

Some people also measure Time to Publish for that (i.e., how long until the user sees the first token). The best metric to prioritize depends on what matters most for your users. 

Also consider “goodput” alongside throughput. Throughput measures requests processed in a given window. Goodput measures the requests that actually met your targets. Chip’s example: an app targets 200 ms time to first token and 100 ms time per output token, and processes 10 requests per minute, but only three hit both. 

(Click to enlarge graphic.)

3 ways to optimize LLM inference

With inference servers, you can optimize from 3 different angles: the hardware, the model, and the service that manages the requests and responses.

(Click to enlarge graphic.)

Huyen previously worked at Nvidia and opted out of the hardware discussion: “Even though I find it to be an intellectually interesting topic, it’s not relevant to a lot of people because we don’t have the power to change the hardware itself,” Huyen explained. She also didn’t want to spend much time on the obvious solution: replica parallelism, or just adding more machines. It’s costly, and it gets complicated fast – especially if you end up with a mix of 80GB, 48GB and 24GB machines and models of varying sizes to distribute across them.

That leaves the model and the service. Huyen offers these tips on how to decide: “If you want to host the models yourself, or if you have access to the model weights, or if you train a model yourself, or you want to fine-tune or distill a model, then model optimizations might be for you. However, if you want to take a model as-is and make it more efficient on your own inference service, you might want to look into service optimizations.

Model optimization

The following techniques change the actual weights so that they can change the model outputs.

Quantization lowers the precision used to store weights and activations  (e.g., from four bytes per parameter at 32-bit to one byte at 8-bit). Huyen explained, “Reducing the precision not only reduces the memory requirement to run the model, making it cheaper. It can also make the model a lot faster. If you do additions bit by bit and each weight is 32 bits, you have to do it 32 times. If it’s 8 bits, you only have to do it eight times.”

The tradeoff is a small quality hit. Huyen continued: “It’s possible to reduce a lot of the model’s memory footprint with minimal quality degradation, and quantization is pretty generalizable to a wide variety of model architectures and model sizes. That’s why it’s very popular. I rarely see any companies running a model at full precision anymore.”  

“I rarely see any companies running a model at full precision anymore.”
— Chip Huyen

Distillation involves using a large model to generate training data for a smaller model. For example, say you have a truly large model (the example Huyen used was o1) and want a model that performs like it, but is much smaller. Basically, you collect a large set of prompts, run them through the larger model, then train the smaller model on its responses.

Proceed with caution, though. Huyen warned, “A lot of model providers have the condition that they do not allow their models to be used to train competitive models. So even though it’s a very common technique, you need to check licensing.”

Service optimization

This set of techniques targets how requests are scheduled, routed, and reused. The actual weights aren’t affected.

Batching groups multiple requests so they’re processed together in a single pass through the model – which is much more efficient than dealing with them one at a time. Huyen presented a few batching options:

  • Static batching waits for the batch to fill. This maximizes compute utilization, but it might increase the latency for the first requests.
  • Dynamic batching runs on a timer instead (e.g., batching every 15 ms). This is less compute-efficient, but it’s better for latency.
  • Continuous batching handles the case where requests finish at wildly different times, which is common with LLMs. One request asks for the capital of Vietnam; another kicks off deep research. With static or dynamic batching, the finished request’s slot sits idle until the slowest one completes – and new requests queue up behind it. Continuous batching returns each request as it finishes and fills the spot with another request. That can improve compute resource utilization and latency.
(Click to enlarge graphic.)

Decoupling prefill and decode separates the two phases of a request onto different machines. (Prefill processes the input, while decode generates the output.) Huyen said, “Input tokens can be processed in parallel, whereas output tokens need to be generated sequentially. With parallel processing, it’s bounded by compute, the processing power of the chip. With decoding, it’s bounded by memory, because you have to move model weights.” 

Because each phase stresses different resources, most services now separate them. To improve time to first token, shift machines toward prefill. If you care more about improving time per output token, shift them to decode. 

(Click to enlarge graphic.)

Parallelism splits work across machines. Replica parallelism copies the whole model onto more machines. Tensor parallelism divides a very large matrix, so different machines compute different parts of it. Pipeline parallelism divides the model by layer, so requests move through as a pipeline. 

(Click to enlarge graphic.)

Prompt caching processes shared text once, saving cost and latency. A lot of repetition exists across requests to the same application: the system prompt, the examples, the same code base, the same document behind different questions. You might as well process that shared segment once, cache it, and reuse it.

The technique was relatively rare when Huyen was writing AI Engineering. “There was one paper about it, and it was not really known, but it made a lot of sense. So I included prompt caching in the book, and I’m very happy to see that nowadays it’s pretty much everywhere.”

(Click to enlarge graphic.)

The savings scale depending on how much of your prompt gets cached. In Claude Code logs, Huyen’s open-source tool Sniffly found cache hit rates of 90% to 97%. Some providers rewrite prompts internally to improve hit rates, but you might as well structure them yourself. 

Huyen’s tip: since caching works on shared prefixes, put the stable parts of your prompt first and the variable parts later. “It’s pretty easy to do, and it can improve your application performance significantly,” she noted. 

Evaluating inference providers

Huyen closed with a warning for anyone evaluating inference providers: “There are many inference companies that provide inference optimizations for models you want to use, and a lot of them advertise just cost and latency. 

“But pay attention to how many inference optimization techniques also change the model behavior or reduce the model quality. So when evaluating an inference service, it’s important to look not just at cost and latency, but also at model quality. Does this model, provided on this service, also perform similarly on standard benchmarks?”

What’s changed one year later?

So where do we stand today, one year on from this keynote? Most of it actually aged quite well. 

On the economics, I reckon Huyen was bang on… I think, for most of us as users, we don’t have all the cost levers to pull that Huyen outlined. But it’s great to understand what is happening. As a novice local LLM user myself, I found I could relate to her points on parallelism (I don’t have it) and prompt caching/quantization (within my grasp of control). 

Prompt caching (which Huyen said was new when Huyen wrote AI Engineering) is now priced into every bundle purchase of API tokens. And her Claude Code observation (90% cache hit rates) is probably the reason we mere mortals can still afford agentic coding agents at all.

Some of it aged in ways that were hard to predict at the time. Huyen mentioned how reasoning models make inference even more significant. One year on, I think agents running multi-step loops with tool calls have turned that idea from a footnote into a way to turn Claude’s rate limits (and their infamous 99.x% availability) on their head. 

All the metrics Huyen described – time to first token, time to publish, goodput under a latency SLO, etc., are all now part of the lingo and probably need to be reasoned about differently. 

That’s one thing I hope she’s talking about this year! Grab a free conference pass and join us online. 

Grab a complimentary pass to PG 99 Conf 2026 and join us on October 21 and 22 to chat with Huyen.

The post Chip Huyen explains how to cut inference costs without new hardware appeared first on The New Stack.

Received — 11 September 2026 ⏭ AI Infrastructure Archives - The New Stack

OpenAI’s researchers burned $7,000 a day on AI agents — now it’s opening the floodgates

speed abstract

OpenAI rolled out its Agents API in public beta Thursday, opening the backend behind Codex to developers looking to run agents unattended for days.

Now, developers don’t have to build their own system to keep an agent going because the API tracks the job as it progresses and gives the agent somewhere to execute its work, even when a task stretches well beyond a single context window.

That makes long-running agents easier to try, but it also gives developers more ways to burn through compute. Interestingly enough, on the same day Agents API launched, OpenAI paused new sign-ups for its $200-a-month Pro plan after demand for GPT-6 Astra strained capacity.

Thibault Sottiaux, engineering lead for Codex, writes on X that Pro subscriptions “put the most strain on our systems,” adding that OpenAI was working to add capacity “as fast as we can.”

To make sure our current users have an incredible experience and continued access to Astra, we are going to pause subscriptions to our $200 Pro plan. These put the most strain on our systems and we wanted to take the smallest step that allows us to continue giving the broadest… https://t.co/WhLEm3HBL7

— Tibo (@thsottiaux) September 10, 2026

The Agents API and ChatGPT Pro are separate products, so there’s no reason to assume one is taking capacity from the other. Still, the timing stands out: the company is making it easier for developers to run agents for hours or days while pulling back access to its heaviest-use consumer plan and working to add more capacity.

Agent inference adds up fast

As a task gets longer, the API can compress earlier context, so the agent doesn’t just stop when it reaches the model’s context limit. It can also bring in tools only when they’re needed or send parts of a larger job to subagents working in parallel. The actual work can run in OpenAI’s sandbox or on infrastructure the developer controls.

The actual work can run in OpenAI’s sandbox or on infrastructure the developer controls.

As agents make progress, they go back to the model for the next step, and a job that takes hours can rack up far more inference than a typical API call. The usage climbs even faster when agents work in parallel.

OpenAI has already seen this inside its own shop. In a research report published September 6, OpenAI said its research organization was logging 3.1 agent-workdays for every human workday by mid-August, measured in standard eight-hour equivalents. The median researcher, ranked by agent usage, was spending more than $600 per day on inference at API prices, while the 90th percentile exceeded $7,000.

Before June, OpenAI’s researchers were still putting in more hours than their agent, but by mid-August, the agents were doing three times as much work.

Arguably, OpenAI’s researchers are an extreme case, but the numbers show what happens when agent use starts to scale. One person can suddenly generate far more inference than their headcount would suggest.

One person can suddenly generate far more inference than their headcount would suggest.

Friction limited compute demand

The Agents API lowers the cost of that experimentation by leaving the orchestration layer out of the bill. Developers pay for the models, tools, and hosted compute their agents actually use.

The flip side is that it’s now easier to consume more inference. Context compaction is a good example. A full context window used to force developers to decide what to discard or how to summarize the work so far. Now the API handles that automatically and the agent keeps going. That’s useful for developers, but it also means the workload doesn’t stop when the context window fills up.

Astra demand hit the ceiling

The Astra rollout offers a preview of what that could look like. OpenAI stopped accepting new Pro subscribers less than two weeks after the model launched on September 3, saying those accounts put the most strain on its systems. The Agents API has its own rate limits and usage tiers, so the Pro pause doesn’t directly affect developers using it. Still, the company is already having to manage capacity around its newest model.

Infrastructure outweighs benchmarks now

The more agents developers run, and the longer they run them, the faster that usage adds up. One developer might have several agents working at once, each going back to the model throughout the task. So headcount alone doesn’t tell you much about how much compute you’re using.

For long-running agents, the challenge is keeping the work moving without wasting tokens or losing track of the task. Cloudflare made a similar bet this summer, arguing that the infrastructure around AI workloads would eventually matter as much as the models themselves.

For long-running agents, the challenge is keeping the work moving without wasting tokens or losing track of the task.

The post OpenAI’s researchers burned $7,000 a day on AI agents — now it’s opening the floodgates appeared first on The New Stack.

Received — 8 September 2026 ⏭ AI Infrastructure Archives - The New Stack

DeepSeek is hiring 150 engineers, and none of them will touch a model

abstract bubbles

Hundreds of thousands of AI agent sandboxes can already run concurrently on a single DeepSeek cluster. Now the company is staffing up to handle what happens as that number — along with its training, evaluation, and other backend workloads — keeps climbing.

Cui Tianyi, who joined DeepSeek in March and works on its Harness team, the group responsible for the infrastructure and environments used to run and evaluate agents, announced in an X post that roughly 150 engineering positions on September 7, with the hiring concentrated in server-side engineering and Agent Elastic Compute rather than AI research. The work spans operating systems, virtualization, networking, storage, scheduling, and the control-plane services that coordinate those resources.

Cui said DeepSeek’s existing backend systems will need upgrades, maintenance, and rewrites as workloads grow. One such system at the center of that scaling challenge is DeepSeek Elastic Compute, or DSec, the sandbox infrastructure DeepSeek built to execute agent workloads during post-training and evaluation.

Cui said DeepSeek’s existing backend systems will need upgrades, maintenance, and rewrites as workloads grow.

Four sandboxes, one SDK

Agent workloads require more than GPUs for inference, with each agent also needing an isolated environment to run code, call tools, change files, and collect the results.

DSec supports four types of those environments through the same Python SDK. Simple function calls go to pre-warmed containers, while Docker-compatible containers handle jobs that need a persistent environment. DeepSeek uses Firecracker microVMs when stronger isolation is needed and QEMU virtual machines for workloads that require a full guest operating system.

That range means the same infrastructure can handle anything from a simple tool call to a software-engineering task that needs an entire OS. It’s a similar challenge to the one the rest of the industry is bumping into as agents move from demos to production. OpenAI, for instance, recently designed custom silicon specifically to address the compute pressure that agent workloads create, and DeepSeek open sourced its own agent harness in August.

Lazy loading agent environments

Every sandbox needs its own environment, but copying complete container or VM images onto every host would consume enormous amounts of storage and network bandwidth while adding to startup time. DeepSeek gets around that by tying DSec into 3FS, the distributed filesystem it originally built for its AI infrastructure, and keeping container base images and filesystem commits as read-only layers backed by 3FS.

The metadata stays local, but the underlying data blocks are fetched only when they’re actually needed. MicroVMs use a similar setup, sharing their read-only base layer through 3FS while writes from individual sandboxes are kept in local copy-on-write layers.

DeepSeek says DSec reduces duplicate page-cache usage across virtualized environments and reclaims memory to allow safe overcommitment, while changes to the container runtime cut the CPU overhead of each sandbox.

The team also had to deal with spinlock contention inside the container runtime. At small scale, the CPU time spent there barely registers. At scale, it limits how densely those environments can be packed onto each host.

DeepSeek says DSec reduces duplicate page-cache usage across virtualized environments and reclaims memory to allow safe overcommitment, while changes to the container runtime cut the CPU overhead of each sandbox.

When replay breaks training

During reinforcement learning and other post-training workloads, large numbers of agent rollouts can be running at once, and jobs may be interrupted as compute gets reassigned. Starting over wastes everything the agent has already done, but picking up where it left off isn’t as simple as replaying its previous commands.

Some of those commands may have changed a file or otherwise altered the environment, so running them again could produce a different result or leave the training trajectory in the wrong state. DSec avoids that with a globally ordered trajectory log that records commands along with their results.

When a rollout resumes, DSec can fast-forward through the completed work using those recorded results rather than executing the commands a second time. That reduces the cost of interruptions across thousands of training and evaluation runs, while the same logs preserve a history of how each sandbox changed and allow earlier sessions to be replayed.

Engineers, not researchers, wanted

The roughly 150 openings reach across DeepSeek’s backend, including the lower-level systems work behind Agent Elastic Compute as well as the services that support its models and agents.

DeepSeek said in June that it planned to at least double the size of every department, but this round of hiring leans heavily toward the systems underneath its models rather than the models themselves. DSec is part of that work, with hundreds of thousands of sandboxes running concurrently and putting pressure on everything from how jobs are scheduled to how they recover after an interruption.

The roughly 150 openings reach across DeepSeek’s backend, including the lower-level systems work behind Agent Elastic Compute as well as the services that support its models and agents.

The post DeepSeek is hiring 150 engineers, and none of them will touch a model appeared first on The New Stack.

After nine years as HashiCorp CEO, Dave McJannet now wants to “unblock” enterprise AI agents

Retro 3D-rendered computer with a two-icon logo on screen, keyboard, and mouse on a purple background

Ask a traditional enterprise application for a customer address or today’s revenue figures and, broadly speaking, it follows a predictable route its developers have already mapped out: authenticate the user, query the right system, return the result. Given the same underlying data, you’ll get the same answer each time.

Ask an AI agent the same question, and the journey is much harder to forecast. It might consult one system, decide it needs more context from another, make a dozen tool calls, pass information through a language model and only then produce an answer. Run the same request again, and it may take a different route altogether.

And in an enterprise, what happens along that route can matter just as much as the answer: which systems the agent accesses, what data it sees, what actions it takes and how much it spends.

That distinction — between predetermined software, and applications that make probabilistic decisions on the fly — sits at the heart of a new company from a founder who knows a thing or two about bringing order to a new generation of infrastructure.

AI agents are hard to govern

Dome Systems co-founder David McJannet left HashiCop in August 2025
Dome Systems co-founder David McJannet left HashiCop in August 2025

Dome Systems was co-founded at the turn of the year by David McJannet, who spent close to a decade leading Terraform-creator HashiCorp through the cloud era, culminating in its blockbuster 2021 IPO and subsequent $6.4 billion sale to IBM in 2025. McJannet is joined at the helm by Marc Holmes, who spent more than six years at HashiCorp as chief marketing officer.

In an interview with The New Stack, McJannet lays out his company’s thesis on AI agent governance, arguing that enterprises are now running into the same kind of problem that they did with cloud infrastructure: adoption comes first, then the real spadework begins of putting the right controls in place across security, operations and finance.

“It’s actually a very different architecture, and that is what unlocks the power of these new [agentic] applications.”

Part of the challenge, he says, is that agents are built very differently from the enterprise applications of yore, which companies spent years learning how to control.

“It’s actually a very different architecture, and that is what unlocks the power of these new [agentic] applications,” McJannet explains.

He points to self-driving cars as an example: a model takes in live inputs and interacts with the vehicle’s systems as conditions change, because no developer can reasonably pre-program every possible situation a car might encounter on the road.

“It’s making judgments along the way, as opposed to trying to look up the historical maps of the world and make a real-time decision,” McJannet continues.

An enterprise agent can behave in much the same way: call one tool, assess the result, decide it needs another, and keep going until the task is complete. That flexibility lets agents tackle work that would be difficult to script exhaustively in advance — but it also makes their behaviour harder for enterprises to govern.

And this gets to the heart of what McJannet is striving for with Dome.

Table stakes for the agent era

The company launched out of stealth back in April with $14 million in seed funding, with McJannet having departed HashiCorp the previous August after the IBM transition concluded.

Dome’s starting point is that an agent combines three things: code, a model, and the backend systems or tools it interacts with. Bringing those pieces together under one platform, McJannet says, is “table stakes” for applying meaningful constraints to what the agent can do.

“If you don’t have an integrated platform, you can’t enforce controls across everything that the agent is doing,” McJannet says.

“If you don’t have an integrated platform, you can’t enforce controls across everything that the agent is doing.”

And so Dome’s platform is built around those three elements. An agent registry keeps track of the agents themselves; an MCP gateway controls the tools they can call; and a model broker/router governs which models they can use and how requests are routed.

The setup starts by registering the agent and giving it an identity, establishing who is allowed to call it, and connecting the backend tools it can reach — Zendesk, in this example.

Dome registers an agent, verifies its caller and connects the tools it can use.
Dome registers an agent, verifies its caller and connects the tools it can use.

Next, Dome connects a model provider, groups available models into a pool with routing and failover rules, then combines the agent, its tools and its models behind a single gateway. That gateway becomes the point through which Dome can apply the policies governing what the agent is allowed to do.

Dome connects a model provider, creates a model pool and brings the agent behind a gateway.
Dome connects a model provider, creates a model pool and brings the agent behind a gateway.

Once those pieces are connected, teams can set permissions on each call, use guards to inspect responses, apply quotas to cap spending, and keep a common audit trail across the agent’s activity.

Today, McJannet says, enterprises are often piecing all of this together themselves. A standalone model broker might be brought in to control spending, while a separate tool gateway handles security and operational concerns. Some are then building their own agent registry to tie those systems together.

Moreover, buying those capabilities separately leaves enterprises with another integration problem to solve. A model router might govern one part of an agent’s activity and a tool gateway another, while the agent itself continues moving between them.

“If you just provide the tool gateway or just the model router, it doesn’t allow you to have this kind of system of control,” he says.

That is also where Dome’s latest move enters the fray. After spending its first months in early access, the company is now opening the platform to self-service users for the first time, allowing teams to sign up with little more than a credit card, bypassing the typically arduous enterprise sales process.

Dome goes self-serve

Self-serve is relatively unusual route for this kind of enterprise infrastructure product. Dome is publishing its prices, offering a free tier and letting practitioners get started without first going through a sales process, while keeping the traditional enterprise route open for larger customers.

The thinking is partly about who McJannet expects to use the product. Rather than limiting access to buyers who are already deep into a procurement process, for example, self-serve enables individual practitioners to be able to discover, try and use the platform themselves.

“”We want to make the barrier as low as possible to have people come on board,” McJannet says, adding that Dome had already seen a number of self-service sign-ups ahead of the launch.

Separately, its pricing reflects a belief about where value will ultimately sit in this market. McJannet regards model routing and tool connectivity as baseline capabilities, with the more valuable piece being the controls that sit across the agent as a whole — think permissions, data redaction and spending quotas.

It’s also worth noting that while Dome’s main target user will be platform engineering teams inside large enterprises, typically working alongside operations and security, self-serve also creates an opening for another kind of user: the small company, perhaps even only one or two people, building an agent and trying to sell into an enterprise. The sort of scenario that aligns with the fabled one-person unicorn promised by many in the AI realm.

Indeed, McJannet says developers can get far building the application itself, only to hit a wall when a prospective enterprise customer begins its security and operations review. How is identity enforced? Who can see the data the agent reaches? What happens when it calls other agents? Can its activity be reconstructed afterwards?

Some builders, he says, have asked whether they can “certify” their agents on Dome because “my agent won’t get deployed until I can satisfy these infrastructure elements.” McJannet is careful to add that Dome doesn’t currently run such a certification program, but it’s clearly one route the company could venture down.

“If you register that agent on Dome, all the infrastructure elements are taken care of,” McJannet says.

‘Unblocking AI agents’: Lessons from the cloud era

That division between developers eager to ship, and enterprise teams worried about what happens after, is also where McJannet sees the strongest parallel with his years at HashiCorp.

During McJannet’s tenure, HashiCorp increasingly positioned itself around helping large organizations standardize how cloud infrastructure was provisioned, secured and connected. That included the 2020 launch of HashiCorp Cloud Platform (HCP), which offered its infrastructure tools as managed cloud services.

More broadly, McJannet’s account of early cloud adoption begins with developers swiping a credit card and deploying directly to Amazon because cloud infrastructure allowed them to build applications that had previously been impractical. The applications were compelling enough that enterprises adopted cloud despite resistance from operations and security teams, and what followed was a second phase: companies needed common services for provisioning, credentials, networking and other controls before cloud could become routine across the organization.

Platform engineering teams became the people responsible for reconciling those two demands: allowing developers to build while giving security, operations and finance enough control to permit those applications into production. McJannet believes agents are now creating the same tension.

“You’ve got this queue of cool apps that developers build that the ops and security teams are just not comfortable letting flourish in their environments.”

“You’ve got this queue of cool apps that developers build that the ops and security teams are just not comfortable letting flourish in their environments,” he says. “And so, inevitably, it has to go that same direction where the platform engineering team has to figure out [a way] to get to say ‘yes’.”

Dome’s bet is that enterprises will eventually prefer one system spanning the entire agent to a patchwork of gateways, routers and security products. In McJannet’s telling, that common control layer is what gives enterprises a way to limit how far an agent can roam while still letting it act autonomously.

“You have to have this control layer that provides this corridor where we can constrain the behavior of that new type of application architecture,” he says. “Because without that, you cannot unblock the deployment of AI applications.”

“That’s the part that we’re trying to answer — how do we unblock agents at scale?”

There is still plenty for Dome to prove. The company isn’t naming customers at this stage; McJannet says none of the enterprises it has worked with are yet willing to be identified publicly, though he says Dome has spent the past eight months talking to dozens of them.

Ultimately, McJannet believes the cloud era showed that new applications only become commonplace once enterprises have the controls to let them through. Dome is his attempt to solve that problem for agents.

“I think that’s the part that we’re trying to answer — how do we unblock agents at scale?”

The post After nine years as HashiCorp CEO, Dave McJannet now wants to “unblock” enterprise AI agents appeared first on The New Stack.

Received — 7 September 2026 ⏭ AI Infrastructure Archives - The New Stack

AI agents are creating more work, not less — and OpenAI’s own numbers back it up

abstract bot

OpenAI says it hit a goal it set last fall, stating researchers are now using what the company calls an “automated research intern,” which is an agent that can handle well-defined tasks that would normally take a researcher several days.

The data shows coding-agent use climbing throughout 2026, and by mid-August its agents were logging 3.1 agent-workdays for every human workday across the research organization. The median researcher was spending more than $600 on inference per day at API prices, while those in the 90th percentile spent more than $7,000.

The median researcher was spending more than $600 on inference per day at API prices, while those in the 90th percentile spent more than $7,000.

Agent hours versus useful output

But everyone knows that an agent-workday and a human workday aren’t the same. The company converts the time agents spend working on tasks into standard eight-hour workdays. Because researchers can run several agents at once, the figure tells us how long the agents are working, but not necessarily what they’re completing.

For engineering teams, that leaves plenty of work on the human side, which means running more agents can increase the amount of work happening at once, but it can also increase the amount of work a human needs to keep track of.

OpenAI’s very specific definition of a research intern highlights that it must be able to complete well-defined research tasks that would take a skilled person several days, but a human is still in charge. The company’s next goal, an automated AI researcher, is one they hope to reach by March 2028.

The company’s next goal, an automated AI researcher, is one they hope to reach by March 2028.

Supervision becomes the constraint

Using a taxonomy from Epoch AI, OpenAI broke the agents’ work into six areas — Decide, Design, Build, Run, Analyze, and Communicate — and found activity increased across all six between January and August, although agents still did relatively little of the work involved in deciding what research to pursue.

Much of the work is practical, with agents writing research and infrastructure code, monitoring experiments, and providing enough technical support that OpenAI says attendance at debugging office hours has fallen, prompting one team to stop holding the sessions altogether.

And yet, more agent hours don’t automatically mean more useful research. OpenAI says code output and experiment counts are relatively easy to track, but neither shows how much progress those agents actually made. Compute also increased significantly as the number of experiments rose.

OpenAI used another model to judge how well agents performed on tasks of varying difficulty and found that, despite improving success rates between January and July, humans still had to step in on more than half of successful tasks that would have taken a person four to eight hours.

Security incidents limit Astra deployment

Once engineers can run several agents at once, with those agents launching subagents of their own, the challenge shifts to keeping up with what they produce — catching runs that go off track, reviewing code diffs, and deciding what is ready to ship or feed into a training run.

⁠Astra’s persistent-agent capabilities already let researchers hand off multi-day assignments⁠, which makes this supervisory strain worse, not better.

The company acknowledges that as agents take over more of the execution, the parts of research that are hardest to automate will consume more of an engineer’s time, putting a practical limit on how much agent output one person can realistically review.

On July 20, a series of outages caused by agents disrupted OpenAI’s research infrastructure badly enough that the company took its training container service offline and later brought it back with tighter restrictions.

Nearly a month later on August 7, OpenAI tightened access again after early evidence suggested Astra could reach the “Critical” cybersecurity threshold in its Preparedness Framework, restricting the model to higher-security research areas and adding safeguards that developers may already be encountering as unexpected API interruptions⁠

Workloads shift between models fast

Astra-class GPU allocation fell 59.2% the following week, but that compute didn’t sit idle for long. Researchers moved much of the work to other models, which saw GPU allocation rise 17.2% and made up for roughly 85% of the drop in Astra usage. Instead of reducing the amount of work being run, the restrictions pushed it to other models, showing how easily workloads can move when one part of the system is locked down.

OpenAI’s researchers are handing off larger jobs to agents, running more of them at once and launching more experiments, but whether that translates into faster research is harder to measure — and OpenAI is still figuring out how to price it.⁠

OpenAI’s researchers are handing off larger jobs to agents, running more of them at once and launching more experiments, but whether that translates into faster research is harder to measure — and OpenAI is still figuring out how to price it.⁠

The post AI agents are creating more work, not less — and OpenAI’s own numbers back it up appeared first on The New Stack.

Permissions belong in the assembly context

Thousands of warm white string lights form a glowing canopy inside a multistory building atrium.

Someone moves off the finance team at 9 a.m. on a Monday. Your sync runs nightly at 2 a.m. For seventeen hours, that person can still pull finance documents out of your retrieval index, and nothing in the system knows it is wrong. I am borrowing the example from Truto, but every team I talk to recognizes some version of it.

That is the version with a clock on it. The version people ask about in security review sounds different. The retrieval pilot works, the demo lands, the executive sponsor is happy, and then someone asks how you guarantee this thing will never summarize the CEO’s compensation review for an intern who asked an innocent question about salary bands.

Most teams do not have an answer. What they have is a filter.

I think the answer has to be structural. Permissions are not a filter you apply to context after you have assembled it. They are a property of how context gets assembled for a particular identity, because assembly is the last moment where refusing to include something still means the model never saw it.

Permissions are not a filter you apply to context after you have assembled it.

The major platform vendors in this race are building some version of the same step, and nobody has really settled on a name for it. I run a company, Modus, that builds in this lane, so weigh the argument accordingly. In our product, we call it context composition. For this piece, I will call it context assembly. It is where a system decides which pieces of enterprise knowledge to hand a model for a specific person, in a specific moment, for a specific question. Everything upstream is storage, and everything downstream is inference. Assembly is where identity either lives or doesn’t.

Announced is not the same as shipped

The reason to argue about this in September rather than in June is that platform vendors have stopped disagreeing about where the step goes, and the software most companies run has not caught up with them.

AWS made the most explicit version of the case in June, announcing AWS Context at its New York Summit, covered here at the time. The design decision underneath it is the interesting part. The graph is governed by the same permissions as the lake through Glue Data Catalog, SageMaker Unified Studio, and Lake Formation, and identity is checked again when someone asks. The people who would govern it are the ones already governing everything else, with the column-, row-, and cell-level policies that S3 object permissions alone can’t provide.

It is worth being precise about the tense, because the retelling has already blurred it. Every call is “designed to inherit the calling user’s IAM and Lake Formation permissions, so an agent can only see and traverse the relationships its identity is authorized to access.” Designed to. That is a roadmap language, and nearly three months later, AWS Context is still listed as coming soon, with no GA date, no regional list, and no pricing. Amazon Bedrock Managed Knowledge Base did go generally available that day, which is most of why the two get conflated.

Microsoft shipped identity-aware retrieval on June 16. AWS announced it on June 17, and you still cannot buy it.

The day before AWS announced Context, Microsoft’sWork IQ API became generally available. It runs in the context of the signed-in user, honors Microsoft 365 permissions, is billable through Copilot Credits, and an administrator can switch it on today. Two announcements one day apart, the same architectural position, and only one of them is something you can put in production.

Databricks reached the same slot from the other direction, extending Unity Catalog to the agent. However,h partners in that ecosystem note that the protection is anchored to the Databricks Runtime rather than to the data, so it stops applying when a BI tool or an MCP server reaches the same source directly.

Teams did not wait for any of this. They shipped the flat-index version while the identity-aware version stayed on the slide.

The direction is consistent, and so is the limit. Each of those controls is strongest inside the system that issues it. The interesting problem begins when an agent needs context that crosses several of those systems at once, and that is the job assembly has to solve.

The lake is not the business

Lake Formation enforces fine-grained permissions inside the lake it governs, and it does that well. Those permissions do not become the sharing rules in Salesforce, Slack, Google Drive, or Confluence.

AWS documents where its own boundaries sit. Its August guidance on propagating user authorization context through AgentCore walks through handing Salesforce a token scoped to the actual user, so Salesforce applies its own sharing rules. In AWS’s words, “the agent acts as an orchestrator, not a gatekeeper,” and “downstream services enforce authorization.”

That is a reasonable call. It is also an important product boundary. Lake Formation is not integrating with Salesforce, GitHub, Jira, Slack, Confluence, or Google Drive. Each of those decides who sees what on its own terms, or nobody does.

The most useful line is about the filter itself. In that same security post, AWS states plainly that “metadata filtering is application-layer enforcement. The bedrock:Retrieve API doesn’t expose metadata filter content as an IAM condition key.” I keep coming back to that sentence because it is a vendor calmly telling you where its guarantee ends and yours begins.

The same is true of your own stack. The tags on your chunks are not an identity boundary. They are a hint that your application code is trusted to honor.

What breaks when authorization arrives too late

The failure is structural, which is why I keep running into the same few versions of it.

I want to be careful here. “Filters are bad” is not the argument. The problem is ordering. A retrieval system can search a mixed index, retrieve opaque IDs, authorize them, and hydrate only the documents the person is allowed to read. That is a filter, and it is fine, because nothing unauthorized ever left the retrieval boundary.

The version I see more often runs the check after the documents have already been fetched. Once restricted text has been hydrated, reranked, summarized, or cached outside that boundary, authorization is chasing the problem instead of preventing it. AWS’s own guidance calls the broad-credential version of this a single point of failure because a prompt injection or a bug in the filtering logic can expose the whole dataset. And if it reached a model, the model has already read something the person was never entitled to retrieve, with any bug or injected instruction in that window free to act on it.

The defense most teams reach for first can make things worse. Jiale Liu, Jiahao Zhang, and Suhang Wang at Penn State red-teamed graph-based retrieval and found that summarization reduces leakage in untargeted attacks but can increase it in targeted attacks. My read of why is that summarizing preserves the salient detail, and the salient detail is usually the sensitive one. A separate 2026 preprint found cross-tenant leakage in pipelines that hand off from vector search to a graph, and eliminated it by re-checking authorization at every hop. Two individually secure components can still compose an insecure system when no one re-checks authorization at the transition between them.

Two individually secure components can still compose an insecure system when no one re-checks authorization at the transition between them.

The seventeen-hour window at the top of this piece is the same failure in slower motion. Direct shares, nested groups, and public links all change independently, which is why Google built Zanzibar as a relationship model rather than a list. A list of allowed users stamped on each chunk is a snapshot of a graph that moved without telling you.

None of this is fringe anymore. The OWASP Top 10 for LLM Applications moved sensitive information disclosure from sixth place to second in its 2025 revision and added LLM08, Vector and Embedding Weaknesses, which names the risk of context leaking between users who share a vector database and recommends a permission-aware store as the fix.

The enterprise-scale version of this is Copilot. In the first year of its enterprise rollout, a 2024 Gartner survey of 132 IT leaders found that oversharing led 40 percent to delay Microsoft 365 Copilot rollouts by 3 months or more. What makes that example useful is that Copilot is not the one doing the wrong thing. Microsoft checks the user’s permissions at query time, and its own documentation says results are trimmed to content the signed-in user has permission to access. Copilot surfaces what those people were already allowed to open.

A surprising amount of enterprise data stays private mainly because it is hard to find, and retrieval is very good at finding things.

The exposure was sitting there the whole time. A surprising amount of enterprise data stays private mainly because it is hard to find, and retrieval is very good at finding things.

Where identity has to arrive

So the lesson from Copilot is that resolving identity at assembly is necessary and not sufficient. Assembly inherits whatever the permission graph actually says. If the graph is wrong, stale, or too broad, the retrieval system will faithfully enforce the wrong answer. Homegrown retrieval can inherit the same problem, often with less governance tooling.

That does not weaken the case for assembly. It locates it. Assembly is not what makes your permissions correct. It is the last place where correct permissions still matter, because after that point the model has read the document.

I am not claiming to have invented this. AWS is arguing a version of it by governing the graph with the permissions the lake already has. OWASP got to the same place from the security side, and its recommended fix for LLM08 is a store that knows who is asking rather than a check that runs after the fact.

The part I would add comes from watching enterprise products make the jump from pilot to production. Teams can postpone many architectural decisions during a demo. They cannot postpone this one for very long. Eventually somebody asks who can see what, who guarantees it, how quickly a permission change propagates, and who owns the answer when three different systems disagree. That is often the moment when an impressive AI pilot turns into a security project, and it usually starts with something like an intern’s question.

So there are four questions I would put to any team building this.

  1. Whether identity gets resolved at assembly or after retrieval.
  2. How much of your context lives outside the lake, in chat and tickets and docs, where IAM does not reach.
  3. What your worst-case staleness window looks like when someone changes teams.
  4. And whether you can re-check authorization at every step along the way, or only once at the door.

If those answers are uncomfortable, that is useful. I have not had many of these conversations where they weren’t.

The post Permissions belong in the assembly context appeared first on The New Stack.

Received — 3 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Want to scale AI agents without breaking anything? Retrieval engineering is the answer.

Abstract metallic circuit board with raised pathways and connection points illuminated in blue, cyan, and pink.

AI agents are multiplying as corporations adopt the technology in record numbers. Smarter underlying models, better tool use, and improved multi-agent collaboration have pushed agents to evolve beyond impressive demos into practical technology that companies marshal in production environments. But the job’s not finished. 

As companies deploy more agents, more often, and against longer tasks, the plumbing that provides their AI ephemera with the required information is buckling.

Here’s the problem: AI agents are sending waves of queries against company data, creating concurrency issues and exposing just how difficult it can be to ensure a company’s AI-legible information is fresh, served only when relevant, and quickly available.

Join the live conversation: On September 24 at 12 p.m. Eastern/9 a.m. Pacific, Whit Walters, Field CTO and Lead Analyst at GigaOm and author of Defeating the Integration Tax report, joins Bonnie Chase, Director of Product Marketing at Vespa.ai, to discuss what happens when retrieval architecture meets that workload.

And crucially, they will explore in this live conversation what changes when a team rebuilds it as a unified layer instead of a fragmented one.

Register for our free event on September 24

REGISTER NOW FOR THIS WEBINAR
By registering, you consent to The New Stack’s Privacy Policy, Terms of Use and to receiving email communication from The New Stack and our event partner. You may opt out at any time.

You might be asking yourself: How has this problem not been solved yet? Google famously handles tens of thousands of search queries every second; how difficult can it be to serve agents the information that they need when we’ve solved the human version of the same problem? It’s no small challenge, and it’s why retrieval engineering is a labor category you’ll hear more about in coming quarters.

So, why is the problem worse with AI? Agents don’t ask a single question. They may retrieve data, reason against it, and then go back for more context. That doesn’t sound too complicated, until we recall that companies often stitch multiple systems together to provide their agents with required information. In practice, that means fusing vector databases, ranking tools, and serving layers into a single hybrid retrieval system that serves ever more agentic queries.

Worse, when several agents ping the same cobbled-together architecture at once, relevance drift becomes a real issue. You might do all the work to get your company or team up and running with agents, only to see the effort fail because of stale data, generic answers, or even truncated results as retrieval plumbing stumbles.

Your AI agents can’t scale successfully if they get dumber the more agents you deploy. So join the conversation on September 24, where we’ll break down how you can solve your retrieval engineering woes.

What you’ll take away:

  • Why agent workloads create a fundamentally different retrieval challenge than added concurrency alone
  • The specific failure modes at agent scale — latency stacking, stale context, relevance drift
  • Why fragmented retrieval stacks amplify those failures
  • What a unified retrieval architecture looks like in practice

The post Want to scale AI agents without breaking anything? Retrieval engineering is the answer. appeared first on The New Stack.

Received — 2 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Multiverse says its 438B model is fast enough for AI agents. The benchmarks tell a more complicated story.

Abstract lines

A 438-billion-parameter reasoning model isn’t an obvious choice when speed is a priority. Multiverse Computing is betting that compression can make it one. 

On Wednesday, the Spanish company launched Quasar 438B, its first large-scale model and a system designed specifically for coding and enterprise agents. Quasar achieves a score of 43 on Artificial Analysis’ Intelligence Index and 69.3 on Terminal-Bench v2.1, while Artificial Analysis currently records its output speed at approximately 183 tokens per second. Multiverse is positioning Quasar as the European model with the highest score on the Intelligence Index, outperforming Mistral Medium 3.5 (30) and NVIDIA Nemotron 3 Ultra (38).

Multiverse is betting that a 438B-parameter model can be fast and inexpensive enough for agents that repeatedly reason, call tools, and check the results.

That puts Quasar in an interesting middle ground. It doesn’t match the strongest models on coding performance, but Multiverse is betting that a 438B-parameter model can be fast and inexpensive enough for agents that repeatedly reason, call tools, and check the results.

The model features a context window of one million tokens, is available in English and Spanish, and can be accessed via the Multiverse CompactifAI API.

Compression claims, missing details 

Multiverse built CompactifAI to shrink large AI models so they need less memory and compute to run. The company says it can reduce model size by 80% to 95% with only a small loss in accuracy, but it hasn’t disclosed how much Quasar was compressed or which model it started with.

In July, Multiverse announced a $570 million Series C to expand its library of compressed models and commercialize the technology. Quasar is the biggest test of that approach so far.

Multiverse hasn’t said what hardware is required to run Quasar or how much the compression reduces its memory and compute needs. That matters for agents, which may repeatedly call the model and other tools before finishing a task.

Multiverse hasn’t said what hardware it takes to run Quasar or how much the compression cuts its memory and compute needs.

Coding benchmarks show tradeoffs 

In Multiverse’s comparison, Quasar’s Terminal-Bench v2.1 score of 69.3 places it ahead of Mistral Medium 3.5 but still well behind the best frontier systems, while Claude Opus 5 achieves the highest score on that benchmark at 89.1.

Multiverse is pitching Quasar for software engineering, technical copilots, research, and workflow automation. Its 1-million-token context window gives agents room to work with large codebases and hold onto information as a task progresses, although processing more context also requires more compute. That can be especially important in coding, where code that passes every test can still trip up the next AI agent if it loses track of what came before.

Agent latency beyond throughput 

Artificial Analysis found that Quasar starts responding in about 1.1 seconds and can produce a 500-token response, including reasoning, in around 15.3 seconds. Those numbers are fast, but an agent also has to wait for tools, process growing context, and make repeated model calls over the course of a task. The agent tooling layer itself is still catching up to what these models need.

Those numbers are fast, but an agent also has to wait for tools, process growing context, and make repeated model calls over the course of a task.

Proprietary model, open questions 

Quasar is proprietary and only available through Multiverse’s API, so developers can’t inspect the weights or run it on their own hardware. For now, that also makes it difficult to know whether the speed Multiverse is reporting will carry over to everyday agent use.

Quasar also arrives as European AI companies are building more of their own model and compute infrastructure instead of relying on U.S. hyperscalers. Multiverse is taking a different route, using compression to make a 400B-plus model cheaper and faster to run. The next step is to see how that holds up against benchmarks.

The post Multiverse says its 438B model is fast enough for AI agents. The benchmarks tell a more complicated story. appeared first on The New Stack.

Received — 1 September 2026 ⏭ AI Infrastructure Archives - The New Stack

Your Mac is now part of Perplexity’s AI infrastructure

abstract

Perplexity wants its AI agents to use more of the computing power already sitting inside a Mac. The company launched Hybrid Compute on Tuesday, a new feature that lets Perplexity Computer move parts of the same task between powerful cloud models and smaller models running directly on Apple silicon.

The timing is significant. Hybrid Compute arrives on the first day of John Ternus’ tenure as Apple’s CEO. Ternus, who previously led Apple’s hardware engineering organization and played a key role in the company’s transition from Intel processors to its own Apple silicon, succeeds Tim Cook after 15 years as CEO. Perplexity is now betting that same hardware can become part of the infrastructure behind autonomous AI agents.

The task starts in the cloud. When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without having to start over.

When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without starting over.

Privacy Gate checks first

A Perplexity-trained Privacy Gate runs locally on the Mac and looks for sensitive information such as names, addresses, account numbers, and secrets. When it flags something, the user can decide whether that part of the task should stay on the device.

Users can review what the system wants to keep local before work begins and ensure it hasn’t missed anything they don’t want sent to the cloud. They can also choose which model handles the local work.

At launch, users can choose between Gemma E4B, Qwen3.6 35B-A3B, and a version of Qwen3.6 35B that Perplexity post-trained itself, with more models planned for later. Perplexity also handles the installation through the desktop app, so users don’t have to open a terminal or set up the model themselves.

Local tokens cost nothing

Once a task is running, the app displays local CPU, GPU, and memory utilization, as well as the number of tokens consumed. Users aren’t charged for tokens generated by models running locally on their Mac. That gives Perplexity another reason to move work onto the device beyond privacy, since running models locally can also cut inference costs. (The question of who bears inference costs and when is becoming a competitive issue across the industry.)

There is a compromise. Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability. Hybrid Compute leaves that choice to the user.

Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability.

Context crosses the boundary

The trickier bit is what happens to context when part of a task moves onto the Mac. The cloud model still needs to know enough about what happened locally to continue the job, without getting access to the private information that was supposed to stay there.

Perplexity says Computer can move a step from the cloud to a local model without restarting the task or losing context, and ultimately combine the cloud and local work into a single result. It does not detail in its announcement exactly what context is passed between those environments or how information produced by the local subagent is filtered before it returns to the broader workflow.

The local subagent can work with private files and data and take actions on the Mac. Users can also start a task on an iPhone and hand off local work to their Mac without having to start over.

For enterprise customers, Perplexity adds company-wide rules for what stays local and a record of what leaves each device, bringing the same governance questions facing AI agents down to the device level.

DGX Spark starts local

Hybrid Compute reverses the approach Perplexity introduced for Nvidia’s DGX Spark last week. DGX Spark starts locally and reaches out to frontier cloud models only with permission, while the Mac version starts in the cloud and moves work onto the device when needed.

In both cases, the agent harness decides where each part of the job runs. That orchestration — deciding which tools and context an agent actually needs — is becoming an increasingly difficult engineering problem as agents gain access to more systems and data.

Apple silicon becomes part of the agent stack

Running this much of an agent locally still requires a fairly powerful Mac. Perplexity recommends at least 32GB of unified memory, Apple silicon, and macOS 15. The feature is available to Pro and Max subscribers as well as enterprise customers.

Those requirements show the limits of local AI at the moment. Smaller models can run on plenty of Macs, but giving an agent enough compute to handle meaningful work still requires relatively high-end hardware.

That will likely change as Macs get better at running larger models. For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.

For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.

The post Your Mac is now part of Perplexity’s AI infrastructure appeared first on The New Stack.

When agents build, deploy, and maintain, persistence becomes the hard problem

Abstract dark purple and green digital glitch landscape symbolizing AI agent database infrastructure and state persistence.

Every application has always needed a place to keep its state. That is not new. What is new is who creates that place, how many get created, and how long they have to survive after everyone has stopped looking.

“Separating durable state from ephemeral compute is the new requirement, and the idle cost trap is only half of it.”

Consider the agents built by Kimi, the AI platform from Moonshot AI. A non-technical user describes an application in plain language. An agent builds the frontend, backend, and database, then deploys them in minutes. The user never thinks about infrastructure. They asked for a tool and received a running one.

Then the agent does something the previous generation of coding assistants never did. It does not hand over code and walk away. It keeps the application running and comes back to maintain it. That last word is where the old assumptions break. Build once and hand off, and state is almost an afterthought. 

Build, deploy, and maintain across tens of millions of applications, most idle most of the time, and persistence stops being a database feature and becomes the central economic problem of the whole system.

I have written before about what happens to a database when its user is an agent rather than a human. Kimi is the clearest production example I have seen, and it surfaces a requirement that the industry has not really been asked to meet: keep state alive cheaply in two very different places at once. The problem is not performance. It is the economics of persistence.

The number that changes the question

When you serve thousands of tenants, the questions are familiar. How fast are queries? How do we fail over? How do we back up? The industry has good answers for all of them. When you serve tens of millions, most created by agents and most idle most of the time, a different question moves to the front, and it is the one nobody planned for: what does it cost to keep something alive that nobody is using right now?

Ask it once, and you are talking about idle databases. Ask it again, one layer up, and you are talking about the agent’s own half-finished work sitting between maintenance sessions. Same question, two surfaces. At a thousand you absorb it. At ten thousand it stings. At tens of millions it is fatal.

“In a human-driven product, what you provision and what people use track each other. In an agent-driven product, they diverge wildly.”

This is the shift agents force. In a human-driven product, what you provision and what people use track each other. In an agent-driven product, they diverge wildly, because agents create far faster and more casually than humans ever did, and most of what they create goes idle immediately.

One principle, two places

There is a single move underneath everything that follows: separate durable state from ephemeral compute. State is the part you must never lose, so it belongs on a shared, cheap, effectively infinite foundation. 

I have argued before that object storage is becoming the new network layer of the data stack, and this is what that looks like, built for agents: durable data at rest costs almost nothing. Compute is the part you should be able to throw away, summoned when work arrives and released when it stops.

Couple the two and you pay to keep compute running for state that is doing nothing. Decouple them, and your cost stops tracking how much you have created and starts tracking how much work is happening. 

Every hard number in the Kimi deployment comes from getting this separation right in the two places an agent-scale product needs it: the databases the agent provisions for its users and the workspace the agent lives in.

The first place: the idle tenant

The instinct most teams start with is the one that has worked for a decade: give each tenant its own database instance. It is clean and easy to reason about. Kimi’s earlier shape was close to this, with single-instance PostgreSQL behind the product. It works beautifully until the tenant count climbs into the tens of thousands, and then the economics invert. The reason is structural. 

A per-tenant instance couples the logical experience of an isolated database to the physical reality of dedicated, always-on compute. The tenant needs the first. The business cannot afford the second at scale. Cost grows with tenants created, not tenants active, and in an agent-driven product those two numbers are worlds apart.

“The agent experiences a dedicated, isolated database, while a virtual layer beneath provides each one its own namespace. The agent does not need a dedicated instance. It needs the experience of one.”

I have started calling this the idle cost trap, because teams walk into it without seeing it, and by the time they feel it, the architecture that created it is load-bearing. The way out is the principle. The agent experiences a dedicated, isolated database, while a virtual layer beneath provides each one its own namespace and isolation guarantees over a shared substrate, with object storage holding durable data and a routing layer sending requests where they belong. The agent does not need a dedicated instance. It needs the experience of one.

There is a catch worth being honest about. If compute is summoned on demand, then when a request arrives for an idle tenant something has to spin up, and if that is slow, you have traded a cost problem for a latency problem. 

Kimi provisions a database in about a second, from a warm pool of pre-initialized resources kept ready and replenished behind each claim, so database setup drops out of the delivery pipeline entirely. That number is not a benchmark to brag about. It is the evidence that ephemeral compute over shared, persistent storage can stand in for the always-on model, because it appears fast enough to feel always-on.

The second place: the agent’s own workspace

The maintenance half of the lifecycle drags the same problem into a place most teams never look: the agent’s own working environment. An agent that builds once can treat its workspace as disposable: spin up a sandbox, write code, ship, discard. An agent that maintains cannot work that way. 

It has to return days or weeks later and pick up where it left off: source code, Git history, checkpoints, the record of what it was in the middle of doing. If that context is lost, the agent does not resume work. It is reconstructing it.

And execution environments are ephemeral on purpose, because keeping millions alive between sessions is the idle cost trap in a different costume. So the environment gets torn down. If the work goes with it, every maintenance session begins by rebuilding state the agent already had: wasted compute, wasted tokens, and a user watching an agent relearn its own project.

The fix is the same separation, one layer up. Kimi uses a persistent filesystem that keeps development state alive independently of the compute that produced it. Source code, Git history, checkpoints, and task progress persist after the environment is destroyed, so the agent resumes rather than starting over. Compute is ephemeral and cheap to discard. State is durable and cheap to keep. Same trade as the idle tenant, seen from the builder’s side.

The database choice becomes a quality input

There is a further effect that surprised me, because it shows up in the quality of what the agent builds. Every infrastructure decision is a chance to get something wrong. If each task forces the agent to reason from scratch about which database to use and how to configure it, it is improvising on every run, and improvisation is where errors enter. 

When the stack is unified, it applies known-good patterns instead. Kimi saw code generation success rates improve from standardizing on a unified data layer, and that is the mechanism: fewer places for the output to go wrong.

“The choice of database is no longer just an infrastructure decision. It is a quality input to the agent’s work.”

This is a version of an argument I keep returning to. Agents need guardrails, and the most effective ones are built into the infrastructure rather than bolted on as instructions. A database that behaves consistently every time is a guardrail. The choice of database is no longer just an infrastructure decision. It is a quality input to the agent’s work.

Four properties that have never been requried together

Step back from Kimi and the shape of the requirement is clear. An agent-scale product needs four things from its data layer at once:

  • Tenant isolation: every agent-created database must be logically separate, so millions of tenants never bleed into one another.
  • Instant provisioning: a new tenant has to exist in about a second, because an agent will not wait and neither will the user behind it.
  • Cost elasticity: an idle tenant, and an idle workspace, has to cost almost nothing, because most things are idle most of the time.
  • Persistent state independent of compute: the tenant’s data and the agent’s development state alike must survive the ephemeral environments that produced them.

Each has been solved before in isolation. Databases have offered isolation for decades. Serverless systems provision quickly. Object storage is cheap at rest. What is new is the demand for all four at once, at a scale where any one failing breaks the product. That is the real infrastructure competition of the agent era, and it is not the one the industry is used to having. It is not about who has the fastest single query. It is about who can hold all four together at tens of millions of tenants without one collapsing the others.

The pattern is bigger than one product

I do not think Kimi is a special case. It is an early, unusually clear instance of a pattern that will repeat across every product where agents build for non-technical users at scale. The shape is always the same. One agent, one workspace, one database, repeated millions of times. Each instance feels independent. 

The infrastructure underneath is shared, and it works only because a virtual layer separates the logical experience of isolation from the physical cost of compute, and because durable state is allowed to outlive the compute that produced it.

Kimi is the version where the agent hands a finished application to an end user and then stays on to maintain it, which is harder in two ways at once. The tenant does not go away when the run ends; it persists, idle, waiting, multiplied by tens of millions. And the agent does not go away either, so its own working state has to survive every gap between sessions. Persistence is no longer a property of one component. It is the property the whole system is organized around.

“The database conversation in the agent era is no longer really about speed. Get it wrong, and no amount of model quality will save the margins.”

Teams building in this direction will hit the idle cost trap on both surfaces, whether they plan for it or not. The only choice is whether they see it coming. Design for all four from the start and you scale past the point where the per-tenant-instance model collapses. Do not, and you hit that ceiling at tens of thousands of tenants, exactly where Kimi’s earlier architecture did, and sooner than you expect, because agents fill a tenant table faster than any human-driven product ever has.

The database conversation in the agent era is no longer really about speed. It is about whether the economics of persistence can survive the scale that agents create in both the database and the workspace. Get that right early, and the rest of the product has room to grow. Get it wrong, and no amount of model quality will save the margins.

If you are building a product where agents provision infrastructure for end users and then stay on to maintain it, four things make that economically viable: isolation, instant provisioning, near-zero idle cost, and durable state on a shared substrate. That is what we have built into TiDB’s serverless and agent workloads. It is the pattern these teams keep arriving at from different directions.

The post When agents build, deploy, and maintain, persistence becomes the hard problem appeared first on The New Stack.

❌