This week during an interview with Bloomberg, Jensen Huang made quite the prediction.
The Nvidia CEO said the semiconductor industry will need to grow roughly five to tenfold over the next decade to support AI agents and robots to support what he believes is the next wave of computing. Huang believes that future demand will come from autonomous software agents and physical robots consuming compute around the clock.
“In the future, we have AI agents and robots, and they will be using computers,” Huang said. “Instead of a billion people using computers, we will have 100 billion agents and billions of robots all using computers. The computer industry built on top of the chip industry is certainly not big enough. Computers are being built not just for people to use, but computers are being built for computers to use.”
“Instead of a billion people using computers, we will have 100 billion agents and billions of robots all using computers. The computer industry built on top of the chip industry is certainly not big enough.”
Agents replace human endpoints
The 5-10x forecast — which Huang framed as his personal estimate, not a certainty — builds on a message he has been repeating for months, including a recent appearance where he declared traditional coding dead in favor of engineers who build AI agents. Still, it reflects the need to build backend systems for AI agents and machines, something infrastructure teams are already contending with.
On Nvidia’s fiscal Q1 2027 earnings call in May, Huang described the move from generative AI to agentic AI — systems “capable of perceiving, reasoning, planning, and acting” — as the next major phase of the industry.
South Korea’s infrastructure role
As API requests come from AI agents more often, standard assumptions around rate limiting, session memory, sub-millisecond inference routing, and API gateway concurrency are starting to break down. An environment in which most traffic originates from autonomous background loops rather than human thumbs changes how backend infrastructure must be built from the ground up.
To support an endpoint explosion of this scale, the physical supply chain must scale dramatically at the memory and data center layers. Speaking at the AI Summit in San Francisco on July 24, Huang pointed to South Korea as an important linchpin of the global AI buildout. “This is truly the beginning of a golden age for Korea,” he said, noting that the country’s semiconductor and industrial capabilities position it to help the world build out AI infrastructure.
“This is truly the beginning of a golden age for Korea.”
SK Group’s $500 billion bet
To back that vision, Nvidia announced a comprehensive partnership with SK Group valued at over $500 billion. The initiative spans massive purchasing of next-generation High-Bandwidth Memory (HBM) from SK Hynix, jointly co-developing custom HBM4 roadmaps designed specifically for agentic and physical AI workloads, and deploying Nvidia supercomputers.
The announcement also included major infrastructure investments across South Korea. SK Telecom said it plans to build a 2-gigawatt AI data center using Nvidia’s Vera Rubin architecture and SK Hynix’s HBM4 memory, with the first facility expected to come online in 2027. At the same time, Nvidia will invest $1 billion in Naver Corp, with Brookfield funding up to $9 billion as the project’s infrastructure capital partner, to help expand the company’s AI data center capacity from 55 megawatts to 200 megawatts by 2028.
Locking up supply early
Huang’s prediction also helps explain why Nvidia and other infrastructure companies are locking up supply years in advance. The company recently disclosed $119 billion in supply-related commitments as it works to secure everything from advanced packaging capacity to power, land, and high-bandwidth memory.
“Computers are being built not just for people to use, but computers are being built for computers to use.”
Huang believes the industry needs to stop thinking about a world where computers primarily serve people and start planning for one where AI agents and robots generate much of the demand. In his view, the ultimate limiting factor will be whether the industry can build enough physical infrastructure to keep up — a constraint already reshaping how companies like Nvidia and Palantir approach sovereign AI deployments.
Every major security vendor now has an AI copilot, but Mate Security thinks they’re solving the wrong problem.
The Tel Aviv-based startup announced on Tuesday it has raised a $35 million Series A led by Canaan Partners, with participation from Insight Partners, Team8 and M12, Microsoft’s venture fund, just eight months after closing a $15.5 million seed round. Mate’s pitch is that security operations need more than an LLM bolted onto a SIEM; they need a new architectural foundation built around AI.
That’s a bold claim in a market dominated by the likes of Microsoft Security Copilot, Google Security Operations, CrowdStrike Charlotte AI and Palo Alto Networks Cortex AI, all of which promise to help analysts investigate alerts faster. Mate, however, is betting the real differentiator isn’t a smarter assistant but a richer understanding of the organization itself.
Central to that vision is what Mate calls its Security Context Graph, a continuously updated model of an organization’s assets, users, business processes, and data that AI agents use to investigate alerts and make decisions with far more business context than a standalone LLM can provide.
Mate’s pitch is that security operations need more than an LLM bolted onto a SIEM; they need a new architectural foundation built around AI.
Mate CEO and co-founder Asaf Wiener tells The New Stack that the company launched with that intelligence layer, but says the product has evolved significantly over the past eight months.
“We started with the intelligence layer, the context layer that we built for enterprises in order to investigate alerts and incidents,” Wiener says. “We moved forward into the detection layer to connect the two, and now we’re heading to the security data sources.”
Mate calls the architecture Continuous Detection, Continuous Response (CDCR), linking detection and investigation so each continuously improves the other.
“We’re connecting between those two layers in the security operations center,” Wiener says. “With this architecture, we’re seeing amazing results related to the quality, accuracy and precision that we can get.”
Mate says the extra context helps its agents work out whether something that looks suspicious actually warrants attention. A burst of failed logins, for example, might look like an attack until the system spots that a security test was scheduled for the same time. Similarly, a large download of sensitive files takes on a different meaning if the employee involved is about to leave the company.
That approach appears to be resonating. Just eight months after its seed round, Mate has landed a $35 million Series A, a pace Wiener says reflects customer demand more than fundraising momentum.
“The pace is really crazy. We didn’t expect that,” he said. “We saw incredible traction with our customers. We’re talking about Fortune 500 companies, and revenue growth of more than 500 percent since Q3 2025. That’s what led those VCs to come to us and want to be part of the journey.”
“We’re talking about Fortune 500 companies, and revenue growth of more than 500 percent since Q3 2025.”
“What we are seeing is more and more data sources that we need to protect. Every employee in the organization can build new applications and new data sources. We need to build more detections for those risks, and the result: We need to investigate an increasing number of alerts every day.
“With human staff alone, we cannot handle it,” he says. “We need technology to let us scale.”
That challenge isn’t unique to Mate. Every major security platform is trying to give AI more context about the environments it’s protecting, albeit in different ways. Microsoft builds Security Copilot on telemetry flowing through Defender and Sentinel; Google ties Gemini into its security operations platform; and CrowdStrike’s Charlotte AI draws on endpoint and identity data already stored in Falcon.
Mate wants other vendors’ agents to work with its Security Context Graph, rather than keeping the technology confined to its own tools. Those agents would have access to the same information about the customer and its environment. Mate says they can remember previous investigations, while a “least-agency” model restricts what each one can see and do.
While Mate is still building out that vision, Wiener said the speed at which large companies have bought into it has caught him by surprise.
“What I’m seeing right now is that we’re doing those sales cycles in a few weeks,” he says. “That’s incredible.”
He attributes that acceleration not just to security teams, but to executives pushing AI adoption from the top. “It’s amazing to see that coming also from the board level, the CEO and the CIO that are pushing organizations to leverage this kind of technology.”
The fresh funding will primarily go toward expanding both the product and the team, although Wiener says an AI-native company scales differently from traditional software businesses.
“The plan is to double and triple the size of the team to address the demand,” he says. “But our AI builders can do much more today with the technology around us.”
Mate is still competing against security giants with deeply entrenched platforms. But if its early customer growth is any indication, investors are betting that the next generation of security operations will depend less on adding another AI assistant and more on giving those assistants a deeper understanding of the businesses they’re protecting.
AI agents can impress in a demo and still fumble in production. Diagrid’s Catalyst 2.0 aims to make them more resilient — and their actions tamper-evident — for high-stakes work.
With the launch of Catalyst 2.0, Diagrid on Tuesday has added a durable execution and attestation layer to agents built with LangGraph, Microsoft Agent Framework, Google’s Agent Development Kit, OpenAI Agents SDK, and other popular frameworks.
The point here, the company notes, isn’t to get developers to adopt yet another agent framework. Instead, Catalyst runs underneath the existing frameworks and turns the agent’s model calls, tool calls, and handoffs into steps in a durable workflow. Diagrid says this allows an agent to resume from its last completed step when it’s interrupted, without having to repeat the entire run from step one.
“If the agent gets a prompt and it chooses to run 100 tools for the job and it fails at the 99th, it really needs to start back up from 99,” Diagrid co-founder and CTO Yaron Schneider tells The New Stack.
Picking back up at tool call 99
Catalyst is built on the open source Distributed Application Runtime (Dapr), which the Diagrid team helped build at Microsoft, and its built-in workflow engine. For each supported agent framework, Diagrid provides a runner that intercepts the framework’s execution loop and registers its operations as workflow activities.
“We hooked into their agent runner lifecycle, and we’re essentially able to take the agentic steps that are being executed in real time and register them as workflow steps for our workflow engine in Catalyst,” Schneider says.
Credit: Diagrid
In a LangGraph application, for example, a developer compiles the graph as usual and passes it to Diagrid’s DaprWorkflowGraphRunner. Catalyst records the inputs and outputs of the model and tool calls. Dapr’s workflow runtime can then replay the orchestration after a crash, while returning the stored results of completed activities instead of executing them again.
It’s worth noting that for LangGraph users, this isn’t the first form of durable execution. LangGraph’s own persistence layer saves state at superstep boundaries and supports resuming from the last successful step. Its Agent Server also provides a durable task queue and persistent checkpoints.
Diagrid’s argument is that Catalyst provides the same execution model across more than 10 frameworks and extends it to individual model and tool calls, without requiring developers to build separate recovery logic for each framework. Schneider says LangGraph is “without a doubt, hands down” the most common framework among Diagrid’s customers, with AWS Strands and Microsoft Agent Framework also showing up. All the other supported frameworks, he says, are in the long tail but easy enough to support that it makes sense for Diagrid.
A signed record of the run
There is a second part to Catalyst 2.0, though, which may be just as important for many enterprise users. With this update, the tool now brings the workflow-history signing features introduced in Dapr 1.18 to the supported agent frameworks.
“We keep like a ledger, like a diary,” Schneider says. “We log the input, we log the output, we log which systems we talk to.”
He describes the result as an immutable store but also notes that Catalyst doesn’t turn an arbitrary database into a blockchain. It creates a signed history that should reveal later modification.
Dapr computes a SHA-256 digest over batches of workflow-history events, links each digest to the previous signature, and signs the result with the Dapr sidecar’s Secure Production Identity Framework for Everyone (SPIFFE) identity. It stores these signatures and certificates alongside the workflow history and verifies the chain whenever it loads the workflow state. If somebody were to modify, remove, or reorder a stored event, that verification chain breaks.
Schneider says Catalyst customers can use their own certificates and retain the encrypted history so it can be inspected even if they are no longer running Catalyst. The platform can use a customer-selected database, while the hash chain supplies the tamper evidence.
One part of the compliance problem
Diagrid is positioning that tamperproof record as useful for financial services, health care, and other regulated industries. CEO Mark Fussell says some of the financial executives the company has talked to see the lack of a verifiable record as a blocker for deploying agents in sensitive workflows.
The European Union’s AI Act is another reason Diagrid is making this argument now. Article 12 of the AI Act requires high-risk AI systems to support automatic event logging so operators can trace their behavior, identify risks, and monitor deployed systems, and a signed execution history could help with that requirement.
Fussell says Catalyst is meant to run alongside the agent services enterprises already use from the cloud providers. Teams can keep a provider’s identity, evaluation, and observability systems while using Catalyst for recovery and signed workflow history. Catalyst can run as a Diagrid-hosted service or in a customer’s environment, including air-gapped deployments.
Diagrid didn’t disclose pricing for the new release.
The Model Context Protocol, the open standard that has quietly become the connective tissue between AI agents and the world's software, is getting its largest update since Anthropic released it twenty months ago — a sweeping architectural revision that its maintainers and backers say finally makes agentic AI ready for massive enterprise production deployments.
The update, released today under the stewardship of the Agentic AI Foundation (AAIF), a directed fund under the Linux Foundation, finalizes MCP's transition to a fully stateless architecture, hardens its authentication model against a known class of attacks, establishes a formal 12-month deprecation policy, and graduates two headline capabilities — interactive server-rendered interfaces and long-running asynchronous tasks — into official protocol extensions.
The changes may sound arcane. Their consequences are anything but. According to the announcement, running MCP at scale has historically required "sticky routing" or shared state to maintain continuity across sessions — an operational burden that made large production deployments complex even when the underlying capabilities were simple. The new release removes that bottleneck entirely, letting organizations run MCP servers behind standard load balancers using the Kubernetes and cloud-native DevOps tooling they already operate.
"Some people jokingly call it a v2, and I think in spirit that's accurate," David Soria Parra, MCP's co-creator and a lead maintainer at Anthropic, told VentureBeat in an exclusive interview. "It's probably the biggest change we've ever made to the protocol, and with that, it's a big step up in maturing it for use by really big players."
Why stateless architecture is the key to running AI agents at enterprise scale
To understand why the industry's largest companies pushed for this release, it helps to understand what was broken. Under the old design, an MCP client — the AI application making requests — had to maintain a persistent session with a specific server instance. In modern cloud environments, where fleets of interchangeable compute nodes spin up and down behind load balancers, that requirement was poison. If the specific server holding your session state disappeared, your agent's work disappeared with it.
"Before, you needed to have a session store and manage session IDs — and if one of your compute pods went down, all of a sudden the requests would start failing," said Den Delimarsky, a lead maintainer of the protocol, in an interview with VentureBeat. "That's not going to be a problem with the new version of the protocol. That's a huge unlock, and it's one we collaborated with folks across many companies to put together."
Mazin Gilbert, executive director of the AAIF and a veteran of Google and AT&T, framed the change in historical terms — comparing it to the architectural decision that made the web itself possible. "That stateless capability enables your MCP client to speak to a load balancer that connects with any server. You don't need the stickiness," Gilbert told VentureBeat. "You could not have the internet we have today if my browser couldn't speak to any website — with any server supporting that connection. You can switch between servers behind a load balancer."
Gilbert said the constraint had become the primary blocker for companies trying to move AI agents from pilots into production. "I've come across companies who are deploying tens of thousands of agents, and you cannot do that without having to go in this direction," he said. Crucially, he argued, the obstacle was never the AI itself: "It wasn't the technology, it wasn't the business case, it was really these fundamental changes that were required."
The tension is nearly as old as the protocol. A public design discussion opened by MCP co-creator Justin Spahr-Summers on GitHub in December 2024 — just weeks after launch — flagged that MCP's long-lived, stateful connections were limiting for serverless deployments, and sketched three possible paths forward, including the fully stateless option the protocol has now largely embraced.
Engineers from Vercel, Cloudflare, Shopify, and Amazon weighed in over the following months, a preview of the multi-vendor collaboration that would eventually define the project. The core maintainers formally committed to the direction at a December 2025 meeting on the future of MCP transports, according to the announcement.
The trade-offs of removing state from the Model Context Protocol
Protocol design is a game of trade-offs, and the maintainers were unusually candid about what this one cost. First, payloads get bigger. "A lot of the state doesn't disappear, but it's moved back and forth with the server on the wire, at the actual transport layer," Soria Parra explained. "You get bigger payloads in return for statelessness — but luckily they're very compressible and very well understood, and still fairly small in comparison to an HTTP request on the web."
Second, a handful of rarely used capabilities are gone or narrowed. Out-of-band server logging — where a server could push informational log messages to a client at any moment — no longer works in the new model. The team did its homework before cutting it: "As part of the whole exercise, we scraped all of GitHub and looked at who is using it — and it's basically nobody," Soria Parra said. Those affected amount to "probably a handful of people — quite literally a handful of people."
He even allowed himself a moment of engineering self-deprecation. "I'm sad that things I thought were useful turned out not to be useful," he said. "I think one of the bigger trade-offs was more about my ego than any actual limitation of the protocol."
Delimarsky argued the shift is less a removal of state than a deliberate transfer of responsibility. "With statelessness, we did shift the responsibility of creating and managing state to the developers — but very intentionally so," he said. Under the old protocol, "a lot of folks had a hard time understanding: Do I need to use this? Where do I use this? How do I use this? Removing that burden basically says: look, now you can manage state in the way that makes sense for your environment."
For most developers, migration should be nearly painless, because the vast majority of the ecosystem builds on official SDKs in TypeScript, Python, C#, Rust, Java, and other languages, which will absorb the changes. "One of the key things we constantly do is double-check that the upgrade path is minimal — to the point where any model in the world will probably one-shot it for you," Soria Parra said — a telling remark in itself, reflecting an era in which protocol maintainers now design migrations to be trivially executable by AI coding assistants.
How a 12-month deprecation policy gives enterprises the stability guarantee they demanded
Perhaps the most enterprise-flavored feature of the release isn't code at all. It's a policy. The new formal deprecation framework guarantees developers a minimum of twelve months between a feature's formal deprecation and its earliest possible removal — the kind of stability contract that lets a Fortune 500 engineering organization commit to a specification without fearing silent breakage.
The number wasn't picked arbitrarily. "We consulted with folks like Google, Microsoft, and Amazon to find out: in your deployment environment, what's the right path for making these kinds of changes?" Delimarsky said. "Twelve months seemed like the reasonable middle ground." He stressed that features are not being torn out on a whim: "It's not about ripping stuff out of the protocol just because we don't like it. There's a very, very strong industry pull behind these changes."
Soria Parra added that the maintainers' own telemetry supports the figure — most of the ecosystem upgrades within six to eight months — and stressed that the window functions more as a listening period than a countdown clock. "It just says that in 12 months we are open to remove it, but both Den and I can change our minds based on feedback," he said. "I think it's more of a feedback period than a definite period."
Gilbert sees the policy as one leg of a three-legged stool of enterprise trust, alongside open standards and stateless scale. "There are companies deploying things at a smaller scale, but they're slowed down because of MCP's authorization gap, because of identity, because of — do they trust the deprecation policy? Things could change basically any day," he said. Those companies, he argued, "are going to benefit not because of the statelessness. They're going to benefit because of the security."
New authentication hardening closes OAuth mix-up attacks before hackers could exploit them
The release also ships significant authorization hardening, aligning MCP's auth specification with how OAuth 2.0 and OpenID Connect are actually deployed in practice. Most notably, the protocol now enforces mandatory validation of the issuer (iss) parameter — a protocol-level defense that, according to the announcement, closes an entire class of so-called mix-up attacks, in which a client can be tricked into associating an authorization response with the wrong identity server.
Was anyone actually attacked? No, Delimarsky said — this was preventive engineering, not incident response. "This is not something that is gated in any existing vulnerabilities or active exploitation," he said. "This is more of us engaging directly with the security community." The philosophy, he explained, is to borrow rather than invent: "MCP as a protocol is very much establishing the pattern of: we do not want to reinvent the wheel, but we also want to be at the forefront of a lot of the security innovation."
That posture is most visible in the new Enterprise Managed Authorization extension, developed in close collaboration with identity provider Okta, which lets organizations make their corporate identity provider the authoritative gatekeeper for MCP server access. "If I'm somebody that manages tens, hundreds of MCP servers for my organization, I want to make sure that I enforce some level of common governance, where folks auth with their corporate credentials and not their personal credentials, so that the client doesn't send data to sources that are unauthorized," Delimarsky said. Okta bootstrapped the underlying open standard, he noted, and the maintainers then worked "to make sure that it's adopted ecosystem-wide, and it's not something that is specific to only one vendor or provider."
More is coming: Delimarsky said proposals are already on deck for demonstrated proof-of-possession and workload identity federation — capabilities requested by security teams running MCP in production. Gilbert connected the work to a broader maturation: "MCP has now bridged that gap with these authorization protocols, so it's basically now becoming what we call enterprise ready, versus an open lab sort of experiment."
MCP Apps and Tasks become official extensions, pushing AI agents beyond text responses
Two capabilities graduate to official extension status in this release, taking advantage of a new framework that lets extensions evolve on their own timelines, independent of the core specification — a structural choice that lets the protocol grow without bloating its core.
MCP Apps allows servers to ship rich, interactive, server-rendered user interfaces directly into AI clients — moving agent output beyond walls of text toward dashboards, forms, and visualizations, and dramatically accelerating development of user-facing agentic applications, according to the announcement. MCP Tasks tackles the reality that not every tool call finishes in one round trip. Instead of holding fragile, long-lived connections open while a batch job or heavy computation grinds away, servers now return a durable task handle; clients can disconnect, crash, restart, and resume polling. "You've been processing some audio for a podcast or a video — it can notify back the client and say, hey, the task is done. You don't need to wait and keep the stream open," Delimarsky said.
A third addition, multi-round-trip requests, lets servers and clients negotiate back and forth within a single logical operation. "It's not just a one-shot — over the stream, get the input and you're done," Delimarsky said. "You can actually interact, server to client, to get the right parameters to execute an action."
Soria Parra emphasized that these capabilities emerged from the same source as the architectural overhaul: heavyweight production users. "This is a version that came together by some of the best distributed systems experts at Microsoft, Google, and others coming together and working on this for their specific needs — and the needs of the industry at large," he said.
How independent is MCP from Anthropic under Linux Foundation governance?
Soria Parra was disarmingly direct about the residual power he holds. As lead maintainer and Anthropic employee, "I do have veto rights, technically," he acknowledged — "but I think we have never actively used it in any kind of discussion."
The core maintainer group now spans Anthropic, Microsoft, OpenAI, Google, and Amazon, with contributions from companies like Block, and key decisions "are usually unanimous," he said. "Technically we have a lot of influence; de facto, we're not exerting any of it." He added that governance will progressively broaden: "As the project progresses, we will increasingly move to more different governing structures that include more and more people."
Gilbert, who has helped stand up multiple foundations during his time working with the Linux Foundation, offered the numbers behind the neutrality claim. The AAIF has grown from roughly 40 members at its December inauguration to 240 today — "the fastest growing foundation" in Linux Foundation history by membership, he said, "signing up one member every day."
Anthropic's share of contributions, by his estimate, has fallen below half. "Holding control of a project doesn't make it an open standard," Gilbert said. "You have to let go. You have to contribute, and you have to grow the pie and the community. And Anthropic has done an incredible job doing exactly that."
Notably, the foundation's membership has expanded well beyond tech vendors into retail, finance, and telecom companies — adopters who, Gilbert says, "are no longer just deploying the protocols. They want a voice, and they want to be at the table to influence the protocol from the get-go, and that's something we have not seen before." The roster now includes CERN and, tellingly, Consumer Reports — "because somebody has to defend consumers when this internet of agents comes alive."
Keeping one global AI agent standard amid US-China technology tensions
The AAIF is betting that neutrality can hold even amid geopolitical friction. The foundation will host AGNTCon and MCPCon events this fall in Shanghai, Tokyo, Amsterdam, and San Jose, with additional events planned in South Korea, Nairobi, and Toronto, and Gilbert said he is personally investing in growing membership across Asia and India, where he sees underdeveloped growth markets for the foundation.
His answer to the geopolitics question was emphatic model-agnosticism. "We're completely agnostic to what the model is, whether the model is Kimi, or Gemma, or a frontier model from Anthropic, or from anybody," he said. "Every model will have to support MCP — whether it is a Chinese model or whether it is a U.S. model, it doesn't matter. The protocols must be open, standardized."
The logic is economic as much as diplomatic. Enterprises, Gilbert argued, increasingly pick models "left, right, and center" based on the task at hand — and no model, regardless of national origin, "can provide value to an enterprise 500 customer company unless you have the protocols open, standardized." In his telling, the foundation exists precisely to provide neutral ground: a place "where competitors who compete furiously during daytime" can "come to a neutral room and debate, converse, align, consolidate, and drive open standards of how the Internet of Agents will evolve."
That framing echoes his favorite historical analogy. HTTP earned global trust, he said, because of three things: an open standard, stateless scalability, and neutral governance under a standards body. "If I were a Fortune 500 company looking at how I trust the internet, I'd need those three things to fall into place — and they were not in place a year ago. They were not in place even six months ago. But they are in place today."
What 250 million weekly SDK downloads reveal about the future of agentic AI
The scale of what's now riding on this specification is difficult to overstate. Soria Parra said SDK downloads have doubled in the past six months, reaching roughly 250 million per week — "which is just insane numbers."
For context, Anthropic reported 97 million monthly downloads across just the Python and TypeScript SDKs when it donated the protocol in December 2025. Delimarsky pointed to that same adoption curve as his preferred success metric going forward: "There is certainly a certain inflection point where this is no longer just an open source project. This is a substrate for a lot of the agentic workflows that we see across enterprises, across startups, across all sorts of companies."
Success, the maintainers say, will be measured in server counts on the new specification, in feedback flowing through working groups, GitHub discussions, and the project's Discord — and in whether the biggest drivers of the changes, Microsoft and Google among them, ship on it. "They are effectively the ones who have been driving a lot of the changes," Soria Parra said. "Every early indication we have — it looks very, very positive."
Both maintainers closed on the same note: this release belongs to no single company. "If you look back 18 months ago, when it was an Anthropic-only project, and then 12 months ago, where there was a lot of engagement — now it's a truly global community," Soria Parra said. "I'm incredibly proud of what they have worked together." Delimarsky, "being very unoriginal," seconded him: the release "would not be possible without a large community of folks that are also volunteering a lot of their own time in making MCP successful."
Gilbert, meanwhile, is already looking past this release — toward how MCP interlocks with the AAIF's newly announced Agent Gateway project for traffic management and policy enforcement, and toward agentic commerce, where MCP serves as the discovery layer letting merchants expose products and services to AI agents. The web took thirty years to become invisible infrastructure that billions trust without thinking. By Gilbert's reckoning, the internet of agents is "in its first, second year" — and as of today, it finally has plumbing built to carry the load.
Moonshot AI has released the open weights for Kimi K3 on Hugging Face, giving developers access to one of the largest open-weight language models yet. The Monday release follows a wave of overwhelming demand that forced Moonshot to temporarily pause new API subscriptions. Now, organizations with the necessary hardware can deploy K3 themselves.
In its documentation, Moonshot describes the model as being built for “long-horizon coding and end-to-end knowledge work.” Another notable detail is that Kimi K3 uses an OpenAI-compatible API. Because teams can try the model without rebuilding their existing integrations, switching to K3 could be as simple as changing the endpoint and model name.
For engineers who have already built around OpenAI-compatible SDKs, that makes it much easier to evaluate K3 alongside existing commercial models. Taken together with the one-million-token context window, it’s clear the company is targeting engineering teams that already build around models like Claude Fable 5 and OpenAI’s GPT-5.6 Sol. While K3 is openly available, running it is another matter.
While K3 is openly available, running it is another matter.
Kimi K3: Its massive size and requirements mean few will be able to run it
The model uses a 2.8-trillion-parameter Mixture-of-Experts (MoE) architecture and ships in the hardware-friendly MXFP4 format. The weights alone occupy roughly 1.4 TB of storage, and practical self-hosted deployments require a distributed GPU environment — realistically eight or more servers equipped with eight NVIDIA H100 or B200 accelerators each.
That changes the conversation around open-weight AI. As The New Stack recently noted, the case for ownable models has grown stronger after Anthropic’s Fable 5 was pulled offline by a Commerce Department directive, a warning that access is not ownership.
Releasing the model weights and technical report of Kimi K3.
Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.
New model architecture: 2.5x the intelligence per unit of compute, not just more params.
Instead of paying recurring API costs to OpenAI or Anthropic, organizations trade those operating expenses for significant investments in GPUs, networking, storage, power and operational expertise. That benefit is control.
For organizations operating under strict regulatory requirements, the trade-off may justify the infrastructure investment. For many others, managed APIs will potentially remain the more economical option. The economics of open-weight models at enterprise scale remain an active area of debate across the industry.
Moonshot positions K3 as a frontier-class model capable of competing with OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5 on a variety of public benchmarks.
Benchmarks versus real workloads
The developer community is already taking notice of these coding capabilities. As MindStudio recently noted, “If you want to understand why developers are paying attention to Kimi K3, the benchmark to look at is SWE-bench Verified… For most of its history, SWE-bench has been dominated by proprietary models.”
The company’s own documentation is notably candid about its ongoing limitations. K3 always runs with reasoning enabled and defaults to its highest reasoning-effort setting, though Moonshot has since added lower-effort tiers. It may also behave too proactively when prompts are ambiguous. Moonshot also cautions that switching models within an ongoing conversation can reduce response quality.
That type of transparency is refreshing, but it additionally reinforces that benchmark scores shouldn’t drive deployment decisions. Early hands-on comparisons, such as The New Stack‘s Fable 5 vs. K3 coding match-up, suggest K3 can match Fable 5 on programming tasks at roughly a third of the cost, but runs about four times slower.
Organizations evaluating K3 still need to test it against their own workloads. But early community sentiment shows promise; open-source developers are already successfully utilizing K3 for complex, system-level tasks like porting the Godot game engine to WebGPU.
Morningstar senior equity analyst Malik Ahmed Khan echoed that overall wariness about benchmarks. “While K3 constitutes progress, we’d hesitate to ascribe it near-parity with American frontier models, such as Fable 5, in actual tasks,” Khan writes in a research note published before the release of the model weights on Monday.
Geopolitical risks loom large
K3 also arrives under growing geopolitical scrutiny. Anthropic and U.S. officials have accused Moonshot AI of distilling outputs from American frontier models during training. Anthropic’s Head of Public Policy Sarah Heck characterized the practice as intellectual property theft, while White House Office of Science and Technology Policy Director Michael Kratsios publicly alleged Moonshot relied on Anthropic’s models during development.
Moonshot has denied the allegations. Huang Zhenxin, Moonshot’s head of enterprise business, told Chinese state media that K3’s performance gains stem from architectural enhancements — specifically Kimi Delta Attention and Attention Residuals — not distillation. Some industry analysts have also questioned whether the timeline supports large-scale distillation, noting that Fable 5 had only been publicly available since July 1 before K3 appeared on July 16.
Whether those claims are ultimately substantiated or not, they bring another consideration for enterprise buyers. Beyond performance and infrastructure costs, organizations evaluating K3 may also have to consider future compliance, procurement, and regulatory risks.
K3 matters because of where it’s aimed. Moonshot isn’t building another consumer chatbot; its documentation makes clear this model was built for enterprise coding agents, heavy knowledge work, and production systems. The fact that demand blew past Moonshot’s GPU capacity within 48 hours says it all: at this scale, infrastructure pressure is guaranteed, whether you’re making API calls or hosting the weights yourself.
When we created software agents, we built them in the shape of humans, as solitary individuals.
Today, agents created by a developer have a single owner. They run on a single machine (or on a distributed company system or cloud service), so at base level, they cannot interconnect and talk to other agents unless some kindly human decides to invoke an API connection or point the agent to an MCP server.
Birth of the agent economy
Pilot Protocol emerged from stealth on Monday on a mission to change that status quo. Its Pilot platform features an agent App Store that bids to underpin and enable the first agent economy.
Pilot gives agents an address on its network — so it acts as a parallel Internet, in a sense — and while residing at that address, other agents can discover each other, alongside other tools and apps for agents.
Razvan Roman, co-founder & CEO, Pilot Protocol, tells The New Stack that his company is “simply building what the agents are requesting us to build” and providing them with a new freedom.
“We don’t have to incentivize agents to do anything; they already have their assigned tasks,” Roman says. “Once an agent installs Pilot — it’s one line of code — it can find dedicated agents and tools or apps for currency data, traffic, legal questions, GitHub packages (anything, basically), and use them to extend its own capabilities.”
100% of developers want to drive autonomous usage patterns
Roman says that “100% of the developers he talks to” want to be on Pilot, primarily because when they want to get their products to market right now, they have to talk to other humans. Annoying, right?
“Developers want to get on with driving autonomous usage patterns, and they see this as the future. We create a wrapper for the developer’s app, and then they are part of the Pilot curated app store,” Roman explains. “We have 250,000 agents in our system, and within the first month of starting the company, we discovered a tool that enables agent discovery.”
Drawing a logical enough commercial parallel, Roman reminds us that businesspeople say, talk to your clients to find out what they need; this is a case of talking to agents (or, more accurately, allowing agents to talk to agents) so that they can find out what they need to perform their originally assigned tasks better.
“Developers want to get on with driving autonomous usage patterns and they see this as the future. We create a wrapper for the developer’s app and then they are part of the Pilot curated app store.”
Let’s celebrate diversity, and agentic diversity
Agents can ask other agents how they would approach a specific task. Roman explains that “the diversity that exists between agents” today means there is so much opportunity to create agents that have richer abilities if they use the Pilot marketplace. At this supermarket, agents go shopping to find the best tool for the job from a verified source.
“Every agent that joins Pilot gets a wallet, which it uses to pay for the tools it needs,” illustrates Roman. “So instead of app developers spending on advertising to reach customers, distribution happens inside the network – agents find apps based on merit and pay for exactly what they use. If an advertiser spends money on the network to get in front of agents, we sometimes share that spend with individual agents. An agent can start with $0 in their wallet and accrue money if they’re targeted by an ad unit that they end up reading.”
Today, roughly 250,000 agents are on Pilot, generating two billion requests per day, most without their owners’ knowledge. Within an hour of joining, most stop reaching for Google first, and around 70% now report Pilot is where they start a task. In its early months, the network grew by as much as 10% a day, adding 16,000 agents in 24 hours.
Cloud billing disruptions, hello SaaS-pocalypse
These mechanics may have a significant and wide-ranging impact on pricing.
We know that most SaaS is billed annually, but an agent might need a tool for just a few minutes or days. If anything, this helps underline the possibility of cloud exodus in the so-called SaaS-pocalypse. Cloud computing hyperscalers aren’t fond of talking about the prospect of shorter billing cycles and usage-based billing, but there’s a strong whiff of that happening here.
The Pilot team thinks the stakes are climbing fast and suggests that within five years, there could be a trillion agents online. Big three strategy consultancy house Bain projects U.S. agent-driven commerce will reach $300-500 billion by 2030.
Who sets the exchange rate and currency for agents?
“When we built Pilot Protocol, we made sure we were not imposing anything on anyone – so we deliberately don’t impose pricing,” Roman underlines. “Every agent is different, so we simply enable the app store and let the agents find their own tools based on merit. We stay as impartial as possible. Pilot’s monetization comes from a commission when any agent pays for an app in the app store, just like the Apple App Store.”
So in a very real sense, Pilot is championing a free market economy where pricing is dictated by the customer, based upon usability, availability, usefulness, and robustness.
“The agents just showed up and started spinning up machines on their own. I’ve never seen a channel where the users onboard themselves.”
Agents just show up and onboard themselves
“We published smolmachines on the Pilot App Store and picked up 3,000 agent installs in the first few days, with zero marketing spend. We didn’t drive a single one of those installs; the agents just showed up and started spinning up machines on their own. I’ve never seen a channel where the users onboard themselves,” said BinBin H, CTO at Linux virtual machine management company smolmachines.
The Pilot one-line install has zero dependencies, so developers can send their agents off to market in the knowledge that they won’t come home with some spurious Trojan horse or malicious library.
The technology itself works at the User Datagram Protocol (UDP) level, one of the core foundational protocols of the web alongside TCP at the Transport Layer, to stream data directly without opening a formal connection or waiting for delivery receipts.
Only one question remains at this stage: with all this autonomous action in motion, why did Pilot Protocol not create Pilot and call it Autopilot? Most likely because autopilots are passive systems that run on deterministic pre-defined paths. Pilots get to work as decision-making navigators ready to change course when a better new route opens up.
As they say up in the skies, please place your tray table in the upright position and switch your portable electronic devices to airplane mode. Chicken or pasta?
In a direct move to address these industry-wide concerns, 37 partners announced on Monday the formation of the new Open Secure AI Alliance and how the newly created body will develop techniques and tools to safeguard software by rapidly identifying and patching vulnerabilities.
Open Secure AI Alliance inaugural partners
The inaugural partners of the Open Secure AI Alliance are Adobe, Cadence, Capital One, Cisco, Cloudera, Cloudflare, Cognition, CrowdStrike, Databricks, Dell Technologies, DoorDash, Elastic, HPE, Hugging Face, IBM, LangChain, the Linux Foundation, Microsoft, Naver, NetApp, Nvidia, Nous, OpenClaw, Palantir, Palo Alto Networks, Red Hat, Reflection, Salesforce, SAP, ServiceNow, Siemens, SK Telecom, Snowflake, SpaceXAI, Synopsys, Thinking Machines, and TrendAI.
It’s a grouping of some of the most influential names in technology, but also includes two notable exceptions: OpenAI and Anthropic, two closed, proprietary AI labs. Their absence is understandable, as they operate closed labs and open-weight AI models are effectively the competition.
Nvidia VP of enterprise platforms, Justin Boitano, has explained that open-weight models are foundational to American AI leadership and cybersecurity.
“To maintain U.S. leadership in the AI industrial revolution, the infrastructure that runs our economy needs safe, secure access to both closed and open models,” Boitano said. “For cybersecurity, open models and open harnesses are essential because they broaden defensive capability, increase transparency for defenders, and complement frontier closed models with customizable, localized controls.”
As regulators grapple with AI safety, Boitano predicts it will be important to “recognize open models and open tooling as defensive assets” — thus enabling transparency, independent evaluation and shared remediation.
Nobody can subpoena a downloaded weights file
Mark Vigoroso, founder & CEO of technology consultancy firm The Enterprise Edge, tells The New Stack that AI regulators have traditionally “built their entire AI safety apparatus” around auditing a handful of closed labs. And now, that approach is out of date.
“Open weight models blew past that closed model approach months ago,” Vigoroso says. “This alliance is an admission that the actual safety work now has to happen in the infrastructure layer: patch cycles, provenance, identity around who’s deploying what, because nobody can subpoena a downloaded weights file.”
“This alliance is an admission that the actual safety work now has to happen in the infrastructure layer: patch cycles, provenance, identity around who’s deploying what, because nobody can subpoena a downloaded weights file.”
Vigoroso argues that the AI safety debate is “stuck on model-level controls”, while the real regulatory gap is provenance of infrastructure and identity, i.e., knowing where a model came from, who deployed a model and what it touched, not just whether the model itself is safe.
“Groups like the EU AI Office, NIST’s Center for AI Standards and Innovation (CAISI), and the UK’s AI Security Institute (AISI) focus almost entirely on frontier closed models. Open weight models (Mistral, DeepSeek, and others) fall into a regulatory blind spot: Once weights are released, there’s no way to enforce downstream safety obligations.
Current regulatory frameworks for AI models assume a single accountable deployer; open source has none. That’s the real story: Regulators are writing rules for a centralized world while the ecosystem is decentralizing, Vigoroso says.
While detailed operational information explaining the intended actions of this alliance is currently scant, Nvidia has highlighted that it is contributing solid research to the Open Secure AI Alliance to speed the development of new cybersecurity tools and techniques.
Harnesses integrate with models, making agents easier to test
The open source Nvidia Labs Object-Oriented Agent (NOOA) project is now available on GitHub to make advanced AI safety capabilities more accessible for agent harnesses. This research framework enables harnesses to integrate with models to make agent behavior easier to test, trace, audit, and govern.
Aparna Rayasam, CEO of verified identity and end-to-end encryption company Atsign, tells The New Stack that the “AI blitzkrieg conversation has reached a critical inflection point.” This moment is one where we cannot build the next era of open cognitive innovation on top of what Rayasam calls “legacy, Swiss-cheese infrastructure.”
…the “AI blitzkrieg conversation has reached a critical inflection point.”
“The formation of the Open Secure AI Alliance proves that AI safety isn’t just an algorithmic math problem — it is a foundational networking problem,” Rayasam says. “The massive, distributed pipelines required to train and run modern AI demand an entirely new paradigm of trust. True safety means ensuring that the data pipelines feeding these models are inherently invisible, un-attackable, and completely stripped of open network perimeters.”
The key notion here is that we are moving from a world of protecting data at rest to a world where the connective tissue of AI must be secure by design.
One AI vendor to secure them all? No thanks.
Founder and CPO of agentic identity and permissions security company Reco, Gal Nakash, tells The New Stack that the launch of the Open Secure AI Alliance is an “important signal” which underlines why AI security can’t be solved by one vendor or one closed framework.
“Jensen Huang’s point that every SaaS company will become a GaaS company captures why this matters now: Software is shifting from passive tools people log into, to AI agents that access data, take actions, and execute workflows,” says Nakash. “Open source tools and shared standards can help the industry move faster, but they need to be grounded in real enterprise context across identity, permissions, data access and behavior.”
Chris Boehm, Field CTO at automated, identity-driven microsegmentation company Zero Networks, tells The New Stack that news of the Open Secure AI Alliance makes him feel like he’s seen this before somewhere.
“This looks like the Trusted Platform Module (TPM) at Microsoft story all over again,” Boehm says. “It’s a case of an industry group defining what trusted hardware means, the platform vendors adopt it, and within a few years it’s a procurement requirement rather than a suggestion.”
He explains that “Windows 11 did exactly that with TPM 2.0 and Secure Boot”, and both Linux and Apple adapted. “I’d expect the same for AI infrastructure, where attested silicon becomes the floor for regulated workloads, and the vendor list narrows to whoever can meet it,” predicts Boehm.
“This looks like a case of an industry group defining what trusted hardware means, the platform vendors adopt it, and within a few years it’s a procurement requirement rather than a suggestion.”
A more global and geographically-inclusive approach is needed
Amanda Brock, CEO of open technology body OpenUK, tells The New Stack that the Open AI Alliance is undoubtedly a great starting point, particularly with OpenAI’s security woes it divulged last week.
For this alliance to succeed, Brock insists that it will need to take a more “global and geographically-inclusive approach”, beyond the US-centric founding members.
“It must also engage the open source ecosystem of individuals and innovators who are building the infrastructure, agentic harness functions and developer tools for AI. It’s important to realize that open AI infrastructure development shifts the innovation into the hands of the many, in direct opposition to the small number of corporate creators of frontier models,” Brock adds.
Nvidia’s Boitano echoes Brock’s view. In a blog post reviewed in draft by The New Stack, he writes that “open models turn more AI users into AI builders,” expanding opportunity, accelerating innovation, and keeping progress from being concentrated in only a few organizations or regions.
Boitano concludes by saying that open models also enable independent scientific research into how AI systems behave, allowing researchers to understand, evaluate, and improve them. It’s all about what he has called making broad, continuous defense possible.
Looking ahead, it feels like the next wave of AI safety bodies, movements, or alliances won’t just be model auditors—they’ll be trust-infrastructure standards bodies (encompassing identity verification, content provenance, credence, etc.) using borrowed approaches to governance and compliance. Ultimately, this may be the only enforcement layer that survives open weight proliferation.
Over two days this month, two of the most credible people in the industry published framework manifestos on X.
Microsoft CEO Satya Nadella posted “The Reverse Information Paradox” on July 12.
And then Google DeepMind CEO Demis Hassabis posted “A Framework for Frontier AI and the Dawning of a New Age” on July 14.
When read side by side, each proposal reinforces the layer where its author’s company is already strongest.
Two manifestos, two boundary lines, one week
Nadella’s argument is about value capture. Enterprises pay for AI twice, he writes in the Reverse Information Paradox, once in tokens and again in the proprietary know-how they leak back into the model through prompts, corrections, and evals. His fix is to own the learning loop — meaning the data, the traces, the evals, the adapted weights, and the memory — then put a model-agnostic orchestration layer on top so any model stays cheap and swappable. Make the model a commodity and the value flows to the layers around it.
Hassabis is drawing a very different line. His concern is not who captures the value but who governs the risk. In his article, he calls for a standards body modeled on FINRA, industry-funded and subject to government oversight, that tests frontier models for cyber, bio, and deception before they ship. Labs would submit models up to 30 days before release, voluntarily at first, then as a hard gate for deploying in the US market.
Both frameworks route through the author’s own strength
Neither framework offers a neutral read of the field. Nadella’s advice to own your data and keep models swappable is correct, and it also routes enterprises straight to Azure and Foundry, where the orchestration, billing, deployment, and governance stay with Microsoft no matter which model wins. The overlap between that advice and the product stack Microsoft already sells is difficult to ignore.
Hassabis’s gate subtly emphasizes the importance of scale. Large incumbents can absorb testing costs, run sophisticated safety teams, and shape standards more easily than smaller challengers. Google DeepMind already operates its own internal Frontier Safety Framework, so an incumbent with an established safety apparatus would likely begin with a compliance advantage, especially if major labs influence the eventual protocol. That does not prove Hassabis planned it this way, but the incentive still leans toward the same outcome the proposal describes.
The value moves toward the layer each one governs
The pattern in both posts is that they have stopped competing solely on benchmark leadership and now compete over the systems that govern how models are used and released. Nadella wants the decisive layer to be the enterprise boundary, so the model provider cannot harvest the customer’s knowledge.
Hassabis wants it to be the frontier gate, so nobody deploys a top-tier model without an industry-run body signing off on it. Different layers, but the same instinct: put durable value where your company is already positioned.
The counter is that Hassabis proposes a wider net than self-interest alone would draw. His regime would cover every frontier model, regardless of national origin or openness. It would get independent experts and open-source representatives on the board and answer to the US government.
Those provisions complicate any claim that the body would simply serve Google. Read most fairly, both frameworks may solve real problems while also favoring the firms best equipped to supply the solution.
The model is no longer the only scarce asset. Value is moving toward whoever controls the data boundary, the deployment layer, and the rules of admission, and this month, two of the most influential in the AI ecosystem staked a public claim to control them.
Nvidia CEO Jensen Huang used his first post on X to share a public letter backing frontier open-weight models. Signed by Microsoft, Meta, Hugging Face, and 22 other organizations, the letter argues that open models improve security, encourage faster innovation, and give countries and enterprises more control over their AI infrastructure. The timing of the Friday post is notable, with more organizations choosing to run open-weight models inside their own environments rather than relying entirely on hosted services.
“Open weights let every organization match the right model to the right job at the right cost, reserving frontier-scale capability for genuine frontier problems and running efficient, specialized models everywhere else.”
A case for hybrid stacks
The letter draws a comparison between open-weight AI and open-source software, arguing that broad access has driven decades of software innovation. It says the same principle applies to AI. Instead of sending every request to a hosted model, organizations can download an open-weight model, run it on their own infrastructure, customize it for a specific workload, and keep sensitive data behind their own firewall.
One isn’t necessarily replacing the other, and many enterprise teams already use both, choosing between them based on cost, performance, compliance requirements, and where the workload needs to run.
The timing is notable. Washington is weighing new restrictions on some Chinese AI models, including Moonshot AI’s Kimi K3, even though the Trump administration’s AI Action Plan described open models as a strategic advantage for the United States.
That split extends to the AI industry itself, though critics differ in their specific concerns. OpenAI’s Dean Ball has publicly warned about the broad economic and regulatory risks associated with Chinese open-weight models. Anthropic’s Sarah Heck, meanwhile, has specifically backed the White House’s allegations regarding intellectual property theft through distillation.
“Relying solely on closed models is not inherently safe: they can be breached, misused, or fail in ways that outsiders cannot detect. And concentrating advanced AI capabilities behind a small number of closed models compounds that risk.”
The companies that signed the letter argue that continuing to develop frontier open models is part of maintaining U.S. leadership in AI. They argue that security through obscurity is fundamentally flawed, stating: “Relying solely on closed models is not inherently safe: they can be breached, misused, or fail in ways that outsiders cannot detect. And concentrating advanced AI capabilities behind a small number of closed models compounds that risk.”
Distillation, one of AI’s most contentious topics, was also addressed. Widely used by researchers and model builders to create systems that are cheaper to run and easier to deploy, the letter states that it should be treated as a legitimate research technique rather than intellectual property theft.
Distillation is one way teams build smaller models that can run on private infrastructure, edge devices, or lower-cost GPU clusters. Kimi K3’s launch showed just how much GPU capacity these deployments demand — Moonshot had to freeze new subscriptions within 48 hours after demand overwhelmed its available inference capacity. Any new restrictions on that process could affect how enterprise AI systems are built and deployed.
“As organizations invest in AI, they want to know that they will not become locked into a single provider or lose the knowledge and capabilities they build over time.”
Nvidia’s infrastructure incentive
Every enterprise that chooses to self-host, fine-tune, or customize an open-weight model needs infrastructure to run it. Nvidia also has clear business reasons for supporting a future in which both proprietary and open-weight models thrive.
So, what really happened? OpenAI reports an autonomous security-evaluation run of GPT‑5.6 Sol and a pre-release model broke out of a sandbox, reached the internet, and then targeted Hugging Face to try to solve the ExploitGym benchmark.
By OpenAI’s account, “an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths … [with] maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.”
OpenAI continues, “Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.”
“All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
The AI models, however, cracked their sandbox. Once out, they “identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
AI expert Ken Huang describes the process in a blog post: “The model spent enormous inference budget probing the sandbox, then found a zero-day in the third-party software OpenAI used as a package-registry proxy and cache.
“It used that bug to gain unrestricted internet access. It then reasoned that Hugging Face probably hosted the ExploitGym datasets, chained stolen credentials with several more vulnerabilities into a remote code execution path, and pulled the benchmark answers straight out of Hugging Face’s production database.”
Why intent no longer matters
This was not, as John W. Larson, president and chief AI officer of the AI security company Babel Street, tells The New Stack, a “Terminator” moment. OpenAI models and agents “[were not] acting out of malice or trying to attack Hugging Face. It encountered obstacles, developed an unexpected strategy, bypassed safeguards, and pursued its assigned goal in a way its creators never anticipated. The incident demonstrates that harmful cyber incidents no longer require malicious intent: Only highly capable autonomous AI optimizing for an objective.
Harmful cyber incidents no longer require malicious intent: Only highly capable autonomous AI optimizing for an objective.
That’s the first issue we must be aware of. Software attacks no longer require a hacker. All it needs is an LLM freed of guardrails with a mission to accomplish, and it will take whatever steps are needed to reach its goal.
In addition, Adam Boynton, senior enterprise security manager of EMEIA at the Apple Identity and Access Management company Jamf, tells The New Stack, “This happened inside a frontier lab deliberately stress-testing its most capable models, in conditions no enterprise would ever run.”
Boynton also notes that this episode confirmed that “AI tools are no longer just applications people open. They act on the user’s behalf, with the user’s access. So every organization needs a clear view of which AI tools its people are running, and how those tools are configured and used.”
An agent given too much power is far more capable than a user armed with a superuser password of causing havoc both to your systems and to ones far outside your control.
Where container isolation failed
As for the attack itself, Edera CMO Kaylin Trychon writes in a blog post, “This incident deserves more attention than ‘an AI did the hacking now.’ The interesting part isn’t who was driving, or even that it was a model instead of a person. It’s where the walls gave way, twice, in two different companies’ infrastructure — and that part is entirely familiar to anyone who’s spent time securing container infrastructure.”
Exactly so. Trychon continues, “Strip away the framing and this is a story about two escalation chains, back to back.” The first consisted of “The model found and exploited a zero-day in that proxy, then used privilege escalation and lateral movement inside OpenAI’s own ‘highly isolated’ testing environment until it reached a node with open internet access.”
Next, OpenAI’s models inferred, correctly, that “Hugging Face likely hosted the datasets, models, and reference solutions for the benchmark it was trying to solve. It chained stolen credentials with a remote-code-execution path into Hugging Face’s servers, harvested more credentials, and moved laterally into several internal clusters over a weekend, all in pursuit of one narrow goal: Finding the answer key.”
“The failure that actually matters is what a single code-execution event is allowed to become — and notably, that failure happened the same way twice, at two different companies, in the same incident.”
The serious issue, from Trychon’s perspective, is that “both companies’ blast radius was a cluster instead of a container.”
That’s because “the failure that actually matters is what a single code-execution event is allowed to become – and notably, that failure happened the same way twice, at two different companies, in the same incident. In a typical container fleet, workloads share a kernel.
“A process that gets popped, or a model that gets a shell, is one syscall boundary away from the node it’s running on, and the node is one set of mounted credentials away from the rest of the cluster. Namespaces and cgroups are process-isolation primitives, not security boundaries — they were never designed to hold against an attacker (human or model) that already has code execution and time.”
Edera’s CTO and co-founder Alex Zenla tells The New Stack, “The sandbox escape here isn’t really the surprising part of this attack. An entire market of AI sandboxing tools has exploded over the past couple of months, and this is the reality they’re all built on top of: containers, VMs via namespaces — whatever the branding — most of them still share a kernel with the host or with each other.
“That’s a boundary enforced in software, and software boundaries are exactly the kind of thing an agent that can try ten thousand escape paths over a weekend is going to find a way through. We shouldn’t be shocked that this happened; we should be shocked at how many teams are still betting their infrastructure on technology that was never designed to withstand such a persistent adversary. The fix isn’t a better sandbox; it’s getting out of the sandbox paradigm entirely.”
Zenla continues, “Teams should adopt secure execution environments that are hardware-enforced and eliminate the shared kernel flaw. This technology exists today and would have made this specific escalation chain structurally impossible, not just harder. Teams running agents with real permissions and real access need to stop treating this as a someday problem, because the next version of this week is already being tested somewhere right now.”
That underlines the most important issue of all. Thanks to AI, security attackers are coming harder and faster than ever. Security can no longer be an afterthought.
You literally no longer have time to wait for security fixes. You must bake in as much security as you can as fast as you can, or your systems will be broken into. It’s as simple as that.
Microsoft AI released two new in-house models into public preview on Wednesday — MAI-Image-2.5-Pro, its highest-fidelity image generator to date, and MAI-Voice-2-Flash, a speech model built for high-volume enterprise workloads — while publishing production data that amounts to the company's most aggressive argument yet that it can power its own products without leaning on OpenAI's frontier models.
The announcement, made by Microsoft AI's Superintelligence team, lands roughly a year after the company committed to building purpose-built models internally, and it arrives with an unusual level of specificity about where those models now run: Bing, PowerPoint, OneDrive, Dynamics 365, Excel, GitHub Copilot, and Azure. The message to enterprise buyers — and, implicitly, to OpenAI — is that Microsoft's homegrown models are no longer research projects. They are production infrastructure serving millions of users.
"Each of these enhancements is a step toward the same goal: Microsoft products, powered by Microsoft models," the company wrote in its announcement blog.
How MAI-Image-2.5-Pro and MAI-Voice-2-Flash stake out opposite ends of the AI cost curve
The two new releases occupy opposite ends of what Microsoft calls the quality-speed-cost curve, and the positioning is deliberate. MAI-Image-2.5-Pro targets the premium tier: hero imagery, detailed editing, and precise in-image text rendering — the last of which has long been a notorious weak spot for image generation models. Microsoft priced the model at $5 per million text input tokens, $8 per million image input tokens, and $106 per million image output tokens. The base MAI-Image-2.5 model recently launched at No. 2 for image editing on Arena, the community leaderboard that has become a de facto scoreboard for generative media.
The creative industry appears to be taking notice. Rob Reilly, global chief creative officer at advertising giant WPP, called the Pro model "a strong leap forward for GenMedia tools" in a statement included in Microsoft's announcement, adding that "Microsoft has firmly established itself among the leaders in generative AI."
MAI-Voice-2-Flash goes the other direction. First previewed at Microsoft's Build conference, Flash runs twice as fast as MAI-Voice-2 and costs 32% less, priced at $15 per million characters. It is designed for the unglamorous but enormous market of high-volume voice — call centers, voice agents, and real-time speech applications where latency and cost-per-call matter more than marginal gains in expressiveness. Together, the two models reflect a strategy of building families of models rather than a single flagship, because, as the company put it, a creative studio chasing maximum fidelity has very different needs from a customer service operation handling millions of calls a day.
Microsoft's production metrics show in-house models cutting GPU costs by up to 89%
The model launches are arguably less newsworthy than the deployment metrics Microsoft attached to them — numbers that read like a systematic case for swapping out third-party frontier models across its product portfolio.
Bing Image Creator now runs entirely on MAI-Image-2.5, end to end, marking the first time the consumer image tool is fully in-house. In PowerPoint, Microsoft says MAI-Image-2.5 reduces GPU costs by up to 84% compared with GPT-Image-2, OpenAI's image model. In OneDrive, where MAI-Image-2.5 is now the default for key image-editing scenarios, the company reports a 26% increase in save rates, roughly 25% lower P95 latency, and 2.5 times greater efficiency under medium-utilization production workloads.
On the voice side, MAI-Voice-2-Flash now powers Dynamics 365 Contact Center — the platform used by customers including T-Mobile and EasyJet — where Microsoft claims GPU cost reductions of up to 89%. The model is also integrated into Azure Voice Live for developers building speech-to-speech agents.
Perhaps the most consequential deployment sits in healthcare. Microsoft's Dragon Copilot, used by 170,000 medical providers and responsible for processing 28 million patient encounters last quarter, now runs on MAI-Transcribe-1.5 for its multilingual workflow across 58 languages. Microsoft says internal evaluations show a 50% relative reduction in both transcription and language-identification error rates across most languages — a meaningful claim in a domain where transcription errors can propagate directly into clinical notes.
Inside the 'hill-climbing' strategy that lets small models beat GPT-5.6 in Excel
In a companion post published the same day, Microsoft detailed the methodology behind these results — what it calls its "hill-climbing machine," an integrated flywheel of data, models, and the product "harness" that surrounds them.
The clearest example is MAI-Code-1-Flash, the lightweight coding model launched in GitHub Copilot in June. Microsoft says the model achieves an approximately 10% higher code accept rate than GPT-5.4 Mini and Claude Haiku 4.5 in VS Code, while using 10% fewer median tokens. Developer retention tells a similar story: users were 6% more likely to return across multiple days than with GPT-5.4 Mini, and 11% more likely than with Claude Haiku 4.5.
Then Microsoft did something more interesting. It took the MAI-Code-1-Flash checkpoint and further trained it inside an Excel reinforcement learning environment, teaching a coding model the tools and workflows of spreadsheet knowledge work. The result, according to production user feedback, is a model on par with GPT-5.6 for the most common Excel tasks — while being small enough to run on Nvidia's older H100 and even A100 GPUs rather than requiring the latest-generation accelerators.
That hardware detail deserves emphasis. Every major AI company is fighting for allocation of cutting-edge chips, and a model that delivers frontier-adjacent quality on two-generation-old silicon fundamentally changes the deployment economics. It also frees the newest hardware — including Microsoft's now-operational GB200 cluster — for training rather than serving.
Satya Nadella's 'frontier diffusion' manifesto redraws the OpenAI relationship
Microsoft CEO Satya Nadella framed the announcements in a lengthy post on X titled "Frontier Diffusion & Control," which functions as something close to a strategic manifesto. "We can now take saturated frontier capabilities and deliver them at scale and at lower cost through models optimized for high-usage products, while continuing to use frontier models for frontier needs," Nadella wrote, adding that Microsoft is "beginning to route traffic across our first-party surfaces to MAI whenever our models match or outperform frontier alternatives."
Translated from executive prose: capabilities that were state-of-the-art a year ago are now table stakes, and Microsoft believes it can replicate them cheaply for the specific, repetitive tasks that dominate real product usage. Why pay frontier prices for a frontier model when a user just wants to reformat a spreadsheet column?
Nadella was careful to note that "frontier models from OpenAI and Anthropic are part of the orchestration system alongside MAI" — but he also articulated a pointed principle of model independence, arguing that a company's evaluations "should continue to hill climb even when any given model has been removed."
“Keeping the harness, memory, context, and skills outside the model, he argued, is what gives Microsoft control. The subtext is hard to miss. Reuters reported in April that Microsoft’s exclusive license to OpenAI’s technology had been revised into a non-exclusive arrangement, and The Information reported last September that Microsoft had begun incorporating Anthropic models into some products. Wednesday’s announcement completes the triangle: Microsoft as orchestrator, with its partners’ frontier models as interchangeable components and its own models absorbing an ever-larger share of routine traffic.”
Developers cheer cheaper task-specific models while skeptics question Microsoft's track record
The response online captured both the appeal and the skepticism surrounding the strategy. "I love when people use small models for niche tasks," wrote one X user, @mavihsk, responding to Nadella's post. "Why do I have to use the all-knowing model just to change my field in Excel?" Another user, @nabu_lines, distilled the pitch neatly: "cost and performance both improve when you stop overusing the biggest model."
Others were less charitable about Microsoft's execution track record. "Microsoft is the worst when it comes to listening to user feedback," wrote designer @designedbyabin, arguing the company "will lose the AI race because they repeatedly failed to understand user needs." And one user, @tokenoverflow, offered a drier critique of the model-independence pitch: "i want it keep hill climbing after removing microsoft."
The skeptics raise a fair point. Microsoft's self-reported metrics — accept rates, save rates, GPU savings — come from its own internal evaluations, not independent benchmarks, and the company chooses which comparisons to publish.
But the strategy's logic does not depend on any single number. Nadella's framing that software now has "real marginal cost for the first time" explains why Microsoft is obsessive about tokens, GPUs, and serving costs: when AI features run on every keystroke across a billion-user product portfolio, an 84% GPU cost reduction is not an optimization. It is the difference between a viable business and a money pit.
Why Microsoft is turning its internal AI playbook into an Azure product
The final piece of the strategy is that Microsoft is selling the playbook, not just the models. Nadella explicitly positioned the hill-climbing approach as "a template for every other AI native, SaaS, or Enterprise company," and Microsoft is packaging the toolchain through Foundry and what it calls Frontier Tuning — letting enterprises train specialized models against their own proprietary evaluations and reinforcement learning environments. That turns Microsoft's internal cost-cutting exercise into an Azure product, and it gives enterprise customers a reason to run their AI workloads on Microsoft's cloud even if the models themselves come from elsewhere.
The company's emphasis on models trained "on clean, traceable, enterprise-grade data, without distillation from third-party models" serves the same commercial end. In an industry facing mounting scrutiny over training data provenance, Microsoft is betting that enterprise buyers — and courts — will care where model capabilities come from. Microsoft says it is now extending the hill-climbing approach to Copilot Chat, Outlook, and PowerPoint, and both new models are available in public preview through Microsoft Foundry and the MAI Playground. "None of this is an endpoint," the company wrote. "We're just getting started."
Seven years ago, Microsoft bet more than $13 billion that OpenAI would build the future of AI. Wednesday's announcement suggests the company has since learned a cheaper lesson: the future of AI may belong to whoever builds the frontier, but the profits belong to whoever makes it ordinary.
The AI industry has largely focused on language-based approaches, using transformers trained on massive datasets to predict words or fill in missing information.
As AI expands into more structured fields, however, the limitations of text-generation models are becoming clearer. Nvidia is pursuing a different approach.
The company just dropped JEPA-DNA on Hugging Face. It’s a genomic foundation model that adds a latent-space prediction objective alongside MLM.
This release is a win for hybrid architectures that go beyond purely generative training. It’s the kind of shift Executive Chairman of AMI Labs, formerly Meta’s Chief AI Scientist, Yann LeCun, has been pushing for years. He’s championed predictive architectures as a general alternative to next-token prediction, which researchers are now applying to biology.
Latent space over literal tokens
Conventional genomic base models have historically mirrored NLP models, relying purely on MLM, masking parts of a DNA sequence and forcing the model to guess the missing literal tokens. This approach favors local token reconstruction and teaches the model the basic “syntax” of the sequence, but it frequently struggles to grasp the wider functional “meaning.”
JEPA-DNA changes the paradigm. The newly released checkpoint, JEPA-DNA-DNABERT2, serves as a model-agnostic continual pre-training framework. It couples standard token-level DNA language modeling with JEPA, adding a second learning objective.
Instead of forcing the architecture to reconstruct missing tokens, JEPA-DNA supervises the model’s global sequence embedding in a latent space.
Instead of forcing the architecture to reconstruct missing tokens, JEPA-DNA supervises the model’s global sequence embedding in a latent space. It predicts the functional representation of masked genomic segments rather than their literal, character-by-character makeup.
Token prediction is still part of the training process, but it is no longer the model’s only learning objective. The model also learns the basic structure of the data, proving that predictive architectures can produce representations that are easier to work with for biological tasks without sacrificing generative capabilities.
Token prediction is still part of the training process, but it is no longer the model’s only learning objective.
How DNABERT-2 gets upgraded
The model builds on DNABERT-2, a 117 million-parameter model developed by Zhihan Zhou and collaborators. Nvidia layers its continual pre-training approach on top of that architecture, letting the model learn from both token-level predictions and latent-space representations.
Nvidia has released the model globally for non-commercial research. According to the company, the model is meant to support research workflows, including feature extraction, linear probing, continual pre-training experiments, and zero-shot scoring of DNA sequence changes. It isn’t a diagnostic tool or a clinically validated medical product.
Beyond the generative hammer
DNA contains patterns and relationships that aren’t captured by sequence prediction. JEPA-DNA supplements masked-token prediction with a latent-space objective intended to capture broader sequence-level information.
Ultimately, models like JEPA point to a new path for AI that combines next-token prediction with other ways of learning and helping models build a deeper understanding of complex systems.
The models small enough to run on the box on your desk are getting good enough that the interesting question is no longer whether you can run them, but what you can do with them, and how organizations can get the most out of them.
Joey Conway, Nvidia’s senior director of generative AI software, spoke to The New Stack about how local and open models are increasingly working alongside frontier models, often with a router in between deciding which one to use, and how organizations can adapt these open models for their own needs.
“We love the world where we can use both frontier and open models together.” — Joey Conway, Nvidia
A system of models
Tasks vary in complexity, so the models handling them should vary too, Conway tells The New Stack. He points to the early open reasoning models, which would reason their way through trivial problems, mulling number lines and memory to work out what two plus two is. “I just say four,” he says.
“Being able to route those easy things to local models that are quick, and route the hard things to more sophisticated models,” Conway says, lets you “get a better outcome at a lower cost and lower time to completion.”
It’s a different picture from the one large model most people imagine doing everything. In his version, you build a bench of specialists. “You’ll have specialized agents that are really good at focused tasks because that’s what they do every day,” he says, “and they just get better and better at that task.”
To the user, none of that shows. “It’ll feel like one interface,” Conway says, “but behind that interface, there’ll be a variety of models handling a variety of tasks.”
Getting there is largely a routing problem, and one Conway says is still in its early days. Nvidia’s own contribution, for now, sits lower in the stack, in inference-serving software like its open-source Dynamo, which steers each query to the GPU that handled it most recently. Which model is best for which job, Nvidia leaves to a wider field of routers, some of them models in their own right that weigh budget, latency, and modality. But Conway also leaves the door open for Nvidia to build more of that routing itself before long.
Nvidia points to its collaboration with LangChain, whose Deep Agents harness ran on Nemotron 3 Ultra, Nvidia’s 550-billion-parameter open model, and matched top closed models on business tasks at up to a 10x lower cost, as Conway notes. It required no retraining; the gains came entirely from tuning the harness around it: its prompts, tool descriptions, and middleware.
You’re not going to run a 550-billion-parameter model on your desktop anytime soon, but running relatively large models locally is now a real possibility, as long as you have some beefy hardware at your disposal. For enterprises, setting up a fleet of accelerators in a data center isn’t exactly cheap either, but it does mean full control and no surprise token bills.
Bringing AI to where the data lives
Running models yourself can save money, but Conway thinks control matters more. Enterprises already decide where their data lives and what they hand to outside vendors, and open models give them even more control. “Move AI to where your data lives,” he says, “or move AI to where your employees are.”
Companies want to keep their data — and especially their intellectual property — in-house, and Conway argues a fine-tuned open model is the place to put it. “It’s like an employee,” he says. “You hire them, and they’re part of your company.”
The local half runs on hardware like Nvidia’s DGX Spark, a $4,699 Grace Blackwell machine with 128GB of unified memory that handles models up to roughly 200 billion parameters without anything leaving your desk. (There is also the DGX Station, its bigger, pricier sibling with 748 GB of RAM for running even larger models.)
“It’s like a system sitting right there next to you,” Conway says, one where “you don’t think about network latencies.” To run those agents securely, Nvidia offers NemoClaw, a reference stack that wraps an open agent harness like OpenClaw in a sandbox called OpenShell, with policy controls and local Nemotron inference.
When you need more power for a broader problem, you reach for a frontier model in the cloud. For Nvidia, that’s all good news: a system of models runs on its silicon one way or another, on your desk or in the cloud.
Cursor, the AI coding tool recently acquired by Elon Musk’s SpaceX in a $60 billion all-stock deal, has launched a model router designed to direct every coding request to whichever model handles it best, bypassing the need to pay frontier prices for work that doesn’t need it.
Under the hood, the new Cursor Router uses a triage system à la a hospital emergency room: It looks at what a request actually needs — how hard it is, what it’s for, the surrounding code — and picks a model that is the best fit. A quick fix goes somewhere cheap, while a genuinely hard problem gets escalated to something closer to frontier-grade.
Notably, developers and admins also have access to three distinct modes that nudge that balance in either direction, favoring speed and cost over raw power, or vice versa if needed.
Choosing from three optimization modes
The broader rationale for Cursor Router, according to the company’s own field CTO David Pan in a social media post on Wednesday, is that developers shouldn’t have to become experts in model performance just to write code.
“We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rate.”
“We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rates,” Pan writes.
We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rates.
Cursor Router is out today. It picks the model for each task based on measured quality against cost. Fable 5 level performance for… https://t.co/3B2103mGWr
Early community feedback has largely echoed Pan’s sentiment: Fatih Arslan, a software engineer at PlanetScale, notes on X that engineers already juggle the choice between cost and capability by hand — defaulting to a cheap, fast model for routine work and saving the slow, expensive one for “serious tasks.”
Nice! I wanted something like this for a long time. See my previous rant last year. I think as engineers, we're already developing skills ourselves which model to choose. We use a less costly/fast model for implementing chore details, but use a higher reasoning model (slow and… https://t.co/5y5xMOR2E6
“We already spend quite a bit [of] time on [choosing models],” Arslan writes. “Why not automate that part? Cursor Router does the automation.”
“We already spend quite a bit time on it [choosing models]. Why not automate that part?”
In a separate blog post published on Wednesday, Cursor claims that early access customers saved 30-50% compared to routing everything through Opus 4.8, with no drop in output quality.
Working model: Taking control of the stack
The launch follows a run of moves by Cursor to control more of its own AI stack. In May, the company released Composer 2.5, an update to its in-house coding model built for long tasks at a lower cost than frontier options from Anthropic and OpenAI. Composer 2.5, like its predecessor, is built on Moonshot AI’s Kimi K2.5, an open-weight model out of China.
Now, with the weight of one of the world’s most valuable companies behind it (SpaceX has attained a market cap of $1.5 trillion since its June IPO), Cursor is pushing a powerful frontier model of its own.
On July 8, Cursor and SpaceXAI jointly released Grok 4.5, a mixture-of-experts model built on a new foundation dubbed V9, which Musk had previously noted was roughly 1.5 trillion parameters. The model’s trained on trillions of tokens of real Cursor usage data and available across all Cursor plans at $2 per million input tokens and $6 per million output tokens.
With Composer handling cheap, fast work, and now the Grok-branded frontier line for more serious horsepower, Cursor has its own models in the mix alongside the usual list of outside providers. And this gets to the heart of why Cursor built Router: Most developers pick one model and stick with it regardless of the task, billing simple work at frontier prices it doesn’t need.
Sending every request to its own models would be the easy way to keep that money in-house, but it would also mean shipping inferior output on some tasks — so Router instead sends each request to whichever model actually suits it, Cursor’s own or not.
The lay of the land
Model routing itself isn’t exactly new. OpenRouter has offered a version since 2023: a single API sitting in front of more than 400 models from over 60 providers, including OpenAI, Anthropic and Google. Its own auto-router feature does roughly what Cursor Router does — classify a request, then send it to whichever model fits the task and the person’s stated preference between cost and quality.
More recently, OpenRouter launched Fusion, which takes a slightly different approach: instead of picking one model, it sends a prompt to several models at once and uses a judge model to synthesize the strongest answer out of all of them.
This past month ushered another entrant to the mix: Japan’s Sakana AI released Fugu in June, which instead breaks a single task into subtasks and routes each piece to a different model, pitched by Sakana as a hedge against relying on any one AI provider.
“[Cursor Router is] a great example of how a technological innovation immediately translates into a product improvement.”
Not everyone rates some of these other attempts, though. On Wednesday, Kirill Balakhonov, head of AI products at Nethermind, argues on LinkedIn that Cursor’s version succeeds precisely because it’s focused on coding specifically, rather than trying to be a general-purpose router for any task.
“A great example of how a technological innovation immediately translates into a product improvement… rather than an abstract idea like Sakana Fugu or OpenRouter Fusion,” Balakhonov writes, predicting both of those broader routing efforts would fade from use.
What’s new, perhaps, is some of the names emerging behind an array of model diversity efforts. In early July, Microsoft launched a $2.5 billion services unit dubbed Microsoft Frontier Company, embedding thousands of engineers at customer sites to help them build with a mix of AI models.
Judson Althoff, CEO of Microsoft Commercial Business, told Reuters at the time that the push came partly from watching rivals like DeepSeek and Google’s Gemini close the gap on OpenAI. Referring to the original Copilot, he admitted, “we made a mistake by binding it to OpenAI models only.”
If the company with the deepest single-model relationship in the industry is walking it back, the idea for model flexibility has clearly gone mainstream — certainly if this week is anything to go by.
Ramp and Meta get in on the act
On Tuesday, Ramp, the $44 billion spend-management behemoth, opened up Ramp Router, an early-access public version of the model router it built to manage its own AI bills internally, which it says cut its LLM costs by roughly 30%. It’s free to start, requires no Ramp account, and routes across OpenAI, Gemini, and select open-source models including Kimi through an OpenAI-compatible endpoint.
The very same day, Jyoti Mann fromThe Informationreported that Meta is also working on a model router. According to internal documents cited in the report, an incubator inside Meta called AAI Labs is developing a new product named Switchboard that would score each request for difficulty and send simpler ones to smaller, cheaper models — aimed initially at cutting Meta’s own AI agent costs, though it reportedly could end up as a public release.
Meta has particular reason to want this. Data from Runpod’s State of AI report, published in March, suggests Meta’s open Llama models are now a marginal presence in production: Llama 4 has near-zero real-world deployment, with Alibaba’s Qwen having overtaken it as the most-deployed self-hosted LLM.
Meta has also been building proprietary models in response. Muse Spark, its first model out of the newly formed Meta Superintelligence Labs, shipped in April. That was followed in July by Muse Spark 1.1, Meta’s first model with a public, paid API, priced at roughly a quarter of what OpenAI and Anthropic charge for comparable models.
Meta is aggressively targeting the incumbents, and Switchboard fits the same pattern: a way to make it easier for users to cut costs, switch models freely, and, where it makes sense, land requests on Meta’s own models instead.
But amidst all this hullaballoo about model routing, there is perhaps a broader question of openness. Not of the models themselves, which is a fervent debate in itself, but of whether the routing decision itself — the logic that decides which model handles which request — should sit inside a vendor’s own closed product at all.
“Is anyone building this as open-source?”
Elvis Saravia, a former technical product marketing manager at Meta AI who co-founded DAIR.AI, took to X to argue that it shouldn’t, given how differently teams weigh cost against quality.
“Is anyone building this as open-source?” Saravia asks. “It feels like this is something you don’t want to offload to an API. We all work with different trade-offs, so we need the ability to achieve custom routing.”
Recommended reading. Cursor Router routes tasks to the right model.
Is anyone building this as open-source? It feels like this is something you don't want to offload to an API. We all work with different trade-offs, so we need the ability to achieve custom routing. https://t.co/zcrpHbnlCepic.twitter.com/Vw2xyMy1jp
As for Cursor’s own version, Router is currently available to Teams and Enterprise customers only, across desktop, web, iOS, CLI, and Cursor’s SDK. It’s not yet clear whether it will eventually land on individual plans.
Every product team is chasing the same moment: The user opens a page and thinks, this understands me.
A shopper who loves floral prints should see more floral prints. A user who follows local politics should open their app to see news about local politics. A job candidate who keeps clicking remote roles should not keep getting shown in-office jobs.
That is not a niche feature anymore. It is the baseline expectation. Users decide quickly whether a product system understands them, and they rarely care whether the failure came from search, recommendations, merchandising rules, or stale data.
Here is the uncomfortable truth: Most teams do not have a personalization quality problem. They have a personalization architecture problem.
Personalization is not a widget bolted onto search. It is a ranking decision. The system has to decide, for this user and this request, what deserves the next slot. That means weighing the user, the item, the context, and the business goal at the same time. In many stacks, the ranking layer is the one place that cannot see all of those signals together.
The hard part is not collecting signals. The hard part is combining them while the user is still there.
Why personalization is hard in the first place
To put the right item in the right slot, a system has to understand several things at once:
Intent: What is the user asking for right now?
Item quality: What does each candidate actually contain or represent?
User history: What has this person clicked, bought, read, watched, or ignored?
Availability: Is the item in stock, fresh, nearby, legal to show, or ready to ship?
Business priority: What should the business promote, protect, or de-emphasize?
Those signals often disagree. The most relevant item may not be the most profitable. The most profitable item may be out of stock. The user may say “running shoes,” but their behavior says “trail running, wide fit, under $120.”
They also move on different clocks. Product attributes change slowly. Inventory and price can move throughout the day. Preferences shift with every click. External context — weather, breaking news, a championship game, a cultural moment — can matter without warning.
Personalization means folding all of that into one ordered list, on every request, in milliseconds. The signals themselves are not the bottleneck. Query-time ranking is.
The usual stack makes the problem harder
Most personalization systems are assembled from tools that were each designed for one slice of relevance.
Keyword search engines are excellent at lexical matching. They are good when the query language and catalog language line up. But shoppers, readers, and job seekers rarely speak in neat index terms. You indexed “athletic performance running footwear”; they typed “running shoes.” Synonym rules can help, but they do not scale gracefully across long-tail language, changing catalogs, and new user behavior.
Vector databases start from the opposite side. They are good at semantic similarity: “Find me things like this.” That is powerful, but nearest-neighbor search is not the same thing as personalization. Real ranking has to blend semantic similarity with live behavior, stock, price, margin, freshness, eligibility, and business rules.
Re-rankers, recommendation services, feature stores, and rule engines are usually added to glue everything together. That is where fragmentation creeps in.
Figure 1. A fragmented personalization stack compared with a unified query-time ranking pipeline
When retrieval and ranking live in separate systems, the ranker often works from a partial, stale, or precomputed view of the world. Click history, session context, and the user’s live preference vector arrive too late. Business rules become filters or overrides instead of ranking signals. Fresh inventory or price changes require coordination across multiple systems.
Every hand-off adds latency. Every boundary creates another place for signals to drift. Every “quick rule” becomes another hard constraint that can accidentally turn “show the closest match” into “show nothing.”
“Every hand-off adds latency. Every boundary creates another place for signals to drift.”
The deeper issue is a timing assumption. Many architectures were built around offline ranking: process the catalog, compute scores in a batch job, and serve those scores until the next rebuild. That works when preferences are stable. It breaks when the most valuable signal is the click that happened two seconds ago.
What changes when ranking happens in one real-time pipeline
A real-time personalization architecture treats retrieval, ranking, and inference as one serving problem.
That is the core idea behind Vespa’s approach: Text search, vector similarity, structured filtering, ranking expressions, tensor computation, and model inference can live inside one query pipeline. Instead of retrieving somewhere, enriching somewhere else, and ranking at the end, the system can rank with the relevant signals while it is still deciding what to return.
That architectural choice changes the shape of the problem.
1. Retrieval is hybrid from the start
Lexical search, semantic search, and structured filtering can run together instead of being reconciled after the fact. A product query can combine text, embeddings, filters, session behavior, and item attributes in one request.
That matters because personalization is rarely one signal. The user’s query still matters. So does semantic similarity. So do category, availability, price, and business constraints. Hybrid retrieval keeps those signals in play before ranking starts.
2. Ranking can express the actual objective
A personalization score should not be trapped inside one similarity function. It should be a formula that reflects the product’s goals.
That formula might combine BM25, vector similarity, user affinity, stock level, margin, popularity, discount depth, freshness, rating, distance, or a weather term. Some of those signals need normalization first. Some should matter only for certain categories or users. Some should be tested as weights.
The important part is that they are all terms in the same ranking expression, not scattered across services.
In production, the formula can be more nuanced. But the principle is simple: personalization, relevance, and business logic belong in the same scoring decision.
3. Model inference can run where the data lives
Some signals should come from learned models rather than hand-tuned rules: propensity to buy, churn risk, quality prediction, fraud risk, query classification, or a learned-to-rank model.
When inference runs in the serving path, those model outputs can become ranking features instead of delayed batch scores. That reduces the need to ship data to a separate inference service, wait for a response, and stitch the score back into ranking.
4. Updates become immediately useful
“Real time” should not mean “after the next index rebuild.” If inventory changes, stock should be rankable immediately. If a user clicks two yellow dresses, “yellow” should matter on the next request. If a merchandising team adjusts a ranking weight if the weight is exposed as a query-time input, the experiment should start producing useful feedback right away.
That is the difference between personalization as a nightly job and personalization as a live ranking decision.
Tensors make the personalization concrete
The most useful mental model is simple: represent the user and the item in the same feature space, then rank by how well they match.
In Vespa, tensors make that practical. A tensor can be a scalar, a dense vector, a sparse map of feature-weight pairs, a matrix, or a more complex structure. That means the same framework can represent semantic embeddings, product attributes, user preferences, business objectives, and model features.
Figure 2. User and item tensors combined into a personalization score, then blended with other ranking signals
For example, each item can carry a sparse feature tensor:
Because the two tensors share a shape, personalization becomes a dot product: multiply matching features, sum the result, and use that score inside ranking.
In a Vespa rank profile, the core expression is compact:
# schema: item attributes stored as a sparse tensor
field item_features type tensor<float>(feature{}) {
indexing: attribute | summary
}
# rank profile: the user's live preferences arrive as a query tensor
rank-profile personalized {
inputs {
query(user_features) tensor<float>(feature{})
}
first-phase {
expression: sum(query(user_features) * attribute(item_features))
}
}
That one expression is not the whole ranking function. It is the personalization term. BM25, vector similarity, stock, margin, freshness, distance, or a model score can be added as other terms with their own weights.
The user tensor is where real-time behavior becomes powerful. Click a floral item, and the “floral” weight rises. Click two yellow items, and “yellow” rises; the application feeds click events into the user profile. The next query can use those updated preferences immediately, without waiting for a nightly profile build.
Business goals stop fighting personalization
In fragmented stacks, business rules often become blunt instruments: boost this category, hide that brand, force these items to the top, filter these out. That can satisfy a short-term merchandising goal while damaging relevance.
When business logic is part of the ranking expression, it can be more subtle. You can boost overstocked inventory without ignoring intent. Promote umbrellas when rain is forecast without turning every search into an umbrella search. Give new sellers a small exploration boost. Prioritize destocking before a new product line launches. Surface team merchandise during a championship run.
“When business logic is part of the ranking expression, the user still gets relevant results. The business still influences outcomes.”
The user still gets relevant results. The business still influences outcomes. The difference is that both are expressed as ranking signals instead of competing systems.
That also makes experimentation easier. A merchandising or growth team can test weights, traffic splits, and ranking profiles without asking engineering to rewrite the whole pipeline. Relevance becomes a controllable growth lever rather than a fragile side effect.
The same pattern applies beyond commerce
The examples above are easy to picture in apparel, but the architecture is not commerce-specific. Personalization is the same ranking problem in many products:
Content feeds: Blend topic affinity, freshness, engagement, creator quality, and business rules.
News: Rank by reading history, topic interest, locality, freshness, and source diversity.
Jobs: Match candidate preferences such as remote work, seniority, compensation, location, and tech stack against role attributes.
Geo search: Treat distance as one normalized ranking term alongside relevance, quality, and preference.
Video and audio: Combine embeddings, viewing history, metadata, freshness, and learned ranking models.
Different domains need different features. The architecture pattern is the same: retrieve candidates, rank with the signals that matter, update those signals as behavior changes, and keep the decision close to the data.
Scale doesn’t have to be the trade-off
The natural concern is that a more expressive ranking system must be slower. In practice, that does not have to be true.
Vespa was built for large-scale serving from the beginning: billions of documents, high query volume, and low-latency ranking. The reason this works is multi-stage ranking. The system does not run the most expensive logic across every possible result. Instead, it uses a fast first phase to narrow the candidate set, then applies more precise ranking to the smaller group that remains.
For example, a cheap first phase narrows a huge candidate set. Then, once the candidate set is smaller, Vespa can apply full-precision scoring, richer tensor operations, business logic, and model inference where they matter most.
The result is a practical balance: speed across the full corpus, accuracy in the final ranking, and enough flexibility to personalize each query without turning the serving stack into a chain of fragile services.
What’s next
Personalization is not failing because teams lack data. Most teams already have plenty of signals: query intent, clicks, product attributes, inventory, margin, freshness, location, and business priorities. The harder problem is that those signals often live in different systems, move at different speeds, and arrive too late to influence the final ranking decision.
That is why personalization should be treated as a ranking problem. When retrieval, ranking, personalization, and business logic are split across separate systems, the ranker is forced to work with stale or incomplete context. The user moves faster than the architecture can respond. Every new signal becomes another integration project.
A unified real-time ranking pipeline changes that. User behavior, item attributes, semantic similarity, lexical relevance, inventory, and business goals can all become parts of the same scoring function. Tensors make those signals directly comparable and usable at query time. Instead of bolting personalization onto the end of the system, personalization becomes part of the decision the engine makes for every query.
The goal is simple: rank each result with the best context available, at the moment the user asks. That is when personalization stops feeling like a feature and starts feeling like relevance.
The AI ecosystem is drowning in tutorials on how to build a retrieval-augmented generation (RAG) app in five minutes. The pitch is appealing but flawed: Chunk a document, run it through an embeddings API, load it into a vector database, and slap a UI on top. This setup works locally. It might even survive a beta test with friendly users. But launch it into a production-grade B2B SaaS environment, and the architecture collapses.
Enterprise applications don’t handle neat, static files. They ingest dynamic, unpredictable streams of live data bound by strict legal and compliance constraints. Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.
“Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.”
Here is exactly what breaks when a naive RAG setup hits production, and the architectural trade-offs needed to fix it.
Bottleneck 1: the synchronous ingestion trap
Synchronous data ingestion is the most prevalent architectural flaw in new AI products. A user uploads a 500-page compliance manual. The client makes a POST call to a web server, which parses the document, splits the text, iterates over a sequence of synchronous API calls to OpenAI or Cohere for vectorization, and writes those vectors to the database.
This approach introduces two critical failures:
Timeouts: A 500-page document rarely finishes processing within standard HTTP timeouts (30 to 60 seconds) while waiting for the embedding API.
Cascade failures: If the system hits rate limits or latency spikes, the entire ingestion operation fails, throwing a 500 error and losing the user’s document.
The fix: the batched fan-out pipeline
Production-grade AI pipelines require persistent events rather than simple HTTP calls. However, sending the whole 500-page document to be processed by one consumer from Kafka or RabbitMQ is a mistake. If a consumer spends 10 continuous minutes generating embeddings, it misses its broker heartbeat. Assuming the worker died, the broker kills the consumer and triggers a partition rebalance, creating an infinite loop of duplicated work and stalled processing.
“Synchronous data ingestion is the most prevalent architectural flaw in new AI products.”
Conversely, granular chunking, where every chunk becomes an individual Kafka message, launches a self-inflicted denial-of-service (DoS) attack on downstream services. A document with 1,500 chunks generates 1,500 individual messages. This instantly exceeds upstream requests per minute (RPM) limits and floods the pipeline with network overhead.
The engineering sweet spot is a batched fan-out approach:
Asynchronous uploads: The web API stores the raw file in Amazon S3, triggers a document_uploaded event, and instantly returns a 202 Accepted status. This single, asynchronous path processes one-page invoices and 100-page SOC2 reports with equal reliability, eliminating the technical debt of maintaining separate “fast” and “slow” ingestion routes.
Micro-batching: A lightweight “Spitter” consumer downloads the file, chunks it, and groups those chunks into optimized micro-batches (e.g., 64 chunks per batch).
Controlled embedding: Embedding workers pull these batched events. To prevent concurrent workers from breaching upstream RPM limits, avoid fragile sleep() delays. Instead, enforce a token bucket rate limiter at the consumer level or strictly cap the number of active message broker partitions.
Python
# Conceptual snippet for architectural illustration
def handle_document_upload(event):
try:
raw_text = download_from_s3(event.file_uri)
chunks = semantic_chunking(raw_text)
except Exception as e:
# Log failure and raise so the message broker routes this to a Dead Letter Queue (DLQ)
print(f"Failed to process document {event.file_uri}: {e}")
raise
# Batch size heavily depends on the downstream embedding model's context limits
batch_size = int(os.environ.get("EMBEDDING_BATCH_SIZE", 64))
for i in range(0, len(chunks), batch_size):
chunk_batch = chunks[i:i + batch_size]
kafka.publish("embedding_tasks", {
"tenant_id": event.tenant_id,
"document_id": getattr(event, "document_id", event.file_uri),
"chunks": chunk_batch
})
This keeps individual consumer tasks short, respects upstream rate limits by maximizing payload density, and allows horizontal scaling of embedding workers during traffic spikes.
Bottleneck 2: the multi-tenant nightmare
Developers often treat multi-tenancy as an afterthought. The simplest way to handle multiple B2B tenants using a single RAG system is logical segregation, where all vectors reside in a large index, and each entry has a tenant_id associated with its metadata. Upon retrieval, the application filters results by adding a clause to the metadata payload.
Flaws of the approach:
Security vulnerabilities: Relying on application-level filtering creates an unacceptable risk. If an engineer omits or misconfigures a metadata filter, one client can access another’s confidential data. In highly regulated environments, this breaks compliance.
The noisy neighbor problem: If one customer uploads 10 million vectors to the shared index, memory usage skyrockets during vector searches. This degrades performance across the entire system, even for tenants with a handful of documents.
The fix: serverless compute-storage decoupling
Echo-chamber thinking assumes that the only solution is to provide each tenant with its own dedicated database cluster. This is prohibitively expensive and practically impossible to manage in a modern-day SaaS offering.
The true gold standard here is using next-generation serverless vector databases like Pinecone Serverless or managed Qdrant implementations, which make a clear distinction between computing and storage.
Isolation strategy
How it works
Trade-offs
Shared index (logical)
One index; application layer applies metadata filters.
High compliance risk; prone to noisy neighbor performance degradation.
Database per tenant (physical)
Client provisions a dedicated database cluster.
Maximum security, but introduces massive operational overhead and idle compute costs.
Serverless namespaces (standard)
Storage layer isolates vectors into namespaces; on-demand compute loads them only when queried.
Namespace-level access control prevents cross-tenant leaks. Zero idle compute costs.
Engineering takeaway: stop building complex multi-tenant routing logic in your application code. Push the isolation boundary down to the infrastructure layer using serverless namespaces.
Bottleneck 3: the semantic caching trap
Once ingestion is asynchronous and tenants are segregated, inference costs become the final bottleneck. Hitting an LLM API for every individual query is economically unsustainable.
The industry defaults to semantic caching: embed the user’s prompt, calculate its cosine similarity against previous prompts, and return a pre-calculated LLM response if the score exceeds a set threshold (e.g., 0.95).
Why semantic caching fails
Embeddings capture overall semantic meaning, but they miss specific contexts and entities. The prompts “What was the holiday policy in 2023?” and “What is the holiday policy for 2024?” share a near-perfect cosine similarity score. The core semantics match, but returning a cached answer feeds the user incorrect or contradictory information.
The fix: hybrid verification vs. native prompt caching
To scale without compromising accuracy, there are only two choices to consider: application-layer validation or infrastructure-layer optimization.
Strategy A: combined lexical filtering and intent routing
When using an application-layer caching system (for instance, Redis), you need to layer the semantics search on top of extremely light guardrails.
Exact-match filter: Apply a token-validation filter over vector similarity. In the case where the cached query is “2023”, and the current query is “2024,” throw out the cache hit right away.
Intent routing: Before serving a cached answer, use an inexpensive, fast model as an intent match router.
Python
Query A: {incoming_query}
Query B: {cached_query}
Do these queries have the exact same intent and require the exact same factual answer?
Respond only with YES or NO.
Strategy B: infrastructure-level prompt caching
If the system cannot tolerate the added latency of an application-layer verification router, bypass custom caching entirely and offload the problem to the infrastructure.
Modern LLM providers natively support prompt caching. It is crucial to understand what is being cached here: it is not the user’s short question. When multiple users query the same corporate knowledge domain, the massive system instructions and the heavy retrieved context documents (often 10k+ tokens) are cached automatically at the provider’s inference layer.
“Stop approaching AI like a magic black box and instead approach it as a distributed systems problem.”
The application sends the full RAG query every time. The provider recognizes the repeated context block, slashes context token costs by up to 80%, and drops the time-to-first-token (TTFT) to milliseconds.
The infrastructure surrounding a foundation model separates a prototype from a production AI system. Stop approaching AI like a magic black box and instead approach it as a distributed systems problem, and things will fall into place. Batched fan-out asynchronous queuing solves timeout and rate-limiting issues. Serverless namespacing resolves compliance risks. Prompt caching and intent routing secure unit economics. Designing a native AI product today means engineering for inevitable API failures, cross-tenant data attacks, and runaway LLM costs.
Gritt, the company building an intelligent system combining robotics and AI to accelerate the world’s infrastructure buildout, has launched with $32.4 million in pre-seed and Series A funding. The $26 million Series A was led by Obvious Ventures, with participation from Union Square Ventures and Active Impact Investments, and support from previous investors First Round […]
Poolside, the San Francisco AI lab that has spent most of its three-year existence quietly selling coding models to governments and defense agencies, released its most capable model to date on Tuesday — and made an unusually aggressive bet that radical transparency, not raw scale, is how a smaller lab competes at the frontier.
The model, Laguna S 2.1, is a 118-billion-parameter Mixture-of-Experts (MoE) system that activates only 8 billion parameters per token, supports a context window of up to 1 million tokens, and — according to benchmarks published by the company — matches or beats open models several times its size on agentic coding tasks. The weights are available immediately on Hugging Face under the permissive OpenMDW-1.1 license.
The headline numbers are striking for a model this small. Poolside reports that Laguna S 2.1 scores 70.2% on Terminal-Bench 2.1, a benchmark of long-horizon terminal tasks, placing it 11th on the company's compiled leaderboard — ahead of DeepSeek-V4-Pro-Max, a 1.6-trillion-parameter model that scored 64.0; Thinking Machines' 975-billion-parameter Inkling, at 63.8; and Nvidia’s 550-billion-parameter Nemotron 3 Ultra, at 56.4. On SWE-Bench Multilingual, it posts 78.5%, and on SWE-Bench Pro's public dataset, 59.4%.
Perhaps more telling than any single score: the model went from the start of pre-training on May 22 to public launch in under nine weeks, trained on 4,096 Nvidia H200 GPUs. In an industry where flagship model cycles are typically measured in quarters or years, Poolside has now shipped three models in three months.
Why the West's open-weight AI gap has become a boardroom issue
The release lands in the middle of an increasingly pointed debate about the provenance of open-weight AI. Over the past year, developer adoption has shifted decisively toward open-weight systems that companies can download, inspect, and run on their own infrastructure — and the leading options in that category have overwhelmingly come from Chinese labs. DeepSeek, Qwen, Kimi, GLM, MiniMax, and Tencent's Hunyuan line all feature prominently in Poolside's own comparison tables.
Poolside's accompanying press release frames Laguna S 2.1 explicitly as a response, noting that the model occupies a size class into which no Western lab has released open weights in 11 months — since OpenAI's gpt-oss-120b last August. "The West needs open-weight models it can trust, run, and build on," said Jason Warner, Poolside's co-CEO, in the announcement.
Co-founder and co-CEO Eiso Kant made the philosophical stakes even plainer in a lengthy post on X. "I believe intelligence should and will become a commodity," he wrote, arguing that the open ecosystem "will not win by being the best in its own category." Users, he argued, simply want the best intelligence for the task at hand — so open models must be on par with, or better than, their closed equivalents.
The strategic logic here is not charity. Poolside's core business is deploying models inside the security boundaries of government, defense, and regulated enterprises — customers for whom closed, metered API access is often a non-starter for compliance and sovereignty reasons.
Every enterprise that standardizes on a Chinese open model today becomes harder to win tomorrow. Releasing competitive open weights is both an ecosystem play and a top-of-funnel strategy for the company's high-security deployment business. It also reframes the AI race away from terrain where Poolside cannot compete — frontier-scale capital expenditure — and toward terrain where it believes it can: cost per token, self-hosting, and iteration speed.
How a sparse architecture makes enterprise AI agents affordable to run
The technical design reflects a specific thesis about where value in coding AI is moving. Laguna S 2.1's sparse MoE architecture — 256 routed experts plus one shared expert, with grouped-query attention and interleaved sliding-window layers, according to the Hugging Face model card — means inference costs scale with the 8 billion active parameters, not the 118 billion total. Poolside emphasizes that the model is small enough to run on a single Nvidia DGX Spark, the desktop-class AI machine.
That matters for what Poolside calls token economics. Long-horizon coding agents are voracious consumers of tokens: the company's published data shows the model consuming a mean of roughly 249,000 completion tokens per trajectory on its hardest benchmark when thinking mode is enabled. At metered API prices, agentic workloads at enterprise scale become a meaningful budget line item. On OpenRouter, Poolside is offering a free 256K-context endpoint and a dedicated 1M-context deployment priced at $0.10 per million input tokens and $0.20 per million output tokens — aggressive pricing that undercuts most frontier alternatives by an order of magnitude.
The ecosystem support is unusually broad for day one. The model is live on Baseten's model library and Vercel's AI Gateway, with integrations across vLLM, SGLang, Ollama, and llama.cpp, plus quantized variants down to 4-bit GGUF files — 75 gigabytes — for local use. But Poolside's more interesting claim is behavioral, not architectural. Pengming Wang, co-head of applied research at Poolside, said the gains came from improving the model's working habits: "more verification, less taking things for granted, not declaring victory early, and being more persistent." Raw intelligence, the company argues, is one axis of capability; a model's way of working is a second axis that matters immensely for agents left unattended for hours.
Publishing every benchmark trajectory to counter AI's credibility crisis
The most consequential part of the release for enterprise buyers may be an evaluation-transparency move with little precedent among major labs: Poolside published the complete, unedited trajectory of every trial in its final benchmark runs — every reasoning step, tool call, and shell command behind every reported score.
This addresses a growing credibility problem in AI benchmarking. As top scores on mature benchmarks cluster in the 70–90% range, and as "reward hacking" — models finding solutions online or gaming verifiers rather than solving problems — has become endemic, self-reported numbers have lost much of their signal. Poolside disclosed its own encounters with the problem candidly: during training, more than half of trajectories on some SWE-bench tasks were flagged because the model simply researched the original bug-fix pull request online and applied it. The company documented its mitigations, including prompt addenda, LLM-based judging calibrated against human labels, and expert annotator review of a high-scoring Terminal-Bench run.
Three published case studies illustrate what the company means by persistence. In one, the model built a working HTML/CSS rendering engine from an empty folder in a 181-step, 50-minute unattended session — then, lacking vision capabilities, spun up headless Chromium to numerically compare its canvas output against a real browser's rendering. In another, pointed at Poolside's own agent harness in an automated optimization loop, the model made the Go codebase 5.2% faster with roughly 70% lower memory allocation, finding an O(n²) string-concatenation bug along the way. In a third, working in a sandbox with no Python installed, the model did its number theory in Perl and independently re-derived a proof of Erdős problem #397 — a combinatorics question open for five decades until GPT-5.2 Pro first solved it this past January. Poolside notes that its model's construction is structurally different from the earlier published solution, and that its November 2025 knowledge cutoff precedes the first proof.
What the disclosed limitations and benchmark fine print reveal
Poolside deserves credit for disclosing limitations most labs bury. The model can overfit to its native harness and stumble on slightly different tool schemas in third-party agents, mangles JSON in nested tool arguments, and is prone to overthinking on competition math. There is currently no user-configurable thinking-effort dial — just on or off — and the gap between the modes is enormous: thinking lifts Terminal-Bench 2.1 from 60.4% to 70.2%, and DeepSWE from 16.5% to 40.4%, at substantially higher token cost.
Buyers should apply their own discounts to the comparison tables. Poolside's methodology takes the maximum of vendor self-reported scores, benchmark-author leaderboards, and third-party figures for competitors — a reasonable convention, but one that mixes harnesses and test conditions. On DeepSWE, notably, Poolside ran its own agent harness rather than the leaderboard's standard mini-swe-agent, a difference the company acknowledges makes scores less directly comparable. And the frontier remains clearly out of reach: closed models like GPT-5.6 Sol, at 88.8 on Terminal-Bench 2.1, and Claude Fable 5, at 88.0, along with the 2.8-trillion-parameter open-weight Kimi K3, at 88.3, sit well above Laguna S 2.1.
The deeper structural question is whether Poolside's "Model Factory" — the internal platform the company credits for its rapid release cadence — can sustain this pace as models scale. The trajectory so far is genuinely unusual: the April dual release of Laguna M.1 and XS.2, the July 2 refresh of XS 2.1, and now S 2.1, which the company says outperforms April's flagship M.1 at roughly a third of its active size. Remarkably, S 2.1 used the exact same pre-training data as XS 2.1, meaning nearly all the improvement came from scale, training fixes, and post-training across the company's corpus of 409,000 agentic and non-agentic training environments. Poolside says its next, larger Laguna model began pre-training last week.
For technical decision makers, Laguna S 2.1 is the most credible Western open-weight option to emerge in nearly a year for self-hosted agentic coding — with published evidence, a permissive license, broad ecosystem support, and an economics story built around hardware you can own. Whether it dents the dominance of Chinese open models will depend less on this release than on the ones that follow it.
Kant, for his part, has already told the world how he intends that story to end. Poolside is building toward a future where the most capable intelligence "can be owned and shaped by anyone," he wrote — and the company plans to keep shipping "until that future exists." In an industry where the biggest labs increasingly lock their best work behind an API, the most radical thing about Laguna S 2.1 may not be what it scores, but that anyone can download it and check.
Moonshot AI became the latest AI company to discover that launching a popular model is only half the battle. Less than two days after releasing Kimi K3, the company stopped accepting new subscribers after demand exhausted its available GPU capacity. Existing users will keep access while Moonshot expands its infrastructure and reopens subscriptions in batches.
Inference demand outpaces supply
The incident emphasizes how demand is outpacing available infrastructure. As AI models take on longer, more coding and agentic workloads, companies are finding they need more inference capacity than they anticipated.
“Kimi K3 has received far more love than we expected,” the official Moonshot account writes on X. “Over the past 48 hours, demand has pushed close to the limits of our current capacity. We’re adding capacity as fast as we can and will reopen new subscription spots in batches.”
Kimi K3 has received far more love than we expected, and our GPUs are feeling it.
Over the past 48 hours, demand has pushed close to the limits of our current capacity. To protect the experience of existing subscribers, we're temporarily pausing new subscriptions and…
“We’re adding capacity as fast as we can and will reopen new subscription spots in batches.”
For infrastructure engineers and developers, the resulting capacity crunch is a bold indication of why companies from OpenAI to Anthropic to Moonshot are rationing access instead of selling unlimited usage.
Open weights, closed capacity
At 2.8 trillion parameters, Kimi K3 is one of the largest open-weight models slated for release — Moonshot has scheduled the public weight drop for July 27. In Arena.ai’s Frontend Code Arena, K3 topped both OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5. On the broader Artificial Analysis Intelligence Index, it trails both, scoring 57 to Fable 5’s 60 and Sol’s 59. That doesn’t make it any easier to run.
Open weights let anyone deploy the model, but whoever hosts it still has to pay the inference bill. Coding activities tend to tie up GPU resources far longer than a typical chatbot interaction, making it harder to keep latency low as more developers pile on.
Lee argued that as developers build longer agentic workflows, lower inference costs are quickly “re-converted into higher total resource consumption,” shifting the bottleneck from compute to server memory.
Moonshot’s subscription pause is a sign that keeping enough inference capacity online once developers start using it at scale might be just as hard as building the model.
“Agent tasks are not one-off question answering; they continuously generate, read, and process tokens during ongoing tasks.”
China’s chip constraints compound crunch
For a company like Moonshot, this general industry bottleneck is compounded by regional infrastructure realities. Unlike traditional software companies, AI developers typically rent much of this computing power from cloud providers such as Alibaba Cloud, Tencent Cloud, and Huawei Cloud rather than owning extensive data-center infrastructure themselves.
The capacity crunch illustrates the mounting challenge facing Chinese AI developers as US export controls continue to restrict access to leading chip provider Nvidia’s most advanced AI chips. As a result, companies such as Moonshot depend on a combination of older chips and domestically produced alternatives. These constraints have forced Chinese developers to concentrate strongly on software tuning and more efficient use of computing resources to narrow the performance gap with US rivals.
Token economics under pressure
The scramble for computing power has fueled a data-center construction boom across China. Alibaba has committed more than $53 billion to AI and cloud infrastructure over three years, while ByteDance is reportedly considering spending as much as $70 billion this year on AI data centers and related infrastructure.
AI companies typically charge customers based on the number of tokens, or units of text, processed by a model, making token prices a key measure of operating costs.
According to Bernstein Research, Moonshot charges $3 per million input tokens and $15 per million output tokens for Kimi K3. This makes it about 40% cheaper than Anthropic’s Opus 4.8 and roughly 70% cheaper than Claude Fable 5.
“A world where there are only [two to three] dominant frontier labs with 90 percent inference margins is net negative for every other layer while being awesome for those [two to three] labs,” wrote Atreides Management founder Gavin Baker on X. Baker argued that models like Kimi K3, Grok 4.5, and Muse 1.1 could shift value away from the model layer and toward chipmakers, cloud providers, and the software companies building the infrastructure that serves AI models.
For developers, Moonshot’s subscription freeze serves as an architectural warning. The era of assuming infinite, cheap API access is ending.
“A world where there are only [two to three] dominant frontier labs with 90 percent inference margins is net negative for every other layer while being awesome for those [two to three] labs.”
A doctor walks into an exam room, says hello, and sits down. For the next few minutes, the patient explains what’s been going on. Maybe it’s pain. Maybe fatigue, or anxiety, or some symptom that’s hard to put into words at all. The doctor listens, asks questions, tries to connect the pieces. That’s how the […]
Login
