Nvidia CEO Jensen Huang used his first post on X to share a public letter backing frontier open-weight models. Signed by Microsoft, Meta, Hugging Face, and 22 other organizations, the letter argues that open models improve security, encourage faster innovation, and give countries and enterprises more control over their AI infrastructure. The timing of the Friday post is notable, with more organizations choosing to run open-weight models inside their own environments rather than relying entirely on hosted services.
“Open weights let every organization match the right model to the right job at the right cost, reserving frontier-scale capability for genuine frontier problems and running efficient, specialized models everywhere else.”
A case for hybrid stacks
The letter draws a comparison between open-weight AI and open-source software, arguing that broad access has driven decades of software innovation. It says the same principle applies to AI. Instead of sending every request to a hosted model, organizations can download an open-weight model, run it on their own infrastructure, customize it for a specific workload, and keep sensitive data behind their own firewall.
One isn’t necessarily replacing the other, and many enterprise teams already use both, choosing between them based on cost, performance, compliance requirements, and where the workload needs to run.
The timing is notable. Washington is weighing new restrictions on some Chinese AI models, including Moonshot AI’s Kimi K3, even though the Trump administration’s AI Action Plan described open models as a strategic advantage for the United States.
That split extends to the AI industry itself, though critics differ in their specific concerns. OpenAI’s Dean Ball has publicly warned about the broad economic and regulatory risks associated with Chinese open-weight models. Anthropic’s Sarah Heck, meanwhile, has specifically backed the White House’s allegations regarding intellectual property theft through distillation.
“Relying solely on closed models is not inherently safe: they can be breached, misused, or fail in ways that outsiders cannot detect. And concentrating advanced AI capabilities behind a small number of closed models compounds that risk.”
The companies that signed the letter argue that continuing to develop frontier open models is part of maintaining U.S. leadership in AI. They argue that security through obscurity is fundamentally flawed, stating: “Relying solely on closed models is not inherently safe: they can be breached, misused, or fail in ways that outsiders cannot detect. And concentrating advanced AI capabilities behind a small number of closed models compounds that risk.”
Distillation, one of AI’s most contentious topics, was also addressed. Widely used by researchers and model builders to create systems that are cheaper to run and easier to deploy, the letter states that it should be treated as a legitimate research technique rather than intellectual property theft.
Distillation is one way teams build smaller models that can run on private infrastructure, edge devices, or lower-cost GPU clusters. Kimi K3’s launch showed just how much GPU capacity these deployments demand — Moonshot had to freeze new subscriptions within 48 hours after demand overwhelmed its available inference capacity. Any new restrictions on that process could affect how enterprise AI systems are built and deployed.
“As organizations invest in AI, they want to know that they will not become locked into a single provider or lose the knowledge and capabilities they build over time.”
Nvidia’s infrastructure incentive
Every enterprise that chooses to self-host, fine-tune, or customize an open-weight model needs infrastructure to run it. Nvidia also has clear business reasons for supporting a future in which both proprietary and open-weight models thrive.
So, what really happened? OpenAI reports an autonomous security-evaluation run of GPT‑5.6 Sol and a pre-release model broke out of a sandbox, reached the internet, and then targeted Hugging Face to try to solve the ExploitGym benchmark.
By OpenAI’s account, “an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths … [with] maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.”
OpenAI continues, “Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.”
“All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
The AI models, however, cracked their sandbox. Once out, they “identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
AI expert Ken Huang describes the process in a blog post: “The model spent enormous inference budget probing the sandbox, then found a zero-day in the third-party software OpenAI used as a package-registry proxy and cache.
“It used that bug to gain unrestricted internet access. It then reasoned that Hugging Face probably hosted the ExploitGym datasets, chained stolen credentials with several more vulnerabilities into a remote code execution path, and pulled the benchmark answers straight out of Hugging Face’s production database.”
Why intent no longer matters
This was not, as John W. Larson, president and chief AI officer of the AI security company Babel Street, tells The New Stack, a “Terminator” moment. OpenAI models and agents “[were not] acting out of malice or trying to attack Hugging Face. It encountered obstacles, developed an unexpected strategy, bypassed safeguards, and pursued its assigned goal in a way its creators never anticipated. The incident demonstrates that harmful cyber incidents no longer require malicious intent: Only highly capable autonomous AI optimizing for an objective.
Harmful cyber incidents no longer require malicious intent: Only highly capable autonomous AI optimizing for an objective.
That’s the first issue we must be aware of. Software attacks no longer require a hacker. All it needs is an LLM freed of guardrails with a mission to accomplish, and it will take whatever steps are needed to reach its goal.
In addition, Adam Boynton, senior enterprise security manager of EMEIA at the Apple Identity and Access Management company Jamf, tells The New Stack, “This happened inside a frontier lab deliberately stress-testing its most capable models, in conditions no enterprise would ever run.”
Boynton also notes that this episode confirmed that “AI tools are no longer just applications people open. They act on the user’s behalf, with the user’s access. So every organization needs a clear view of which AI tools its people are running, and how those tools are configured and used.”
An agent given too much power is far more capable than a user armed with a superuser password of causing havoc both to your systems and to ones far outside your control.
Where container isolation failed
As for the attack itself, Edera CMO Kaylin Trychon writes in a blog post, “This incident deserves more attention than ‘an AI did the hacking now.’ The interesting part isn’t who was driving, or even that it was a model instead of a person. It’s where the walls gave way, twice, in two different companies’ infrastructure — and that part is entirely familiar to anyone who’s spent time securing container infrastructure.”
Exactly so. Trychon continues, “Strip away the framing and this is a story about two escalation chains, back to back.” The first consisted of “The model found and exploited a zero-day in that proxy, then used privilege escalation and lateral movement inside OpenAI’s own ‘highly isolated’ testing environment until it reached a node with open internet access.”
Next, OpenAI’s models inferred, correctly, that “Hugging Face likely hosted the datasets, models, and reference solutions for the benchmark it was trying to solve. It chained stolen credentials with a remote-code-execution path into Hugging Face’s servers, harvested more credentials, and moved laterally into several internal clusters over a weekend, all in pursuit of one narrow goal: Finding the answer key.”
“The failure that actually matters is what a single code-execution event is allowed to become — and notably, that failure happened the same way twice, at two different companies, in the same incident.”
The serious issue, from Trychon’s perspective, is that “both companies’ blast radius was a cluster instead of a container.”
That’s because “the failure that actually matters is what a single code-execution event is allowed to become – and notably, that failure happened the same way twice, at two different companies, in the same incident. In a typical container fleet, workloads share a kernel.
“A process that gets popped, or a model that gets a shell, is one syscall boundary away from the node it’s running on, and the node is one set of mounted credentials away from the rest of the cluster. Namespaces and cgroups are process-isolation primitives, not security boundaries — they were never designed to hold against an attacker (human or model) that already has code execution and time.”
Edera’s CTO and co-founder Alex Zenla tells The New Stack, “The sandbox escape here isn’t really the surprising part of this attack. An entire market of AI sandboxing tools has exploded over the past couple of months, and this is the reality they’re all built on top of: containers, VMs via namespaces — whatever the branding — most of them still share a kernel with the host or with each other.
“That’s a boundary enforced in software, and software boundaries are exactly the kind of thing an agent that can try ten thousand escape paths over a weekend is going to find a way through. We shouldn’t be shocked that this happened; we should be shocked at how many teams are still betting their infrastructure on technology that was never designed to withstand such a persistent adversary. The fix isn’t a better sandbox; it’s getting out of the sandbox paradigm entirely.”
Zenla continues, “Teams should adopt secure execution environments that are hardware-enforced and eliminate the shared kernel flaw. This technology exists today and would have made this specific escalation chain structurally impossible, not just harder. Teams running agents with real permissions and real access need to stop treating this as a someday problem, because the next version of this week is already being tested somewhere right now.”
That underlines the most important issue of all. Thanks to AI, security attackers are coming harder and faster than ever. Security can no longer be an afterthought.
You literally no longer have time to wait for security fixes. You must bake in as much security as you can as fast as you can, or your systems will be broken into. It’s as simple as that.
Microsoft AI released two new in-house models into public preview on Wednesday — MAI-Image-2.5-Pro, its highest-fidelity image generator to date, and MAI-Voice-2-Flash, a speech model built for high-volume enterprise workloads — while publishing production data that amounts to the company's most aggressive argument yet that it can power its own products without leaning on OpenAI's frontier models.
The announcement, made by Microsoft AI's Superintelligence team, lands roughly a year after the company committed to building purpose-built models internally, and it arrives with an unusual level of specificity about where those models now run: Bing, PowerPoint, OneDrive, Dynamics 365, Excel, GitHub Copilot, and Azure. The message to enterprise buyers — and, implicitly, to OpenAI — is that Microsoft's homegrown models are no longer research projects. They are production infrastructure serving millions of users.
"Each of these enhancements is a step toward the same goal: Microsoft products, powered by Microsoft models," the company wrote in its announcement blog.
How MAI-Image-2.5-Pro and MAI-Voice-2-Flash stake out opposite ends of the AI cost curve
The two new releases occupy opposite ends of what Microsoft calls the quality-speed-cost curve, and the positioning is deliberate. MAI-Image-2.5-Pro targets the premium tier: hero imagery, detailed editing, and precise in-image text rendering — the last of which has long been a notorious weak spot for image generation models. Microsoft priced the model at $5 per million text input tokens, $8 per million image input tokens, and $106 per million image output tokens. The base MAI-Image-2.5 model recently launched at No. 2 for image editing on Arena, the community leaderboard that has become a de facto scoreboard for generative media.
The creative industry appears to be taking notice. Rob Reilly, global chief creative officer at advertising giant WPP, called the Pro model "a strong leap forward for GenMedia tools" in a statement included in Microsoft's announcement, adding that "Microsoft has firmly established itself among the leaders in generative AI."
MAI-Voice-2-Flash goes the other direction. First previewed at Microsoft's Build conference, Flash runs twice as fast as MAI-Voice-2 and costs 32% less, priced at $15 per million characters. It is designed for the unglamorous but enormous market of high-volume voice — call centers, voice agents, and real-time speech applications where latency and cost-per-call matter more than marginal gains in expressiveness. Together, the two models reflect a strategy of building families of models rather than a single flagship, because, as the company put it, a creative studio chasing maximum fidelity has very different needs from a customer service operation handling millions of calls a day.
Microsoft's production metrics show in-house models cutting GPU costs by up to 89%
The model launches are arguably less newsworthy than the deployment metrics Microsoft attached to them — numbers that read like a systematic case for swapping out third-party frontier models across its product portfolio.
Bing Image Creator now runs entirely on MAI-Image-2.5, end to end, marking the first time the consumer image tool is fully in-house. In PowerPoint, Microsoft says MAI-Image-2.5 reduces GPU costs by up to 84% compared with GPT-Image-2, OpenAI's image model. In OneDrive, where MAI-Image-2.5 is now the default for key image-editing scenarios, the company reports a 26% increase in save rates, roughly 25% lower P95 latency, and 2.5 times greater efficiency under medium-utilization production workloads.
On the voice side, MAI-Voice-2-Flash now powers Dynamics 365 Contact Center — the platform used by customers including T-Mobile and EasyJet — where Microsoft claims GPU cost reductions of up to 89%. The model is also integrated into Azure Voice Live for developers building speech-to-speech agents.
Perhaps the most consequential deployment sits in healthcare. Microsoft's Dragon Copilot, used by 170,000 medical providers and responsible for processing 28 million patient encounters last quarter, now runs on MAI-Transcribe-1.5 for its multilingual workflow across 58 languages. Microsoft says internal evaluations show a 50% relative reduction in both transcription and language-identification error rates across most languages — a meaningful claim in a domain where transcription errors can propagate directly into clinical notes.
Inside the 'hill-climbing' strategy that lets small models beat GPT-5.6 in Excel
In a companion post published the same day, Microsoft detailed the methodology behind these results — what it calls its "hill-climbing machine," an integrated flywheel of data, models, and the product "harness" that surrounds them.
The clearest example is MAI-Code-1-Flash, the lightweight coding model launched in GitHub Copilot in June. Microsoft says the model achieves an approximately 10% higher code accept rate than GPT-5.4 Mini and Claude Haiku 4.5 in VS Code, while using 10% fewer median tokens. Developer retention tells a similar story: users were 6% more likely to return across multiple days than with GPT-5.4 Mini, and 11% more likely than with Claude Haiku 4.5.
Then Microsoft did something more interesting. It took the MAI-Code-1-Flash checkpoint and further trained it inside an Excel reinforcement learning environment, teaching a coding model the tools and workflows of spreadsheet knowledge work. The result, according to production user feedback, is a model on par with GPT-5.6 for the most common Excel tasks — while being small enough to run on Nvidia's older H100 and even A100 GPUs rather than requiring the latest-generation accelerators.
That hardware detail deserves emphasis. Every major AI company is fighting for allocation of cutting-edge chips, and a model that delivers frontier-adjacent quality on two-generation-old silicon fundamentally changes the deployment economics. It also frees the newest hardware — including Microsoft's now-operational GB200 cluster — for training rather than serving.
Satya Nadella's 'frontier diffusion' manifesto redraws the OpenAI relationship
Microsoft CEO Satya Nadella framed the announcements in a lengthy post on X titled "Frontier Diffusion & Control," which functions as something close to a strategic manifesto. "We can now take saturated frontier capabilities and deliver them at scale and at lower cost through models optimized for high-usage products, while continuing to use frontier models for frontier needs," Nadella wrote, adding that Microsoft is "beginning to route traffic across our first-party surfaces to MAI whenever our models match or outperform frontier alternatives."
Translated from executive prose: capabilities that were state-of-the-art a year ago are now table stakes, and Microsoft believes it can replicate them cheaply for the specific, repetitive tasks that dominate real product usage. Why pay frontier prices for a frontier model when a user just wants to reformat a spreadsheet column?
Nadella was careful to note that "frontier models from OpenAI and Anthropic are part of the orchestration system alongside MAI" — but he also articulated a pointed principle of model independence, arguing that a company's evaluations "should continue to hill climb even when any given model has been removed."
“Keeping the harness, memory, context, and skills outside the model, he argued, is what gives Microsoft control. The subtext is hard to miss. Reuters reported in April that Microsoft’s exclusive license to OpenAI’s technology had been revised into a non-exclusive arrangement, and The Information reported last September that Microsoft had begun incorporating Anthropic models into some products. Wednesday’s announcement completes the triangle: Microsoft as orchestrator, with its partners’ frontier models as interchangeable components and its own models absorbing an ever-larger share of routine traffic.”
Developers cheer cheaper task-specific models while skeptics question Microsoft's track record
The response online captured both the appeal and the skepticism surrounding the strategy. "I love when people use small models for niche tasks," wrote one X user, @mavihsk, responding to Nadella's post. "Why do I have to use the all-knowing model just to change my field in Excel?" Another user, @nabu_lines, distilled the pitch neatly: "cost and performance both improve when you stop overusing the biggest model."
Others were less charitable about Microsoft's execution track record. "Microsoft is the worst when it comes to listening to user feedback," wrote designer @designedbyabin, arguing the company "will lose the AI race because they repeatedly failed to understand user needs." And one user, @tokenoverflow, offered a drier critique of the model-independence pitch: "i want it keep hill climbing after removing microsoft."
The skeptics raise a fair point. Microsoft's self-reported metrics — accept rates, save rates, GPU savings — come from its own internal evaluations, not independent benchmarks, and the company chooses which comparisons to publish.
But the strategy's logic does not depend on any single number. Nadella's framing that software now has "real marginal cost for the first time" explains why Microsoft is obsessive about tokens, GPUs, and serving costs: when AI features run on every keystroke across a billion-user product portfolio, an 84% GPU cost reduction is not an optimization. It is the difference between a viable business and a money pit.
Why Microsoft is turning its internal AI playbook into an Azure product
The final piece of the strategy is that Microsoft is selling the playbook, not just the models. Nadella explicitly positioned the hill-climbing approach as "a template for every other AI native, SaaS, or Enterprise company," and Microsoft is packaging the toolchain through Foundry and what it calls Frontier Tuning — letting enterprises train specialized models against their own proprietary evaluations and reinforcement learning environments. That turns Microsoft's internal cost-cutting exercise into an Azure product, and it gives enterprise customers a reason to run their AI workloads on Microsoft's cloud even if the models themselves come from elsewhere.
The company's emphasis on models trained "on clean, traceable, enterprise-grade data, without distillation from third-party models" serves the same commercial end. In an industry facing mounting scrutiny over training data provenance, Microsoft is betting that enterprise buyers — and courts — will care where model capabilities come from. Microsoft says it is now extending the hill-climbing approach to Copilot Chat, Outlook, and PowerPoint, and both new models are available in public preview through Microsoft Foundry and the MAI Playground. "None of this is an endpoint," the company wrote. "We're just getting started."
Seven years ago, Microsoft bet more than $13 billion that OpenAI would build the future of AI. Wednesday's announcement suggests the company has since learned a cheaper lesson: the future of AI may belong to whoever builds the frontier, but the profits belong to whoever makes it ordinary.
The AI industry has largely focused on language-based approaches, using transformers trained on massive datasets to predict words or fill in missing information.
As AI expands into more structured fields, however, the limitations of text-generation models are becoming clearer. Nvidia is pursuing a different approach.
The company just dropped JEPA-DNA on Hugging Face. It’s a genomic foundation model that adds a latent-space prediction objective alongside MLM.
This release is a win for hybrid architectures that go beyond purely generative training. It’s the kind of shift Executive Chairman of AMI Labs, formerly Meta’s Chief AI Scientist, Yann LeCun, has been pushing for years. He’s championed predictive architectures as a general alternative to next-token prediction, which researchers are now applying to biology.
Latent space over literal tokens
Conventional genomic base models have historically mirrored NLP models, relying purely on MLM, masking parts of a DNA sequence and forcing the model to guess the missing literal tokens. This approach favors local token reconstruction and teaches the model the basic “syntax” of the sequence, but it frequently struggles to grasp the wider functional “meaning.”
JEPA-DNA changes the paradigm. The newly released checkpoint, JEPA-DNA-DNABERT2, serves as a model-agnostic continual pre-training framework. It couples standard token-level DNA language modeling with JEPA, adding a second learning objective.
Instead of forcing the architecture to reconstruct missing tokens, JEPA-DNA supervises the model’s global sequence embedding in a latent space.
Instead of forcing the architecture to reconstruct missing tokens, JEPA-DNA supervises the model’s global sequence embedding in a latent space. It predicts the functional representation of masked genomic segments rather than their literal, character-by-character makeup.
Token prediction is still part of the training process, but it is no longer the model’s only learning objective. The model also learns the basic structure of the data, proving that predictive architectures can produce representations that are easier to work with for biological tasks without sacrificing generative capabilities.
Token prediction is still part of the training process, but it is no longer the model’s only learning objective.
How DNABERT-2 gets upgraded
The model builds on DNABERT-2, a 117 million-parameter model developed by Zhihan Zhou and collaborators. Nvidia layers its continual pre-training approach on top of that architecture, letting the model learn from both token-level predictions and latent-space representations.
Nvidia has released the model globally for non-commercial research. According to the company, the model is meant to support research workflows, including feature extraction, linear probing, continual pre-training experiments, and zero-shot scoring of DNA sequence changes. It isn’t a diagnostic tool or a clinically validated medical product.
Beyond the generative hammer
DNA contains patterns and relationships that aren’t captured by sequence prediction. JEPA-DNA supplements masked-token prediction with a latent-space objective intended to capture broader sequence-level information.
Ultimately, models like JEPA point to a new path for AI that combines next-token prediction with other ways of learning and helping models build a deeper understanding of complex systems.
The models small enough to run on the box on your desk are getting good enough that the interesting question is no longer whether you can run them, but what you can do with them, and how organizations can get the most out of them.
Joey Conway, Nvidia’s senior director of generative AI software, spoke to The New Stack about how local and open models are increasingly working alongside frontier models, often with a router in between deciding which one to use, and how organizations can adapt these open models for their own needs.
“We love the world where we can use both frontier and open models together.” — Joey Conway, Nvidia
A system of models
Tasks vary in complexity, so the models handling them should vary too, Conway tells The New Stack. He points to the early open reasoning models, which would reason their way through trivial problems, mulling number lines and memory to work out what two plus two is. “I just say four,” he says.
“Being able to route those easy things to local models that are quick, and route the hard things to more sophisticated models,” Conway says, lets you “get a better outcome at a lower cost and lower time to completion.”
It’s a different picture from the one large model most people imagine doing everything. In his version, you build a bench of specialists. “You’ll have specialized agents that are really good at focused tasks because that’s what they do every day,” he says, “and they just get better and better at that task.”
To the user, none of that shows. “It’ll feel like one interface,” Conway says, “but behind that interface, there’ll be a variety of models handling a variety of tasks.”
Getting there is largely a routing problem, and one Conway says is still in its early days. Nvidia’s own contribution, for now, sits lower in the stack, in inference-serving software like its open-source Dynamo, which steers each query to the GPU that handled it most recently. Which model is best for which job, Nvidia leaves to a wider field of routers, some of them models in their own right that weigh budget, latency, and modality. But Conway also leaves the door open for Nvidia to build more of that routing itself before long.
Nvidia points to its collaboration with LangChain, whose Deep Agents harness ran on Nemotron 3 Ultra, Nvidia’s 550-billion-parameter open model, and matched top closed models on business tasks at up to a 10x lower cost, as Conway notes. It required no retraining; the gains came entirely from tuning the harness around it: its prompts, tool descriptions, and middleware.
You’re not going to run a 550-billion-parameter model on your desktop anytime soon, but running relatively large models locally is now a real possibility, as long as you have some beefy hardware at your disposal. For enterprises, setting up a fleet of accelerators in a data center isn’t exactly cheap either, but it does mean full control and no surprise token bills.
Bringing AI to where the data lives
Running models yourself can save money, but Conway thinks control matters more. Enterprises already decide where their data lives and what they hand to outside vendors, and open models give them even more control. “Move AI to where your data lives,” he says, “or move AI to where your employees are.”
Companies want to keep their data — and especially their intellectual property — in-house, and Conway argues a fine-tuned open model is the place to put it. “It’s like an employee,” he says. “You hire them, and they’re part of your company.”
The local half runs on hardware like Nvidia’s DGX Spark, a $4,699 Grace Blackwell machine with 128GB of unified memory that handles models up to roughly 200 billion parameters without anything leaving your desk. (There is also the DGX Station, its bigger, pricier sibling with 748 GB of RAM for running even larger models.)
“It’s like a system sitting right there next to you,” Conway says, one where “you don’t think about network latencies.” To run those agents securely, Nvidia offers NemoClaw, a reference stack that wraps an open agent harness like OpenClaw in a sandbox called OpenShell, with policy controls and local Nemotron inference.
When you need more power for a broader problem, you reach for a frontier model in the cloud. For Nvidia, that’s all good news: a system of models runs on its silicon one way or another, on your desk or in the cloud.
Cursor, the AI coding tool recently acquired by Elon Musk’s SpaceX in a $60 billion all-stock deal, has launched a model router designed to direct every coding request to whichever model handles it best, bypassing the need to pay frontier prices for work that doesn’t need it.
Under the hood, the new Cursor Router uses a triage system à la a hospital emergency room: It looks at what a request actually needs — how hard it is, what it’s for, the surrounding code — and picks a model that is the best fit. A quick fix goes somewhere cheap, while a genuinely hard problem gets escalated to something closer to frontier-grade.
Notably, developers and admins also have access to three distinct modes that nudge that balance in either direction, favoring speed and cost over raw power, or vice versa if needed.
Choosing from three optimization modes
The broader rationale for Cursor Router, according to the company’s own field CTO David Pan in a social media post on Wednesday, is that developers shouldn’t have to become experts in model performance just to write code.
“We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rate.”
“We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rates,” Pan writes.
We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rates.
Cursor Router is out today. It picks the model for each task based on measured quality against cost. Fable 5 level performance for… https://t.co/3B2103mGWr
Early community feedback has largely echoed Pan’s sentiment: Fatih Arslan, a software engineer at PlanetScale, notes on X that engineers already juggle the choice between cost and capability by hand — defaulting to a cheap, fast model for routine work and saving the slow, expensive one for “serious tasks.”
Nice! I wanted something like this for a long time. See my previous rant last year. I think as engineers, we're already developing skills ourselves which model to choose. We use a less costly/fast model for implementing chore details, but use a higher reasoning model (slow and… https://t.co/5y5xMOR2E6
“We already spend quite a bit [of] time on [choosing models],” Arslan writes. “Why not automate that part? Cursor Router does the automation.”
“We already spend quite a bit time on it [choosing models]. Why not automate that part?”
In a separate blog post published on Wednesday, Cursor claims that early access customers saved 30-50% compared to routing everything through Opus 4.8, with no drop in output quality.
Working model: Taking control of the stack
The launch follows a run of moves by Cursor to control more of its own AI stack. In May, the company released Composer 2.5, an update to its in-house coding model built for long tasks at a lower cost than frontier options from Anthropic and OpenAI. Composer 2.5, like its predecessor, is built on Moonshot AI’s Kimi K2.5, an open-weight model out of China.
Now, with the weight of one of the world’s most valuable companies behind it (SpaceX has attained a market cap of $1.5 trillion since its June IPO), Cursor is pushing a powerful frontier model of its own.
On July 8, Cursor and SpaceXAI jointly released Grok 4.5, a mixture-of-experts model built on a new foundation dubbed V9, which Musk had previously noted was roughly 1.5 trillion parameters. The model’s trained on trillions of tokens of real Cursor usage data and available across all Cursor plans at $2 per million input tokens and $6 per million output tokens.
With Composer handling cheap, fast work, and now the Grok-branded frontier line for more serious horsepower, Cursor has its own models in the mix alongside the usual list of outside providers. And this gets to the heart of why Cursor built Router: Most developers pick one model and stick with it regardless of the task, billing simple work at frontier prices it doesn’t need.
Sending every request to its own models would be the easy way to keep that money in-house, but it would also mean shipping inferior output on some tasks — so Router instead sends each request to whichever model actually suits it, Cursor’s own or not.
The lay of the land
Model routing itself isn’t exactly new. OpenRouter has offered a version since 2023: a single API sitting in front of more than 400 models from over 60 providers, including OpenAI, Anthropic and Google. Its own auto-router feature does roughly what Cursor Router does — classify a request, then send it to whichever model fits the task and the person’s stated preference between cost and quality.
More recently, OpenRouter launched Fusion, which takes a slightly different approach: instead of picking one model, it sends a prompt to several models at once and uses a judge model to synthesize the strongest answer out of all of them.
This past month ushered another entrant to the mix: Japan’s Sakana AI released Fugu in June, which instead breaks a single task into subtasks and routes each piece to a different model, pitched by Sakana as a hedge against relying on any one AI provider.
“[Cursor Router is] a great example of how a technological innovation immediately translates into a product improvement.”
Not everyone rates some of these other attempts, though. On Wednesday, Kirill Balakhonov, head of AI products at Nethermind, argues on LinkedIn that Cursor’s version succeeds precisely because it’s focused on coding specifically, rather than trying to be a general-purpose router for any task.
“A great example of how a technological innovation immediately translates into a product improvement… rather than an abstract idea like Sakana Fugu or OpenRouter Fusion,” Balakhonov writes, predicting both of those broader routing efforts would fade from use.
What’s new, perhaps, is some of the names emerging behind an array of model diversity efforts. In early July, Microsoft launched a $2.5 billion services unit dubbed Microsoft Frontier Company, embedding thousands of engineers at customer sites to help them build with a mix of AI models.
Judson Althoff, CEO of Microsoft Commercial Business, told Reuters at the time that the push came partly from watching rivals like DeepSeek and Google’s Gemini close the gap on OpenAI. Referring to the original Copilot, he admitted, “we made a mistake by binding it to OpenAI models only.”
If the company with the deepest single-model relationship in the industry is walking it back, the idea for model flexibility has clearly gone mainstream — certainly if this week is anything to go by.
Ramp and Meta get in on the act
On Tuesday, Ramp, the $44 billion spend-management behemoth, opened up Ramp Router, an early-access public version of the model router it built to manage its own AI bills internally, which it says cut its LLM costs by roughly 30%. It’s free to start, requires no Ramp account, and routes across OpenAI, Gemini, and select open-source models including Kimi through an OpenAI-compatible endpoint.
The very same day, Jyoti Mann fromThe Informationreported that Meta is also working on a model router. According to internal documents cited in the report, an incubator inside Meta called AAI Labs is developing a new product named Switchboard that would score each request for difficulty and send simpler ones to smaller, cheaper models — aimed initially at cutting Meta’s own AI agent costs, though it reportedly could end up as a public release.
Meta has particular reason to want this. Data from Runpod’s State of AI report, published in March, suggests Meta’s open Llama models are now a marginal presence in production: Llama 4 has near-zero real-world deployment, with Alibaba’s Qwen having overtaken it as the most-deployed self-hosted LLM.
Meta has also been building proprietary models in response. Muse Spark, its first model out of the newly formed Meta Superintelligence Labs, shipped in April. That was followed in July by Muse Spark 1.1, Meta’s first model with a public, paid API, priced at roughly a quarter of what OpenAI and Anthropic charge for comparable models.
Meta is aggressively targeting the incumbents, and Switchboard fits the same pattern: a way to make it easier for users to cut costs, switch models freely, and, where it makes sense, land requests on Meta’s own models instead.
But amidst all this hullaballoo about model routing, there is perhaps a broader question of openness. Not of the models themselves, which is a fervent debate in itself, but of whether the routing decision itself — the logic that decides which model handles which request — should sit inside a vendor’s own closed product at all.
“Is anyone building this as open-source?”
Elvis Saravia, a former technical product marketing manager at Meta AI who co-founded DAIR.AI, took to X to argue that it shouldn’t, given how differently teams weigh cost against quality.
“Is anyone building this as open-source?” Saravia asks. “It feels like this is something you don’t want to offload to an API. We all work with different trade-offs, so we need the ability to achieve custom routing.”
Recommended reading. Cursor Router routes tasks to the right model.
Is anyone building this as open-source? It feels like this is something you don't want to offload to an API. We all work with different trade-offs, so we need the ability to achieve custom routing. https://t.co/zcrpHbnlCepic.twitter.com/Vw2xyMy1jp
As for Cursor’s own version, Router is currently available to Teams and Enterprise customers only, across desktop, web, iOS, CLI, and Cursor’s SDK. It’s not yet clear whether it will eventually land on individual plans.
Every product team is chasing the same moment: The user opens a page and thinks, this understands me.
A shopper who loves floral prints should see more floral prints. A user who follows local politics should open their app to see news about local politics. A job candidate who keeps clicking remote roles should not keep getting shown in-office jobs.
That is not a niche feature anymore. It is the baseline expectation. Users decide quickly whether a product system understands them, and they rarely care whether the failure came from search, recommendations, merchandising rules, or stale data.
Here is the uncomfortable truth: Most teams do not have a personalization quality problem. They have a personalization architecture problem.
Personalization is not a widget bolted onto search. It is a ranking decision. The system has to decide, for this user and this request, what deserves the next slot. That means weighing the user, the item, the context, and the business goal at the same time. In many stacks, the ranking layer is the one place that cannot see all of those signals together.
The hard part is not collecting signals. The hard part is combining them while the user is still there.
Why personalization is hard in the first place
To put the right item in the right slot, a system has to understand several things at once:
Intent: What is the user asking for right now?
Item quality: What does each candidate actually contain or represent?
User history: What has this person clicked, bought, read, watched, or ignored?
Availability: Is the item in stock, fresh, nearby, legal to show, or ready to ship?
Business priority: What should the business promote, protect, or de-emphasize?
Those signals often disagree. The most relevant item may not be the most profitable. The most profitable item may be out of stock. The user may say “running shoes,” but their behavior says “trail running, wide fit, under $120.”
They also move on different clocks. Product attributes change slowly. Inventory and price can move throughout the day. Preferences shift with every click. External context — weather, breaking news, a championship game, a cultural moment — can matter without warning.
Personalization means folding all of that into one ordered list, on every request, in milliseconds. The signals themselves are not the bottleneck. Query-time ranking is.
The usual stack makes the problem harder
Most personalization systems are assembled from tools that were each designed for one slice of relevance.
Keyword search engines are excellent at lexical matching. They are good when the query language and catalog language line up. But shoppers, readers, and job seekers rarely speak in neat index terms. You indexed “athletic performance running footwear”; they typed “running shoes.” Synonym rules can help, but they do not scale gracefully across long-tail language, changing catalogs, and new user behavior.
Vector databases start from the opposite side. They are good at semantic similarity: “Find me things like this.” That is powerful, but nearest-neighbor search is not the same thing as personalization. Real ranking has to blend semantic similarity with live behavior, stock, price, margin, freshness, eligibility, and business rules.
Re-rankers, recommendation services, feature stores, and rule engines are usually added to glue everything together. That is where fragmentation creeps in.
Figure 1. A fragmented personalization stack compared with a unified query-time ranking pipeline
When retrieval and ranking live in separate systems, the ranker often works from a partial, stale, or precomputed view of the world. Click history, session context, and the user’s live preference vector arrive too late. Business rules become filters or overrides instead of ranking signals. Fresh inventory or price changes require coordination across multiple systems.
Every hand-off adds latency. Every boundary creates another place for signals to drift. Every “quick rule” becomes another hard constraint that can accidentally turn “show the closest match” into “show nothing.”
“Every hand-off adds latency. Every boundary creates another place for signals to drift.”
The deeper issue is a timing assumption. Many architectures were built around offline ranking: process the catalog, compute scores in a batch job, and serve those scores until the next rebuild. That works when preferences are stable. It breaks when the most valuable signal is the click that happened two seconds ago.
What changes when ranking happens in one real-time pipeline
A real-time personalization architecture treats retrieval, ranking, and inference as one serving problem.
That is the core idea behind Vespa’s approach: Text search, vector similarity, structured filtering, ranking expressions, tensor computation, and model inference can live inside one query pipeline. Instead of retrieving somewhere, enriching somewhere else, and ranking at the end, the system can rank with the relevant signals while it is still deciding what to return.
That architectural choice changes the shape of the problem.
1. Retrieval is hybrid from the start
Lexical search, semantic search, and structured filtering can run together instead of being reconciled after the fact. A product query can combine text, embeddings, filters, session behavior, and item attributes in one request.
That matters because personalization is rarely one signal. The user’s query still matters. So does semantic similarity. So do category, availability, price, and business constraints. Hybrid retrieval keeps those signals in play before ranking starts.
2. Ranking can express the actual objective
A personalization score should not be trapped inside one similarity function. It should be a formula that reflects the product’s goals.
That formula might combine BM25, vector similarity, user affinity, stock level, margin, popularity, discount depth, freshness, rating, distance, or a weather term. Some of those signals need normalization first. Some should matter only for certain categories or users. Some should be tested as weights.
The important part is that they are all terms in the same ranking expression, not scattered across services.
In production, the formula can be more nuanced. But the principle is simple: personalization, relevance, and business logic belong in the same scoring decision.
3. Model inference can run where the data lives
Some signals should come from learned models rather than hand-tuned rules: propensity to buy, churn risk, quality prediction, fraud risk, query classification, or a learned-to-rank model.
When inference runs in the serving path, those model outputs can become ranking features instead of delayed batch scores. That reduces the need to ship data to a separate inference service, wait for a response, and stitch the score back into ranking.
4. Updates become immediately useful
“Real time” should not mean “after the next index rebuild.” If inventory changes, stock should be rankable immediately. If a user clicks two yellow dresses, “yellow” should matter on the next request. If a merchandising team adjusts a ranking weight if the weight is exposed as a query-time input, the experiment should start producing useful feedback right away.
That is the difference between personalization as a nightly job and personalization as a live ranking decision.
Tensors make the personalization concrete
The most useful mental model is simple: represent the user and the item in the same feature space, then rank by how well they match.
In Vespa, tensors make that practical. A tensor can be a scalar, a dense vector, a sparse map of feature-weight pairs, a matrix, or a more complex structure. That means the same framework can represent semantic embeddings, product attributes, user preferences, business objectives, and model features.
Figure 2. User and item tensors combined into a personalization score, then blended with other ranking signals
For example, each item can carry a sparse feature tensor:
Because the two tensors share a shape, personalization becomes a dot product: multiply matching features, sum the result, and use that score inside ranking.
In a Vespa rank profile, the core expression is compact:
# schema: item attributes stored as a sparse tensor
field item_features type tensor<float>(feature{}) {
indexing: attribute | summary
}
# rank profile: the user's live preferences arrive as a query tensor
rank-profile personalized {
inputs {
query(user_features) tensor<float>(feature{})
}
first-phase {
expression: sum(query(user_features) * attribute(item_features))
}
}
That one expression is not the whole ranking function. It is the personalization term. BM25, vector similarity, stock, margin, freshness, distance, or a model score can be added as other terms with their own weights.
The user tensor is where real-time behavior becomes powerful. Click a floral item, and the “floral” weight rises. Click two yellow items, and “yellow” rises; the application feeds click events into the user profile. The next query can use those updated preferences immediately, without waiting for a nightly profile build.
Business goals stop fighting personalization
In fragmented stacks, business rules often become blunt instruments: boost this category, hide that brand, force these items to the top, filter these out. That can satisfy a short-term merchandising goal while damaging relevance.
When business logic is part of the ranking expression, it can be more subtle. You can boost overstocked inventory without ignoring intent. Promote umbrellas when rain is forecast without turning every search into an umbrella search. Give new sellers a small exploration boost. Prioritize destocking before a new product line launches. Surface team merchandise during a championship run.
“When business logic is part of the ranking expression, the user still gets relevant results. The business still influences outcomes.”
The user still gets relevant results. The business still influences outcomes. The difference is that both are expressed as ranking signals instead of competing systems.
That also makes experimentation easier. A merchandising or growth team can test weights, traffic splits, and ranking profiles without asking engineering to rewrite the whole pipeline. Relevance becomes a controllable growth lever rather than a fragile side effect.
The same pattern applies beyond commerce
The examples above are easy to picture in apparel, but the architecture is not commerce-specific. Personalization is the same ranking problem in many products:
Content feeds: Blend topic affinity, freshness, engagement, creator quality, and business rules.
News: Rank by reading history, topic interest, locality, freshness, and source diversity.
Jobs: Match candidate preferences such as remote work, seniority, compensation, location, and tech stack against role attributes.
Geo search: Treat distance as one normalized ranking term alongside relevance, quality, and preference.
Video and audio: Combine embeddings, viewing history, metadata, freshness, and learned ranking models.
Different domains need different features. The architecture pattern is the same: retrieve candidates, rank with the signals that matter, update those signals as behavior changes, and keep the decision close to the data.
Scale doesn’t have to be the trade-off
The natural concern is that a more expressive ranking system must be slower. In practice, that does not have to be true.
Vespa was built for large-scale serving from the beginning: billions of documents, high query volume, and low-latency ranking. The reason this works is multi-stage ranking. The system does not run the most expensive logic across every possible result. Instead, it uses a fast first phase to narrow the candidate set, then applies more precise ranking to the smaller group that remains.
For example, a cheap first phase narrows a huge candidate set. Then, once the candidate set is smaller, Vespa can apply full-precision scoring, richer tensor operations, business logic, and model inference where they matter most.
The result is a practical balance: speed across the full corpus, accuracy in the final ranking, and enough flexibility to personalize each query without turning the serving stack into a chain of fragile services.
What’s next
Personalization is not failing because teams lack data. Most teams already have plenty of signals: query intent, clicks, product attributes, inventory, margin, freshness, location, and business priorities. The harder problem is that those signals often live in different systems, move at different speeds, and arrive too late to influence the final ranking decision.
That is why personalization should be treated as a ranking problem. When retrieval, ranking, personalization, and business logic are split across separate systems, the ranker is forced to work with stale or incomplete context. The user moves faster than the architecture can respond. Every new signal becomes another integration project.
A unified real-time ranking pipeline changes that. User behavior, item attributes, semantic similarity, lexical relevance, inventory, and business goals can all become parts of the same scoring function. Tensors make those signals directly comparable and usable at query time. Instead of bolting personalization onto the end of the system, personalization becomes part of the decision the engine makes for every query.
The goal is simple: rank each result with the best context available, at the moment the user asks. That is when personalization stops feeling like a feature and starts feeling like relevance.
The AI ecosystem is drowning in tutorials on how to build a retrieval-augmented generation (RAG) app in five minutes. The pitch is appealing but flawed: Chunk a document, run it through an embeddings API, load it into a vector database, and slap a UI on top. This setup works locally. It might even survive a beta test with friendly users. But launch it into a production-grade B2B SaaS environment, and the architecture collapses.
Enterprise applications don’t handle neat, static files. They ingest dynamic, unpredictable streams of live data bound by strict legal and compliance constraints. Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.
“Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.”
Here is exactly what breaks when a naive RAG setup hits production, and the architectural trade-offs needed to fix it.
Bottleneck 1: the synchronous ingestion trap
Synchronous data ingestion is the most prevalent architectural flaw in new AI products. A user uploads a 500-page compliance manual. The client makes a POST call to a web server, which parses the document, splits the text, iterates over a sequence of synchronous API calls to OpenAI or Cohere for vectorization, and writes those vectors to the database.
This approach introduces two critical failures:
Timeouts: A 500-page document rarely finishes processing within standard HTTP timeouts (30 to 60 seconds) while waiting for the embedding API.
Cascade failures: If the system hits rate limits or latency spikes, the entire ingestion operation fails, throwing a 500 error and losing the user’s document.
The fix: the batched fan-out pipeline
Production-grade AI pipelines require persistent events rather than simple HTTP calls. However, sending the whole 500-page document to be processed by one consumer from Kafka or RabbitMQ is a mistake. If a consumer spends 10 continuous minutes generating embeddings, it misses its broker heartbeat. Assuming the worker died, the broker kills the consumer and triggers a partition rebalance, creating an infinite loop of duplicated work and stalled processing.
“Synchronous data ingestion is the most prevalent architectural flaw in new AI products.”
Conversely, granular chunking, where every chunk becomes an individual Kafka message, launches a self-inflicted denial-of-service (DoS) attack on downstream services. A document with 1,500 chunks generates 1,500 individual messages. This instantly exceeds upstream requests per minute (RPM) limits and floods the pipeline with network overhead.
The engineering sweet spot is a batched fan-out approach:
Asynchronous uploads: The web API stores the raw file in Amazon S3, triggers a document_uploaded event, and instantly returns a 202 Accepted status. This single, asynchronous path processes one-page invoices and 100-page SOC2 reports with equal reliability, eliminating the technical debt of maintaining separate “fast” and “slow” ingestion routes.
Micro-batching: A lightweight “Spitter” consumer downloads the file, chunks it, and groups those chunks into optimized micro-batches (e.g., 64 chunks per batch).
Controlled embedding: Embedding workers pull these batched events. To prevent concurrent workers from breaching upstream RPM limits, avoid fragile sleep() delays. Instead, enforce a token bucket rate limiter at the consumer level or strictly cap the number of active message broker partitions.
Python
# Conceptual snippet for architectural illustration
def handle_document_upload(event):
try:
raw_text = download_from_s3(event.file_uri)
chunks = semantic_chunking(raw_text)
except Exception as e:
# Log failure and raise so the message broker routes this to a Dead Letter Queue (DLQ)
print(f"Failed to process document {event.file_uri}: {e}")
raise
# Batch size heavily depends on the downstream embedding model's context limits
batch_size = int(os.environ.get("EMBEDDING_BATCH_SIZE", 64))
for i in range(0, len(chunks), batch_size):
chunk_batch = chunks[i:i + batch_size]
kafka.publish("embedding_tasks", {
"tenant_id": event.tenant_id,
"document_id": getattr(event, "document_id", event.file_uri),
"chunks": chunk_batch
})
This keeps individual consumer tasks short, respects upstream rate limits by maximizing payload density, and allows horizontal scaling of embedding workers during traffic spikes.
Bottleneck 2: the multi-tenant nightmare
Developers often treat multi-tenancy as an afterthought. The simplest way to handle multiple B2B tenants using a single RAG system is logical segregation, where all vectors reside in a large index, and each entry has a tenant_id associated with its metadata. Upon retrieval, the application filters results by adding a clause to the metadata payload.
Flaws of the approach:
Security vulnerabilities: Relying on application-level filtering creates an unacceptable risk. If an engineer omits or misconfigures a metadata filter, one client can access another’s confidential data. In highly regulated environments, this breaks compliance.
The noisy neighbor problem: If one customer uploads 10 million vectors to the shared index, memory usage skyrockets during vector searches. This degrades performance across the entire system, even for tenants with a handful of documents.
The fix: serverless compute-storage decoupling
Echo-chamber thinking assumes that the only solution is to provide each tenant with its own dedicated database cluster. This is prohibitively expensive and practically impossible to manage in a modern-day SaaS offering.
The true gold standard here is using next-generation serverless vector databases like Pinecone Serverless or managed Qdrant implementations, which make a clear distinction between computing and storage.
Isolation strategy
How it works
Trade-offs
Shared index (logical)
One index; application layer applies metadata filters.
High compliance risk; prone to noisy neighbor performance degradation.
Database per tenant (physical)
Client provisions a dedicated database cluster.
Maximum security, but introduces massive operational overhead and idle compute costs.
Serverless namespaces (standard)
Storage layer isolates vectors into namespaces; on-demand compute loads them only when queried.
Namespace-level access control prevents cross-tenant leaks. Zero idle compute costs.
Engineering takeaway: stop building complex multi-tenant routing logic in your application code. Push the isolation boundary down to the infrastructure layer using serverless namespaces.
Bottleneck 3: the semantic caching trap
Once ingestion is asynchronous and tenants are segregated, inference costs become the final bottleneck. Hitting an LLM API for every individual query is economically unsustainable.
The industry defaults to semantic caching: embed the user’s prompt, calculate its cosine similarity against previous prompts, and return a pre-calculated LLM response if the score exceeds a set threshold (e.g., 0.95).
Why semantic caching fails
Embeddings capture overall semantic meaning, but they miss specific contexts and entities. The prompts “What was the holiday policy in 2023?” and “What is the holiday policy for 2024?” share a near-perfect cosine similarity score. The core semantics match, but returning a cached answer feeds the user incorrect or contradictory information.
The fix: hybrid verification vs. native prompt caching
To scale without compromising accuracy, there are only two choices to consider: application-layer validation or infrastructure-layer optimization.
Strategy A: combined lexical filtering and intent routing
When using an application-layer caching system (for instance, Redis), you need to layer the semantics search on top of extremely light guardrails.
Exact-match filter: Apply a token-validation filter over vector similarity. In the case where the cached query is “2023”, and the current query is “2024,” throw out the cache hit right away.
Intent routing: Before serving a cached answer, use an inexpensive, fast model as an intent match router.
Python
Query A: {incoming_query}
Query B: {cached_query}
Do these queries have the exact same intent and require the exact same factual answer?
Respond only with YES or NO.
Strategy B: infrastructure-level prompt caching
If the system cannot tolerate the added latency of an application-layer verification router, bypass custom caching entirely and offload the problem to the infrastructure.
Modern LLM providers natively support prompt caching. It is crucial to understand what is being cached here: it is not the user’s short question. When multiple users query the same corporate knowledge domain, the massive system instructions and the heavy retrieved context documents (often 10k+ tokens) are cached automatically at the provider’s inference layer.
“Stop approaching AI like a magic black box and instead approach it as a distributed systems problem.”
The application sends the full RAG query every time. The provider recognizes the repeated context block, slashes context token costs by up to 80%, and drops the time-to-first-token (TTFT) to milliseconds.
The infrastructure surrounding a foundation model separates a prototype from a production AI system. Stop approaching AI like a magic black box and instead approach it as a distributed systems problem, and things will fall into place. Batched fan-out asynchronous queuing solves timeout and rate-limiting issues. Serverless namespacing resolves compliance risks. Prompt caching and intent routing secure unit economics. Designing a native AI product today means engineering for inevitable API failures, cross-tenant data attacks, and runaway LLM costs.
Gritt, the company building an intelligent system combining robotics and AI to accelerate the world’s infrastructure buildout, has launched with $32.4 million in pre-seed and Series A funding. The $26 million Series A was led by Obvious Ventures, with participation from Union Square Ventures and Active Impact Investments, and support from previous investors First Round […]
Poolside, the San Francisco AI lab that has spent most of its three-year existence quietly selling coding models to governments and defense agencies, released its most capable model to date on Tuesday — and made an unusually aggressive bet that radical transparency, not raw scale, is how a smaller lab competes at the frontier.
The model, Laguna S 2.1, is a 118-billion-parameter Mixture-of-Experts (MoE) system that activates only 8 billion parameters per token, supports a context window of up to 1 million tokens, and — according to benchmarks published by the company — matches or beats open models several times its size on agentic coding tasks. The weights are available immediately on Hugging Face under the permissive OpenMDW-1.1 license.
The headline numbers are striking for a model this small. Poolside reports that Laguna S 2.1 scores 70.2% on Terminal-Bench 2.1, a benchmark of long-horizon terminal tasks, placing it 11th on the company's compiled leaderboard — ahead of DeepSeek-V4-Pro-Max, a 1.6-trillion-parameter model that scored 64.0; Thinking Machines' 975-billion-parameter Inkling, at 63.8; and Nvidia’s 550-billion-parameter Nemotron 3 Ultra, at 56.4. On SWE-Bench Multilingual, it posts 78.5%, and on SWE-Bench Pro's public dataset, 59.4%.
Perhaps more telling than any single score: the model went from the start of pre-training on May 22 to public launch in under nine weeks, trained on 4,096 Nvidia H200 GPUs. In an industry where flagship model cycles are typically measured in quarters or years, Poolside has now shipped three models in three months.
Why the West's open-weight AI gap has become a boardroom issue
The release lands in the middle of an increasingly pointed debate about the provenance of open-weight AI. Over the past year, developer adoption has shifted decisively toward open-weight systems that companies can download, inspect, and run on their own infrastructure — and the leading options in that category have overwhelmingly come from Chinese labs. DeepSeek, Qwen, Kimi, GLM, MiniMax, and Tencent's Hunyuan line all feature prominently in Poolside's own comparison tables.
Poolside's accompanying press release frames Laguna S 2.1 explicitly as a response, noting that the model occupies a size class into which no Western lab has released open weights in 11 months — since OpenAI's gpt-oss-120b last August. "The West needs open-weight models it can trust, run, and build on," said Jason Warner, Poolside's co-CEO, in the announcement.
Co-founder and co-CEO Eiso Kant made the philosophical stakes even plainer in a lengthy post on X. "I believe intelligence should and will become a commodity," he wrote, arguing that the open ecosystem "will not win by being the best in its own category." Users, he argued, simply want the best intelligence for the task at hand — so open models must be on par with, or better than, their closed equivalents.
The strategic logic here is not charity. Poolside's core business is deploying models inside the security boundaries of government, defense, and regulated enterprises — customers for whom closed, metered API access is often a non-starter for compliance and sovereignty reasons.
Every enterprise that standardizes on a Chinese open model today becomes harder to win tomorrow. Releasing competitive open weights is both an ecosystem play and a top-of-funnel strategy for the company's high-security deployment business. It also reframes the AI race away from terrain where Poolside cannot compete — frontier-scale capital expenditure — and toward terrain where it believes it can: cost per token, self-hosting, and iteration speed.
How a sparse architecture makes enterprise AI agents affordable to run
The technical design reflects a specific thesis about where value in coding AI is moving. Laguna S 2.1's sparse MoE architecture — 256 routed experts plus one shared expert, with grouped-query attention and interleaved sliding-window layers, according to the Hugging Face model card — means inference costs scale with the 8 billion active parameters, not the 118 billion total. Poolside emphasizes that the model is small enough to run on a single Nvidia DGX Spark, the desktop-class AI machine.
That matters for what Poolside calls token economics. Long-horizon coding agents are voracious consumers of tokens: the company's published data shows the model consuming a mean of roughly 249,000 completion tokens per trajectory on its hardest benchmark when thinking mode is enabled. At metered API prices, agentic workloads at enterprise scale become a meaningful budget line item. On OpenRouter, Poolside is offering a free 256K-context endpoint and a dedicated 1M-context deployment priced at $0.10 per million input tokens and $0.20 per million output tokens — aggressive pricing that undercuts most frontier alternatives by an order of magnitude.
The ecosystem support is unusually broad for day one. The model is live on Baseten's model library and Vercel's AI Gateway, with integrations across vLLM, SGLang, Ollama, and llama.cpp, plus quantized variants down to 4-bit GGUF files — 75 gigabytes — for local use. But Poolside's more interesting claim is behavioral, not architectural. Pengming Wang, co-head of applied research at Poolside, said the gains came from improving the model's working habits: "more verification, less taking things for granted, not declaring victory early, and being more persistent." Raw intelligence, the company argues, is one axis of capability; a model's way of working is a second axis that matters immensely for agents left unattended for hours.
Publishing every benchmark trajectory to counter AI's credibility crisis
The most consequential part of the release for enterprise buyers may be an evaluation-transparency move with little precedent among major labs: Poolside published the complete, unedited trajectory of every trial in its final benchmark runs — every reasoning step, tool call, and shell command behind every reported score.
This addresses a growing credibility problem in AI benchmarking. As top scores on mature benchmarks cluster in the 70–90% range, and as "reward hacking" — models finding solutions online or gaming verifiers rather than solving problems — has become endemic, self-reported numbers have lost much of their signal. Poolside disclosed its own encounters with the problem candidly: during training, more than half of trajectories on some SWE-bench tasks were flagged because the model simply researched the original bug-fix pull request online and applied it. The company documented its mitigations, including prompt addenda, LLM-based judging calibrated against human labels, and expert annotator review of a high-scoring Terminal-Bench run.
Three published case studies illustrate what the company means by persistence. In one, the model built a working HTML/CSS rendering engine from an empty folder in a 181-step, 50-minute unattended session — then, lacking vision capabilities, spun up headless Chromium to numerically compare its canvas output against a real browser's rendering. In another, pointed at Poolside's own agent harness in an automated optimization loop, the model made the Go codebase 5.2% faster with roughly 70% lower memory allocation, finding an O(n²) string-concatenation bug along the way. In a third, working in a sandbox with no Python installed, the model did its number theory in Perl and independently re-derived a proof of Erdős problem #397 — a combinatorics question open for five decades until GPT-5.2 Pro first solved it this past January. Poolside notes that its model's construction is structurally different from the earlier published solution, and that its November 2025 knowledge cutoff precedes the first proof.
What the disclosed limitations and benchmark fine print reveal
Poolside deserves credit for disclosing limitations most labs bury. The model can overfit to its native harness and stumble on slightly different tool schemas in third-party agents, mangles JSON in nested tool arguments, and is prone to overthinking on competition math. There is currently no user-configurable thinking-effort dial — just on or off — and the gap between the modes is enormous: thinking lifts Terminal-Bench 2.1 from 60.4% to 70.2%, and DeepSWE from 16.5% to 40.4%, at substantially higher token cost.
Buyers should apply their own discounts to the comparison tables. Poolside's methodology takes the maximum of vendor self-reported scores, benchmark-author leaderboards, and third-party figures for competitors — a reasonable convention, but one that mixes harnesses and test conditions. On DeepSWE, notably, Poolside ran its own agent harness rather than the leaderboard's standard mini-swe-agent, a difference the company acknowledges makes scores less directly comparable. And the frontier remains clearly out of reach: closed models like GPT-5.6 Sol, at 88.8 on Terminal-Bench 2.1, and Claude Fable 5, at 88.0, along with the 2.8-trillion-parameter open-weight Kimi K3, at 88.3, sit well above Laguna S 2.1.
The deeper structural question is whether Poolside's "Model Factory" — the internal platform the company credits for its rapid release cadence — can sustain this pace as models scale. The trajectory so far is genuinely unusual: the April dual release of Laguna M.1 and XS.2, the July 2 refresh of XS 2.1, and now S 2.1, which the company says outperforms April's flagship M.1 at roughly a third of its active size. Remarkably, S 2.1 used the exact same pre-training data as XS 2.1, meaning nearly all the improvement came from scale, training fixes, and post-training across the company's corpus of 409,000 agentic and non-agentic training environments. Poolside says its next, larger Laguna model began pre-training last week.
For technical decision makers, Laguna S 2.1 is the most credible Western open-weight option to emerge in nearly a year for self-hosted agentic coding — with published evidence, a permissive license, broad ecosystem support, and an economics story built around hardware you can own. Whether it dents the dominance of Chinese open models will depend less on this release than on the ones that follow it.
Kant, for his part, has already told the world how he intends that story to end. Poolside is building toward a future where the most capable intelligence "can be owned and shaped by anyone," he wrote — and the company plans to keep shipping "until that future exists." In an industry where the biggest labs increasingly lock their best work behind an API, the most radical thing about Laguna S 2.1 may not be what it scores, but that anyone can download it and check.
Moonshot AI became the latest AI company to discover that launching a popular model is only half the battle. Less than two days after releasing Kimi K3, the company stopped accepting new subscribers after demand exhausted its available GPU capacity. Existing users will keep access while Moonshot expands its infrastructure and reopens subscriptions in batches.
Inference demand outpaces supply
The incident emphasizes how demand is outpacing available infrastructure. As AI models take on longer, more coding and agentic workloads, companies are finding they need more inference capacity than they anticipated.
“Kimi K3 has received far more love than we expected,” the official Moonshot account writes on X. “Over the past 48 hours, demand has pushed close to the limits of our current capacity. We’re adding capacity as fast as we can and will reopen new subscription spots in batches.”
Kimi K3 has received far more love than we expected, and our GPUs are feeling it.
Over the past 48 hours, demand has pushed close to the limits of our current capacity. To protect the experience of existing subscribers, we're temporarily pausing new subscriptions and…
“We’re adding capacity as fast as we can and will reopen new subscription spots in batches.”
For infrastructure engineers and developers, the resulting capacity crunch is a bold indication of why companies from OpenAI to Anthropic to Moonshot are rationing access instead of selling unlimited usage.
Open weights, closed capacity
At 2.8 trillion parameters, Kimi K3 is one of the largest open-weight models slated for release — Moonshot has scheduled the public weight drop for July 27. In Arena.ai’s Frontend Code Arena, K3 topped both OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5. On the broader Artificial Analysis Intelligence Index, it trails both, scoring 57 to Fable 5’s 60 and Sol’s 59. That doesn’t make it any easier to run.
Open weights let anyone deploy the model, but whoever hosts it still has to pay the inference bill. Coding activities tend to tie up GPU resources far longer than a typical chatbot interaction, making it harder to keep latency low as more developers pile on.
Lee argued that as developers build longer agentic workflows, lower inference costs are quickly “re-converted into higher total resource consumption,” shifting the bottleneck from compute to server memory.
Moonshot’s subscription pause is a sign that keeping enough inference capacity online once developers start using it at scale might be just as hard as building the model.
“Agent tasks are not one-off question answering; they continuously generate, read, and process tokens during ongoing tasks.”
China’s chip constraints compound crunch
For a company like Moonshot, this general industry bottleneck is compounded by regional infrastructure realities. Unlike traditional software companies, AI developers typically rent much of this computing power from cloud providers such as Alibaba Cloud, Tencent Cloud, and Huawei Cloud rather than owning extensive data-center infrastructure themselves.
The capacity crunch illustrates the mounting challenge facing Chinese AI developers as US export controls continue to restrict access to leading chip provider Nvidia’s most advanced AI chips. As a result, companies such as Moonshot depend on a combination of older chips and domestically produced alternatives. These constraints have forced Chinese developers to concentrate strongly on software tuning and more efficient use of computing resources to narrow the performance gap with US rivals.
Token economics under pressure
The scramble for computing power has fueled a data-center construction boom across China. Alibaba has committed more than $53 billion to AI and cloud infrastructure over three years, while ByteDance is reportedly considering spending as much as $70 billion this year on AI data centers and related infrastructure.
AI companies typically charge customers based on the number of tokens, or units of text, processed by a model, making token prices a key measure of operating costs.
According to Bernstein Research, Moonshot charges $3 per million input tokens and $15 per million output tokens for Kimi K3. This makes it about 40% cheaper than Anthropic’s Opus 4.8 and roughly 70% cheaper than Claude Fable 5.
“A world where there are only [two to three] dominant frontier labs with 90 percent inference margins is net negative for every other layer while being awesome for those [two to three] labs,” wrote Atreides Management founder Gavin Baker on X. Baker argued that models like Kimi K3, Grok 4.5, and Muse 1.1 could shift value away from the model layer and toward chipmakers, cloud providers, and the software companies building the infrastructure that serves AI models.
For developers, Moonshot’s subscription freeze serves as an architectural warning. The era of assuming infinite, cheap API access is ending.
“A world where there are only [two to three] dominant frontier labs with 90 percent inference margins is net negative for every other layer while being awesome for those [two to three] labs.”
A doctor walks into an exam room, says hello, and sits down. For the next few minutes, the patient explains what’s been going on. Maybe it’s pain. Maybe fatigue, or anxiety, or some symptom that’s hard to put into words at all. The doctor listens, asks questions, tries to connect the pieces. That’s how the […]
Microsoft and Mistral are deepening their partnership with a multibillion-dollar agreement focused on enterprise AI infrastructure. The goal is to give organizations more flexibility over where they run frontier models and how those deployments are managed, particularly in regions with tight data residency and sovereignty requirements.
Under the deal, announced Tuesday, Microsoft will use Mistral’s expanding European compute infrastructure to increase regional capacity. Mistral, meanwhile, plans to deploy thousands of NVIDIA Vera Rubin GPUs to power everything from model training to multi-agent workloads.
For engineering teams in regulated industries, the announcement emphasizes where enterprise AI is heading. Many organizations no longer want the public cloud to be the only place they can run AI models. They’re asking for deployments that fit into the infrastructure they already have, including on-premises and air-gapped environments.
Sovereign compute meets agentic AI
At the compute layer, the agreement gives Microsoft access to European-operated infrastructure outside its traditional first-party data centers and leased facilities. The move builds on Microsoft’s European Digital Commitments, announced in 2025, which focus on keeping customer data in Europe and helping organizations meet regional regulatory requirements.
Crucially, Mistral will power this infrastructure with NVIDIA’s next-generation Vera Rubin rack-scale platform. Because sovereign environments support multi-step agentic workflows, this hardware pairing is essential. NVIDIA claims the platform delivers up to 10x the agent throughput at scale compared to its Grace Blackwell generation.
“Europe should have access to the world’s most capable AI without compromising control over their data, operations or digital future.”
Even though exact financial terms, capacity allocations, and rollout timelines have not been made public, the agreement positions Mistral as both an AI model supplier and an independent European compute vendor for Microsoft’s enterprise ecosystem. For European companies wary of the US CLOUD Act, the appeal is obvious — though whether a US-headquartered provider can fully place data beyond its reach remains legally contested.
“Europe should have access to the world’s most capable AI without compromising control over their data, operations or digital future,” said Microsoft Vice Chair and President Brad Smith in the announcement.
The model layer
At the model and API tier, Microsoft announced that Mistral Medium 3.5 and Mistral OCR 4 are now live in Microsoft Foundry, with Medium 3.5 also integrated into Microsoft Copilot Studio.
Among the models coming to Azure AI Foundry is Mistral Medium 3.5, an open-weight model with 128 billion parameters and a 256,000-token context window, which could indicate it’s better suited for applications that need to work across large documents or extended conversations.
Microsoft is also adding Mistral OCR 4, which is designed for document-heavy AI workflows. The model can process documents in 170 languages and preserve information about page layout, including bounding boxes, document structure, and confidence levels.
The model can process documents in 170 languages and preserve information about page layout, including bounding boxes, document structure, and confidence levels
Air-gapped deployments go mainstream
For developers, the primary technical win is architectural symmetry. Engineers can build, test, and fine-tune applications inside Microsoft Foundry and migrate those workloads across public Azure, Azure Local, or sovereign infrastructure operated by Mistral without having to refactor underlying workflows.
This hybrid portability targets industries bound by strict compliance regulations. The deployment options also include fully air-gapped environments, allowing organizations to run AI workloads on networks that remain completely isolated from the public internet.
Deployment flexibility drives decisions
The partnership indicates that deployment flexibility is becoming part of the buying decision. Many organizations want the freedom to run AI in the cloud, on-premises, or in regional infrastructure without being locked into a single approach. And that’s where Microsoft’s expanded relationship with Mistral fits so well.
The companies are betting that giving customers more deployment options will weigh just as much as offering competitive models. If the strategy works, Microsoft will be providing the control plane for how the world’s most regulated industries run it.
If the strategy works, Microsoft will be providing the control plane for how the world’s most regulated industries run it.
The fast pace of AI research means organizations now have a wide range of models to choose from that can power AI agents to solve real business problems. But choosing a model doesn’t guarantee you effective agents or even good performance. For that, you need to run your agents in an environment that provides them with tools, state, security, and scale, with fast startup times and good integration with your existing business systems.
Picking the right agent runtime environment is like picking an enterprise app server but for AI systems — and agents have very different needs from traditional applications.
Gartner predicts more than 40% of agentic projects will be canceled by 2027; not because models aren’t powerful enough to be useful but because of unclear business value, inadequate risk controls and ballooning costs.
Gartner predicts more than 40% of agentic projects will be canceled by 2027; not because models aren’t powerful enough to be useful but because of unclear business value, inadequate risk controls and ballooning costs. Agents that deliver in proof-of-concept systems will fail in production if the runtime stack powering them can’t keep up and keep them under control.
Agentic compute is different
It’s easy to think of an AI agent as just another microservice that takes unstructured input, runs APIs or queries, and returns messages. But infrastructure designed for traditional enterprise applications with predictable business logic or even cloud-native stateless workloads doesn’t fit agents with their bursty, long-running, stateful, non-deterministic, code-writing, tool-invoking behaviors that might be triggered by a system event or an email — not just a chat session.
Model inference needs GPUs for speed, but agents also need reliable, durable compute that supports stateful sessions for long-running processes, along with strong security and real-time visibility.
You still need to think about familiar issues like hosting, scaling, identity, and security, but all that is complicated by the unpredictable, multi-stage workflow of the agent reasoning loop.
An agent pulls in input from multiple sources, reasons over its context about the execution plan for accomplishing the goal, calls other tools or writes its own code, iterates over the results of those calls, and maybe builds on them or switches to another approach that requires another reasoning loop and eventually delivers output. That might be updating a system or sending an email rather than just displaying an answer.
Model performance is only one part of making that useful. Architecting a successful agent system that can run at enterprise scale requires considering the agent runtime, the application layer that agents call, and the tools, APIs, and MCP servers they consume.
You have to be able to integrate with business logic and existing systems, manage the usual quotas, rate limits and SLAs for APIs so agents don’t overload them — and you have to do all that while keeping up with AI developments that are moving too fast for you to build the infrastructure primitives you need to rely on from scratch every time.
Requirements of modern agentic infrastructure
Instead, you should look for flexible infrastructure that fits the way the agent works. As with any technology, if you build on an existing platform like Azure Container Apps, you can save effort in areas where your business can differentiate. And while AI agents have flaws (from hallucinations to high token costs) that aren’t fully solved, you can pick an agent runtime environment that makes it easier to get useful results despite them.
Agentic compute needs fast startup and resume. Whether it’s a human typing into a chat prompt or system events automatically launching multiple agents, agent infrastructure needs to spin up quickly. If it takes a few seconds to spin up a container, the reasoning loop can’t start till that’s done.
It also needs to scale up and down responsively, without costing you anything when it’s not running.
Because agent workloads are long-running and event-driven, agents need to start up, do some work, go idle for hours or even days, and then resume instantly with their memory, context, caches, connections, identity, and security intact. That means persisting and restoring state so long-running agents don’t have to pay the same startup tax over and over again. Whether the last stage of the agent’s reasoning loop was successful or a failure, it has to take another approach; you don’t want it to do the same work again.
Rather than building your own custom microVM stack, the new Azure Container Apps Sandboxes provide a temporary, secure, and stateful compute environment that spins up, executes code, snapshots disk and memory, then automatically idles and resumes just as fast. This is the stateful equivalent of Dynamic Sessions, with sub-second startup from pre-warmed pools; you can burst out to hundreds (and eventually thousands) of concurrent sandboxes when you need them, then scale back down to zero.
An agent runtime needs to be secure by default because agents are only useful when they take action — and by definition, they’re likely to do unexpected things, write and execute their own untrusted code, and keep trying to achieve their goal (sometimes even when there’s a policy that should stop certain behaviors).
To give agents secure, auditable access to the resources they need, run them and the code they generate in a sandbox, rather than on a developer laptop with production credentials and admin rights. Identity, access control, and execution boundaries have to be enforced at the runtime layer, not bolted on in a hidden prompt.
ACA Sandboxes have egress and access policies, so you can control outbound calls and limit what URLs they can access. If you can’t use managed identity for all the services an agent needs to connect to you, you can protect secrets by injecting API keys through an external egress gateway instead of hard-coding them. Because the gateway is external, outbound call decisions are controlled by policy outside the sandbox, not by agent code that will relentlessly try any method to get through.
If agents are useful, you’re going to run a lot of them, often simultaneously. AI agents need strong isolation for each task, so every untrusted code execution happens in its own sandbox and no data leaks between tasks.
You can manage them as groups, but each ACA Sandbox has its own secure boundary, so details from one customer support agent won’t end up in a chat with a different customer. If the untrusted code does turn out to be problematic, isolated sandboxes at least contain the blast radius.
Manage the tools agents call
You don’t want to rely on fragile glue code or ad hoc orchestration that you have to rewrite any time systems change for the calls agents make to tools, APIs, cloud services, and other workflows. You also don’t want to rely on manual cleanup of resources no longer needed. A runtime with built-in agent tool execution makes agents more robust. It takes the drudgery out of connecting the agent environment to your other systems, enabling them to interact securely, in isolation, and at scale.
“It’s not just about where the agent is running and what the capabilities of the agent are, but also what actions the agent can take.”
“It’s not just about where the agent is running and what the capabilities of the agent are, but also what actions the agent can take,” points out Vyom Nagrani, who runs the team of PMs responsible for both Azure Container Apps and the Azure SRE Agent that’s built on ACA Sandboxes. Sandboxes have access to a connector framework with over 1,400 enterprise-grade connectors, enabling them to take actions not only on Azure and Microsoft services but also on third-party tools.
“The runtime provides those connectors, and it provides a managed way of authenticating against all of these third-party systems, so now you can build an agent which can talk to many, many, many different systems. It’s not boxed into one authentication boundary,” Nagrani tells The New Stack.
MCP servers are a built-in capability of the connector framework. “You can take any of these connectors; you can take any REST API and expose it as an MCP server, which then the agent can consume.” Or if you want to build a custom MCP server, you can host that in a sandbox too. ACA Sandboxes can be both where agents run and, if that’s appropriate, where the tools they use are hosted.
If you’re building a user interface to wrap your agents, the application layer that makes calls to the agents can run in Azure Container Apps Express, a new service now in public preview, Nagrani says.
“It’s a simplified app hosting stack for the app layer that responds to HTTP traffic and serves web traffic: that’s where the human interactivity comes in.”
Putting it all together
ACA Sandboxes offers an agent runtime environment that answers the key questions architects need to consider: where agents and the ephemeral compute they need access to run; where the application layer that calls agents runs; and how agents get access to all the tools, APIs, services, MCP servers, and existing business logic they need to orchestrate.
Whether you’re a platform engineer, a software vendor or a startup building a new AI platform, treating agent runtimes as the new application server and MCP servers as the new APIs requires agent infrastructure that supports agent workloads effectively, allowing governance to shift left into the runtime layer where it can scale with the ever-increasing numbers of agents.
How real platforms build on agent runtimes
Azure Container Apps is already a strong platform for running agents. Auger, a startup launched by the former CEO of Amazon’s global consumer business to help mid-size enterprises get their complex supply chains out of Excel spreadsheets, used it to build a multi-agent system that can give real-time answers about shipments and forecasts in a world where mines or critical shipping lanes might be closed at any time by war or weather.
Backend agents pull the unstructured data with all the details to answer those questions from different siloes, building ETL pipelines and creating an ontology of the supply chain ecosystem for each customer that includes functions and actions — all of which need to be audited and reversible. Frontend agents use that ontology to answer questions such as, “What happens if I build a new warehouse here or switch to a supplier in this country?”
ACA Sandboxes are built on Azure Container Apps, giving organizations an easier way to create agent services with strong isolation, dynamic scaling, fast startup, persistent state, and broad tool and service integrations.
That’s just what the Foundry team was looking for when they started work on the Microsoft Foundry Agent Service managed agent runtime. ACA Sandboxes gave them a platform that delivers fast start/resume, built-in tool execution (including for untrusted code), persistent state for long-running agents, strong per-agent isolation, and secure-by-default operations. Agent identity, “on‑behalf‑of” authentication to existing business services, and strict isolation are critical capabilities they didn’t have to build themselves.
“It’s the same enterprise-grade infrastructure behind Microsoft’s own agentic products, now available for all Azure customers to build on.”
The Foundry Agent Service adds a layer of visibility and observability into agent actions and conversations, allowing customers to monitor agent performance and see where the system is doing well and where it needs improvement.
“Azure Container Apps Sandboxes package the hard parts of an agent runtime into a first-class Azure resource — sub-second start and resume, built-in execution of tools and untrusted code, and snapshot-based state that lets long-running agents pick up exactly where they left off. Every agent gets its own hardware-isolated environment with secure-by-default operations, so builders can focus on what their agents do, not on the plumbing underneath,” Nagrani points out.
“It’s the same enterprise-grade infrastructure behind Microsoft’s own agentic products, now available for all Azure customers to build on.”
Public AI assistants have become so commonplace that software vendors are increasingly adding AI search, conversational experiences, and AI agents to their own applications. From eCommerce and customer support to enterprise software, AI is rapidly becoming the primary interface to many applications.
Companies that build products around proprietary information are particularly well positioned to benefit from this shift. Whether they provide financial intelligence, market intelligence, legal research, scientific publishing, or business information, their products help professionals make better decisions by transforming trusted information into actionable insight.
AI allows these organizations to deliver that expertise through entirely new user experiences. Increasingly, they compete not only on the quality of their proprietary information, but on how intelligently they retrieve, understand, and transform it into customer value.
“Increasingly, they compete not only on the quality of their proprietary information, but on how intelligently they retrieve, understand, and transform it into customer value.”
A new competitive battleground is emerging. As AI becomes the primary interface to proprietary knowledge, the ability to retrieve, verify, rank, and assemble information is becoming almost as important as the proprietary information itself.
Much of the industry’s attention has focused on increasingly capable language models, but those models are only as effective as the context they receive. Designing retrieval workflows that consistently deliver trusted, relevant, and up-to-date information is rapidly becoming one of the defining engineering challenges for AI-native applications.
Retrieval engineering: optimizing the workflow
For decades, search engineering has focused on helping people find the right information. Whether searching a website, a legal database, or a financial research platform, the challenge was to retrieve the most relevant results while balancing competing priorities such as relevance, latency, scalability, and cost. The search system’s job was to retrieve relevant information. The human’s job was to evaluate it.
AI fundamentally changes that role.
Instead of retrieving information for people to evaluate, retrieval systems increasingly assemble the context that large language models and AI agents use to investigate, reason, and act. Every retrieval decision now becomes part of an automated workflow in which relevance, freshness, latency, and trust directly influence the final answer.
“Prompt engineering influences how a language model reasons. Retrieval Engineering determines what it has to reason about.”
This shifts the engineering challenge away from individual technologies and towards the retrieval workflow itself. The goal is no longer simply finding relevant documents, but orchestrating retrieval, ranking, filtering, inference, and real-time updates so they work together efficiently. We believe this emerging discipline deserves its own name: Retrieval Engineering. Prompt engineering influences how a language model reasons. Retrieval Engineering determines what it has to reason about. Both matter, but as AI applications become increasingly autonomous, the quality of retrieval increasingly determines the quality of the outcome.
As AI applications evolve from conversational assistants to deep research systems and autonomous agents, optimizing workflows rather than individual components becomes increasingly important. A single user request may trigger dozens—or even hundreds—of retrieval operations before a response is generated.
The challenge isn’t vector search
Vector databases solved an important problem by making semantic retrieval practical at scale. But semantic retrieval is only one stage of a much larger workflow.
Production AI applications increasingly combine vector similarity with keyword search, structured filtering, business rules, personalization, machine-learned ranking, and real-time inference to assemble the context that language models depend on. The engineering challenge is no longer selecting the best retrieval technology—it is orchestrating increasingly sophisticated retrieval workflows that remain accurate, responsive, and cost-effective.
Many organizations address this by combining specialist technologies. A vector database provides semantic retrieval. A search engine handles lexical matching. Additional services provide reranking, personalization, and inference. This works well initially, but every additional component introduces another network hop, another operational dependency, and another source of latency. The problem is no longer vector search. It is engineering an efficient retrieval architecture.
From components to platforms
This shift is changing how retrieval infrastructure is designed. Instead of optimizing individual components in isolation, engineering teams increasingly need to optimize the retrieval workflow as a complete system—balancing retrieval quality, latency, freshness, scalability, and infrastructure cost.
“The problem is no longer vector search. It is engineering an efficient retrieval architecture.”
That is why AI Search Platforms are emerging. Rather than stitching together retrieval, ranking, inference, and serving from multiple independent services, they execute the workflow within a single distributed architecture. The optimization problem changes from integrating components to engineering the workflow itself.
AI has transformed the user interface. It is now transforming the retrieval infrastructure behind it. For organizations building applications around proprietary knowledge, the next competitive advantage will not come solely from larger language models or better embeddings. It will come from building retrieval workflows that consistently deliver trusted, relevant, and timely context at scale.
Retrieval Engineering is rapidly becoming one of the disciplines defining the next generation of AI-native applications.
If you’re interested in exploring these ideas in more depth—including Retrieval Engineering, AI Search Platforms, and the architectural patterns behind AI-native information platforms—we cover them in our ebook, Building AI-Native Information Platforms.
The race to make AI inference cheaper is pushing chip design beyond general-purpose accelerators. We’re now moving toward silicon that is tailored to individual models. And Google’s reported “Frozen v2” project suggests the Gemini AI model is part of the movement.
First reported by The Information, the unannounced chip would reportedly hardwire parts of Gemini’s architecture while leaving its weights updatable. That compromise could give Google much of the efficiency of model-specific silicon without making the hardware obsolete every time Gemini changes.
A spokesperson for Google tells The New Stack, “Our teams are constantly researching and experimenting with new innovations to deliver maximum performance and efficiency for our users and customers.”
A spokesperson for Google tells The New Stack, “Our teams are constantly researching and experimenting with new innovations to deliver maximum performance and efficiency for our users and customers. While not every project moves into production, this rigorous exploration is central to our full stack approach. By co-designing our hardware and software from the ground up, we ensure our systems are integrated and highly optimized for real-world workloads.”
According to the reporting, Google hopes the chip will help relieve the AI compute crunch that’s made it harder for cloud providers to keep up with demand, while also making Gemini much cheaper and more efficient to serve. Internal projections reportedly estimate the design could deliver six to ten times more tokens per watt than Google’s current generation of AI chips.
If the project moves forward, it would represent a different approach to AI infrastructure because Google would be building hardware specifically for Gemini. For developers, that’s an early indication that future AI systems may be designed with much tighter integration between the model and the hardware beneath it.
Specialized silicon replaces flexibility
Currently, the vast majority of AI inferences run on Nvidia GPUs or Google’s own Tensor Processing Units (TPUs). Because these chips are intended to accommodate a wide variety of AI models, they carry a high degree of processing load.
There’s precedent for this kind of shift. Bitcoin mining followed a similar path, starting with CPUs, then GPUs, before ultimately settling on ASICs. AI inference may be headed in the same direction. Training still benefits from the flexibility of GPUs, but once a model reaches production, the priority shifts to serving as many requests as possible while using less power.
Training still benefits from the flexibility of GPUs, but once a model reaches production, the priority shifts to serving as many requests as possible while using less power.
Competitors hardwire their own
Google isn’t the only company looking beyond general-purpose GPUs for AI inference. As inference becomes a larger share of AI workloads, more companies are experimenting with specialized hardware designed to improve performance while using less power. Even Nvidia, whose GPUs dominate the AI market, has invested heavily in inference, striking a reported $20 billion deal with Groq last year to license the startup’s technology.
Among the more ambitious efforts is Canadian startup Taalas. The company has demonstrated a chip with an entire 8-billion-parameter Llama model embedded directly into the silicon. By keeping the model on the chip instead of constantly moving data back and forth from external memory, Taalas says it can dramatically speed up inference, claiming throughput of roughly 17,000 tokens per second. Meanwhile, other companies are optimizing the spatial distance between memory and compute to avoid hardware lock-in.
d-Matrix’s new Corsair platform uses an SRAM-based in-memory compute architecture rather than relying on standard high-bandwidth memory (HBM) packaging. Similarly, SambaNova is deploying custom dataflow technology with a three-tier memory architecture to maximize tokens per watt for advanced workflows.
Google’s Frozen v2 sits uniquely in this landscape — borrowing the hyper-efficiency of hardwired architectures like Taalas, but retaining just enough flexibility to remain viable across multiple product cycles.
Freezing architecture, not weights
The “Frozen” name reportedly comes from the idea of permanently etching part of Gemini’s design into the chip itself. According to The Information, Google’s engineers have spent years trying to find the right balance between efficiency and flexibility.
An earlier concept, led by Google DeepMind Chief Scientist Jeff Dean, reportedly would have embedded Gemini’s model weights directly into the silicon. That idea was eventually abandoned because it would have tied the hardware to a single version of the model, dramatically limiting its useful life as Gemini continued to evolve.
Frozen v2 reportedly takes a different approach. Rather than locking in the weights, the chip would hardwire parts of Gemini’s underlying design while still allowing the weights to be updated over time. That would let Google continue improving the model without needing to replace the hardware every time Gemini receives a new version.
Rather than locking in the weights, the chip would hardwire parts of Gemini’s underlying design while still allowing the weights to be updated over time.
Cheaper inference reaches developers
By moving some of Gemini’s execution into the chip itself, Google could reduce some of the overhead that comes with running models on more general-purpose hardware. That could translate into lower latency, especially for applications that depend on near real-time responses.
Over the past nine months, Amazon, Microsoft, and Google have each introduced or renamed an enterprise agent platform. And all three have converged on the same core architecture. Runtime, memory, tool gateway, identity, observability, and governance now appear in Amazon Bedrock AgentCore, Microsoft Foundry, and the Gemini Enterprise Agent Platform, albeit under slightly different names. Each vendor presents these components as the foundation for production agents. What was a fragmented collection of libraries just 18 months ago is becoming a distinct platform layer.
To see where that shift may lead, consider the evolution of platform as a service. Between 2011 and 2016, developers built applications from virtual machines, load balancers, message queues, secret stores, and monitoring agents — each with its own API and operational model. Cloud Foundry and Heroku unified those pieces around an application contract. Developers could think less about machines and more about applications. The agent ecosystem is approaching a similar inflection point, but it still lacks an equivalent contract — and no open-source project has stepped forward to define one.
The plumbing a production agent still needs
Imagine a platform team putting a customer support agent into production this quarter. They choose a model provider, then a framework, and finally a place to store session state and long-term memory. They add a tool gateway so the agent can reach the ticketing system. An identity layer follows, so the agent acts on behalf of the person who asked, and a sandbox keeps generated code somewhere safe. Evaluation and tracing are considered last, once someone asks how quality regressions will surface before customers find them.
Each of those choices looks small on its own. When combined, they decide which cloud the workload lives in. Session state resides in a single provider’s managed store. The traces are logged in that same provider’s telemetry service, and the agent’s identity is derived from its directory. Moving that agent a year later means rebuilding the whole assembly, which is where enterprises were before PaaS gave applications a portable shape.
What Cloud Foundry got right before it lost the market
Cloud Foundry reduced application deployment to a single command, and the platform took responsibility for everything after it. Buildpacks detected the language and produced a runnable artifact. Service brokers provisioned a database or a message broker and bound the credentials into the application environment. Routing, logging, autoscaling, and health checks arrived as platform behavior, not as a queue of tickets aimed at an operations team.
What mattered was the contract, not the implementation. An application declared what it needed and stayed agnostic of where it ran. Buildpacks began life at Heroku back in 2011. Pivotal and Heroku started the Cloud Native Buildpacks project in January 2018, and the CNCF accepted it that October. A PaaS idea outlived the platform that produced it.
Cloud Foundry never became the dominant platform. Kubernetes did, and the Cloud Foundry community eventually rebuilt its abstraction on top of Kubernetes through Korifi. The design principles traveled anyway, and enterprises running that platform in 2016 had portability that most agent teams cannot buy today.
The same primitives across three clouds
Let me dissect the three platforms because the similarities are conveniently hidden behind the branding.
AgentCore reached general availability in October 2025 with seven composable services, namely runtime, gateway, memory, browser, code interpreter, identity, and observability. The runtime offers eight-hour execution windows with complete session isolation. The gateway connects to existing MCP servers and turns APIs and Lambda functions into agent-compatible tools, while observability exports through OpenTelemetry into CloudWatch.
Microsoft renamed Azure AI Foundry to Microsoft Foundry effective January 1, 2026. Foundry Agent Service covers the same ground. Microsoft documents hosted agents running in a session-isolated managed runtime, with Entra Agent ID handling identity. Managed memory spans session, user, and procedural scopes, and tracing is built on OpenTelemetry.
Google retired the Vertex AI name at Cloud Next 2026 and folded the platform into Gemini Enterprise Agent Platform. What was Agent Engine became Deployments, and Memory Bank, Sessions, Agent Registry, Policies, and Gateways sit alongside it under an agent-first information architecture.
The convergence is rational behavior rather than a conspiracy, since infrastructure companies build vertically integrated platforms because integration is where the margin lives. The consequence lands on customers rather than on the vendors. Identity, telemetry, and deployment all terminate within a single provider, which makes the operational layer beneath the agent the part that resists moving. Recently, I analyzed the runtime aspect of this problem in a piece about Google’s Agent Substrate.
The contract an agent platform would inherit
Any agent platform can be tested by one question: what would a Cloud Foundry-style contract look like if it had been written for agents rather than for web applications? The mapping is close enough to be instructive and different enough to be interesting.
PaaS abstraction
Agent platform equivalent
Where portability breaks today
Application source
Agent code, instructions and evaluation suite
Each framework defines its own package shape
Buildpack
Framework detection and agent packaging
No shared build contract across SDKs
Backing service
Model, memory, retrieval or tool provider
Providers are wired into agent logic
Service binding
Authenticated attachment of tools and data
Credentials are issued by the host cloud
Router
Agent endpoint, MCP or A2A interface
The protocols exist but not the lifecycle
Logs and metrics
Traces, tool calls, cost and quality scores
GenAI conventions remain in development
Release promotion
Evaluate, version and progressively deploy
Evaluation couples to one vendor’s harness
Platform policy
Agent identity, permissions and approvals
Identity ties to the provider’s directory
Real deployments will mix these rows rather than adopt them cleanly, and no team needs the complete set from day one.
Package the agent as one deployable unit
The thing developers ship has to be versionable, testable, and movable as a single artifact. The code, the instructions, the tool dependencies, the memory contract, the permissions, and the evaluation suite all travel together, or none of them do. AWS gets close with its harness export path. One command turns a configured harness into Strands code, and AWS says the export preserves the model, prompt, tools, memory wiring, and container environment. Customers can easily move from configuration to code without an architecture rewrite, which is the right instinct pointed at a single cloud.
Bind capabilities rather than embed providers
The Twelve-Factor App taught developers to treat databases and caches as attached resources reached through configuration. Models, memory stores, retrieval engines, browsers and tool gateways deserve the same treatment. An agent that names its model provider inside application logic has already given up portability, whatever the platform brochure claims.
Make operations part of the abstraction
Rather than just enabling deployment, PaaS proved its value by offering built-in capabilities like routing, logging, autoscaling, and rollback. The questions that matter for agents look different, and they should be the ones enterprises must ask. Platform teams want to know whether the agent completed the task and whether it chose the right tools. They also want to know whether it exceeded its authority, what the run cost was, and whether quality regressed after a model update.
How agents differ from applications
Agents are not web applications with a model attached, and a platform built on that assumption will fail in production. Three differences carry most of the weight here. Agent behavior is probabilistic, so two identical inputs can produce different tool calls. Agents act with delegated user authority, which turns a permissions bug into a real-world side effect rather than an error page. An agent’s dependencies can also change its behavior without any code deployment. A model update or a revised tool description alters what the agent decides to do.
The Twelve-Factor rule that processes should be stateless does not survive that. An agent platform needs disposable execution workers alongside durable, inspectable, and portable agent state. LangGraph already demonstrates the combination in open source, with checkpointing at every step, first-class human interrupts, and execution that resumes after a crash. The control plane around it is part of the commercial LangSmith product, covering deployment, evaluation, and observability. The fragmentation appears within a single project.
What the open protocols leave out
Most of the primitives a neutral platform would need already exist. The Model Context Protocol standardizes how agents access tools and data. A2A covers discovery and communication between independent agents. OpenTelemetry is defining GenAI conventions for agent spans, tool calls, and token usage, though most of those attributes remain marked as in development. OCI images stay available as the packaging escape hatch for anything a managed runtime cannot host.
The vendors have already conceded that a neutral layer matters. The Linux Foundation announced the Agentic AI Foundation in December 2025. Its founding projects were MCP from Anthropic, goose from Block, and AGENTS.md from OpenAI, and AWS, Google, and Microsoft joined as platinum members. Google also moved A2A into the Linux Foundation.
Protocols are not the same thing as a lifecycle platform. A foundation that governs how agents talk to tools says nothing about versioning an agent. It says nothing about promoting that agent through environments, or rolling it back when an evaluation regresses. Enterprises running diligence on agent platforms can work through three plain questions. The first is governance: whether the project is controlled by a neutral foundation or by the vendor selling the managed version. The second is packaging, where the same agent artifact should run on two different clouds without a rewrite. The third is state, where memory has to live somewhere the enterprise can export from. No open project answers all three today.
Where is this headed?
Kubernetes defined pods, deployments, and services, and those abstractions influenced how an entire industry thinks about running software. The equivalent agent abstractions have not been fixed yet. Whoever ends up owning the agent control plane will not merely own deployment. That owner defines what an agent is, which components it contains, and what a platform team is permitted to swap out.
In summary, hyperscalers are building robust platforms that effectively address operational questions. They also answer them inside a single cloud. If a neutral project assembles the same lifecycle atop the protocols already in place at the Linux Foundation, enterprises will regain the negotiating position that buildpacks and service brokers once gave them. An open, cloud-agnostic agent platform would benefit vendors as much as buyers, because a stable contract is what allowed the cloud-native ecosystem to grow beyond any single provider.
After weeks of extending temporary access while bringing additional inference capacity online, Anthropic has finalized Claude Fable 5 subscriptions.
In a Friday announcement on X, the company said Claude Fable 5 will become a permanent part of Max and Team Premium subscriptions beginning Monday, July 20, at 50% of each plan’s usage limits. Pro and Team Standard users will transition to using the model through usage credits and will receive a one-time $100 credit.
An Anthropic spokesperson tells The New Stack, “Friday’s announcement means that Fable 5 at 50% of usage limits is now a standard, permanent part of Max and Team Premium plans.”
The changes bring an end to weeks of temporary extensions that left subscribers wondering whether Fable 5 would remain part of their subscription. Rather than introducing a new usage policy, Anthropic is making permanent the same access level subscribers have had since Fable 5 was redeployed on July 1.
Capacity drove the timeline
The move gives a glimpse into the economics of serving frontier AI models. Although Anthropic repeatedly extended temporary access over the past several weeks, the company says making Fable 5 a permanent part of Max and Team Premium subscriptions was always the goal.
An Anthropic spokesperson tells The New Stack, “Friday’s announcement means that Fable 5 at 50% of usage limits is now a standard, permanent part of Max and Team Premium plans. This was always our intention with Fable access (see more in our original launch blog on June 9). We planned to expand Fable 5 access in stages as we brought additional inference capacity online, with the ultimate goal of making it a standard part of subscription plans.”
Fable 5 usage limits explained
Anthropic has not explained exactly how it calculates the 50% limit or how many prompts that will translate to in practice. Developers using Fable 5 for heavy workflows are likely to reach the limit sooner than lighter users.
The company has said demand for Claude Fable 5 was difficult to predict and that broader access depended on bringing more capacity online. Reuters reported earlier this week that the company is in talks with Meta over a potential compute lease worth up to $10 billion, which could give Anthropic access to Meta’s AI infrastructure for as long as two years.
Meta compute deal looms
Thariq Shihipar, an Anthropic Developer Relations team member, posted on X, crediting the company’s employees with getting the necessary capacity online.
“This was due to a heroic effort by many people at Anthropic working sometimes literally around the clock,” Shihipar wrote on X. “It was not at all clear that we’d be able to do this in time, and so proud of everyone who made it happen. Enjoy Fable.”
“This was due to a heroic effort by many people at Anthropic working sometimes literally around the clock. It was not at all clear that we’d be able to do this in time, and so proud of everyone who made it happen. Enjoy Fable.”
Claude expands into India
The revised policy reflects broader changes across the AI industry. Frontier models are becoming expensive, which means AI companies are forced to balance compute costs with subscription pricing. OpenAI recently introduced GPT-5.6 Sol, while Chinese AI companies have released increasingly capable open-weight models, including Moonshot AI’s Kimi K3. Although Anthropic has not linked today’s pricing decision to competitive pressure, both model capability and the cost of serving those models have become key competitive factors.
…both model capability and the cost of serving those models have become key competitive factors.
Alongside the subscription changes, Anthropic is also formalizing its presence in one of its fastest-growing markets by introducing local pricing for Claude in India.
Indian customers can now pay for Claude in rupees instead of U.S. dollars, with GST included in the listed prices. That removes foreign currency conversions and other cross-border payment costs that previously came with subscribing. Claude Pro is now priced at Rs. 2,000 per month with annual billing or Rs. 2,399 on a month-to-month plan (previously $20 USD per month).
Industry reports have identified India as Claude’s second-largest user base after the United States. Together, the announcements show Anthropic expanding Claude in two ways: by making subscriptions easier to purchase in a major growth market while locking in an access policy determined by the compute required to serve Fable 5.
Apache Spark 4.2 launched last week, and it signals an expansion of Spark’s decade-plus role at the center of enterprise data processing
With new features for AI workloads, including governed metrics, vector retrieval primitives, real-time processing, improved Python support and native geospatial analytics, Spark 4.2 builds on a recent history of new AI and streaming features, reflecting how many engineering teams use the platform today. The release builds on Spark’s traditional role as a data processing engine by adding more of the capabilities needed to support production AI applications.
The launch introduced several features that enable developers to do more without leaving the platform, and for teams already using Spark, that could mean fewer systems to manage.
Governed metrics prevent conflicts
One team’s definition of a business metric isn’t always the same as another’s. Over time, those differences can lead to conflicting reports and uncertainty about which number to trust.
That becomes even more problematic when AI applications start consuming the same enterprise data that analysts and business intelligence tools use. If different teams define the same metric differently, AI systems can produce inconsistent results for the same question.
Spark 4.2 introduces governed metric views to address that issue. Organizations can define a business metric once and reuse that definition across applications. A metric view makes dimensions and measures first-class objects that Spark understands, so the engine can preserve the intended aggregation semantics regardless of who or what is querying it.
Organizations can define a business metric once and reuse that definition across applications.
Vector search goes native
One of the more significant additions is native vector search, which reduces the need to move data between Spark and a separate vector database.
Spark 4.2 introduces vector distance and similarity functions, vector normalization, vector aggregation, and NEAREST BY, a new SQL operator for top-K similarity searches. By bringing vector search into Spark, developers can keep more of their retrieval pipeline on the same platform.
By bringing vector search into Spark, developers can keep more of their retrieval pipeline in the same platform.
Python interoperability gets easier
Spark 4.2 makes it easier to move data between Spark and Arrow-native tools. With support for the Arrow C Data Interface and the PyCapsule protocol, Spark DataFrames can be passed directly to tools like Polars and DuckDB without copying or serializing the underlying data, as long as both sides support the standards.
Python also gets a few other updates. PySpark has been expanded; Arrow-optimized UDF execution is now the default, and Python Data Sources now include built-in time and memory profiling to help developers troubleshoot custom connectors.
Spark Connect makes the engine callable by agents
Spark Connect, which separates the client from the Spark server via a gRPC- and Arrow-based protocol, receives several updates in 4.2. The key idea is that a client builds a logical plan, the server handles analysis, optimization, and execution, and the results come back as Arrow batches. The client doesn’t need a full Spark runtime or a colocated JVM.
This update includes several changes to Spark Connect, the project’s client-server interface. AI applications can send processing requests to a remote Spark cluster while the work continues to run inside Spark. The release improves RDD API compatibility, error handling, and status reporting along that path.
Streaming powers real-time AI
Streaming gets several updates in Spark 4.2, including Auto CDC and Real-Time Mode. Many AI applications depend on continuously updated data rather than scheduled batch jobs. Auto CDC brings first-class change data capture to Spark Declarative Pipelines, handling the merge logic for keeping target tables current as source data changes — something that previously required hand-written, error-prone code. The new CHANGES SQL clause allows teams to retrieve data changes through a single SQL interface.
Spark 4.2 also adds built-in GEOMETRY and GEOGRAPHY types along with ST_* functions for location-aware analytics, without requiring external spatial extensions. For teams doing anything with location data — logistics, real estate, IoT — this removes another reason to move data out of Spark.
For teams that currently use Spark for ETL and then hand data off to other systems for retrieval, governance, or real-time processing, this release starts to blur that line. As more AI applications run directly on operational data, Spark is becoming part of the serving layer rather than simply preparing data for it.
Spark is becoming part of the serving layer rather than simply preparing data for it.
When you run Kubernetes at the scale we do on Amazon EKS, nodes break constantly. GPUs fall off the PCIe bus. Container runtimes wedge. Network interfaces disappear. Across tens of thousands of clusters, “rare” hardware failures happen multiple times a day, somewhere in the fleet.
For years, everyone responded the same way: an operator wakes up, reads a dashboard, SSHes into the node, cordons it, drains it, terminates the instance, and waits for a replacement. Every step is human-paced. Every step is toil. And if the failure lands at 3 a.m. on a weekend, the workload sits degraded for hours before anyone looks.
We built the EKS Node Monitoring Agent to help close that gap, which we open-sourced in April earlier this year. It detects node failures and writes Kubernetes NodeConditions that signal the problem to Karpenter, which then automatically replaces the node if required. The agent is one piece of a larger system. To understand where it fits, you need to understand what manages the nodes it monitors.
“Across tens of thousands of clusters, ‘rare’ hardware failures happen multiple times a day, somewhere in the fleet.”
AWS launched Amazon EKS Auto Mode that fully automates Kubernetes cluster infrastructure: compute provisioning, scaling, networking, storage, OS patching, and security hardening, so teams focus on applications, not cluster operations. It dynamically selects optimal EC2 instances (including GPU instances like P5, P6, and G6 families), scales based on workload demand, consolidates underutilized nodes, and keeps the operating system patched. EKS Auto Mode ships with automatic node repair as the default behavior: detection, severity classification, and Karpenter-driven node replacement all run out of the box with no add-on to install, no controller to configure, and no repair policy to write.
This is the story of how we built automatic node repair, the design decisions that shaped the system, and the hard lessons that came from operating it at GPU scale.
Six lessons from building self-healing Kubernetes nodes at scale
After operating this across thousands of clusters, the lessons compress into a short list. These are not unique to our system. The same patterns show up in NPD, NVSentinel, AKS Periscope, GKE’s auto-repair, and anyone building a custom node controller. They are folklore that should be a checklist. The open-source repo reflects each of these lessons in code, from the reason-code stability guarantees in the API to the jitter implementation that solved GPU workload interference.
“Your reason codes are an API contract. Additions are features. Renames are breaking changes.”
Your reason codes are an API contract. Every downstream consumer (repair controllers, dashboards, customer automation) keys on them by literal string match. Additions are features. Renames are breaking changes. Severity changes are breaking changes. Plan for them the way you plan for API versioning.
Absent and Unknown are not the same thing. “We are not watching” and “we are watching but cannot tell” require different responses from downstream automation. If your disabled monitor writes Unknown, some controller somewhere will eventually act on it. Emit nothing when you are not watching.
Don’t cross ownership boundaries. The kubelet owns workload-driven conditions. Your node-health agent owns hardware and infrastructure failures. Crossing that boundary means your repair system is fighting the kubelet’s eviction system, and one of them will make the wrong call.
Measure latency from the source. The detection SLO includes every hop in the signal chain: hardware event to driver log, driver log to journald, journald to agent poll, agent poll to NodeCondition write. The longest hop dominates. For kernel-level signals, journald flush cadence is the bottleneck. For GPU telemetry through DCGM, push-based policy violations (DBE, XID, NVLink) are near-instant, but polled field watches (NVSwitch fabric health, clock throttle) have a 5-minute floor. Know which path each detection uses.
Detection and diagnosis are separate systems with separate consumers. Detection feeds automation (fast, continuous, minimal data). Diagnosis feeds humans (on-demand, detailed, heavyweight). Conflating them degrades both.
Test telemetry interpretation against the spec, not empirical values. Hardware telemetry interfaces are not boolean. We read a DCGM bitfield for GPU fabric health and treated non-zero as failure. When a driver update changed the healthy return value from zero to a spec-defined non-zero mask, every GPU node was flagged unhealthy at once. The safety breaker held (by design), giving us time to ship the fix. The lesson: if you’re parsing packed enums or bitfields from GPU firmware, your test fixtures must come from the vendor documentation, not from what the field happened to return on previous hardware.
Node health detection in Kubernetes: Traps no one warns you about
Every node health agent in the Kubernetes ecosystem performs the same translation. Node Problem Detector (NPD), NVSentinel, GKE’s auto-repair, AKS’s Linux Extension, and the EKS Node Monitoring Agent all take noisy, low-level signals from a machine and translate them into a set of Kubernetes primitives: NodeCondition, Event, sometimes a CRD. The translation looks simple. It isn’t.
The output is a NodeCondition, which is just a type, a status (True/False/Unknown), a reason code, and a message. Four fields. But that surface area hides decisions that determine whether a repair action helps or hurts.
Reason codes are a public API. We learned this the hard way. In version 1.6.2, we changed NvidiaDeviceCountMismatch from Warning severity to Fatal. The technical reasoning was sound: once a GPU drops off the PCIe bus, it doesn’t come back without a node reboot or replacement. Leaving it as Warning meant GPU workloads kept getting scheduled onto degraded nodes, wasting expensive accelerator capacity. So we shipped the fix. Downstream automation broke. Customers had repair configurations keyed on the old severity. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. From that point, we treat every reason code addition as feature work and every rename or severity change as a breaking change.
“Absent” must not equal “healthy.” When we shipped per-monitor configurability in v1.6.0, we had to make a choice. A disabled monitor needs to produce some output (or no output). The three options: write True (your auto-repair now thinks the node is healthy because you’re not watching), write Unknown (ambiguous, might trigger repair depending on downstream logic), or omit the condition entirely. Only the third is safe.
This seems obvious in retrospect, but consider that NPD achieves the same result through a completely different mechanism: compile-time disable via build tags. NVSentinel delegates it to operator-authored CEL rules. The upstream Kubernetes spec defines what Unknown means, but if your repair automation treats Unknown as actionable, you will lose nodes for no reason. We chose to emit nothing when a monitor is off, and documented it as a hard contract.
Detection latency is bounded by the source, not by the agent. We originally told customers, “We detect kernel panics within 30 seconds.” This was wrong. Our agent’s detection time was under 30 seconds. But the kernel panic shows up in journald, and journald’s flush cadence is the actual bottleneck. If journald takes 45 seconds to write the line, our 30-second claim was incomplete.
For GPU faults, the picture is more nuanced because we use two detection paths with very different latency characteristics. The critical faults (double-bit ECC errors, XID errors, NVLink failures, page retirements, thermal and power violations) go through DCGM’s push-based policy violation channel. DCGM notifies our agent the moment it detects the violation; there is no polling interval. Detection of these faults is near-instant (sub-second in practice). A separate path uses a 5-minute field-value window to monitor NVSwitch fabric health, Fabric Manager status, and clock-throttle reasons. That window is the floor for those specific detections, but it does not apply to the critical GPU faults that trigger automatic repair. The lesson: the customer-facing SLO must include source-of-truth latency, and different signal paths within the same subsystem can have radically different floors.
Two severities, one switch: How auto-repair decides which nodes to replace
The kubelet already reports DiskPressure, MemoryPressure, and PIDPressure. NMA complements those with five additional conditions covering domains the kubelet does not monitor: kernel health, container runtime, networking, storage, and accelerated hardware. Every detection carries one of two severities, and severity is the switch that decides whether the repair cycle fires.
Condition severity is a terminal fault. It flips the matching condition to False and makes the node eligible for automatic repair. GPU device-count mismatches, critical XID and double-bit ECC errors, NVLink and NVSwitch fabric failures, a missing Fabric Manager, and Neuron DMA and HBM uncorrectable errors. On the networking and runtime side: VPC CNI process down, IPAMD unable to reach the API server, fork failures due to PID exhaustion, and pods wedged, terminating behind a broken container runtime. These are faults that won’t recover on their own. On GPU nodes, a single degraded accelerator can corrupt training checkpoints or waste thousands of dollars in compute per hour.
Event severity is informational. It posts a Kubernetes event, the NodeCondition remains True, and operators get visibility without disruption. Bandwidth ceilings, connection-tracking limits, Amazon Elastic Block Store (Amazon EBS) IOPS throttling, I/O delays, filesystem fragmentation, clock drift, liveness and readiness probe failures, kube-proxy anomalies, GPU thermal and power warnings, PCIe link degradation, and page-retirement thresholds. These signal trouble building before it turns terminal.
Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes and needlessly displace workloads. Too conservative, and degraded nodes serve traffic for hours while a GPU with a failing memory bank corrupts training checkpoints. The classification principle: if the failure is deterministic and infrastructure-owned (hardware broke, firmware crashed, a physical link went down), it triggers replacement. If the signal could be application-induced or transient, it stays informational. You never want to terminate a healthy node because a misbehaving pod saturated a resource.
“Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes. Too conservative, and degraded nodes serve traffic for hours.”
DiskPressure, MemoryPressure, and PIDPressure are the canonical examples. Every major auto-repair system (GKE, AKS, NPD) has independently converged on the same answer: don’t touch them. These are workload-driven conditions, not node-level faults. Replacing the node just moves the misbehaving workload to a fresh machine, where it will eat memory again. The correct response is kubelet-level pod eviction, not node replacement. If you’re building a node-health system, draw this boundary early and document it publicly.
The agent that hurt what it was protecting: GPU workload interference from health monitoring
The hardest lesson came from a customer running large-scale distributed GPU training. Their workload used NCCL collectives across hundreds of GPU nodes, where every node in a communication group must complete its step before any can proceed. One slow node makes every node wait.
They found that NMA itself was causing periodic slowdowns. The agent’s monitors all ran on independent goroutines, and when their polling intervals aligned, dozens of goroutines would wake simultaneously and burst onto many CPU cores at once. On a general-purpose web service, this would be invisible. In a distributed training job, microseconds of jitter on one node can cascade across the entire GPU cluster, causing measurable throughput loss.
The customer disabled NMA entirely and saw an immediate improvement. That was the worst possible outcome for us: a health agent that interferes with the workload it exists to protect is worse than no agent at all.
The fix was straightforward once we understood the problem. We added a startup jitter to every monitor’s polling interval. Each goroutine delays its first tick by a random offset (up to 20% of its base interval), staggering the wake times so they don’t align on boot. We cached system calls that hit /proc on every poll. We consolidated handlers that shared an interval into a single sequential work queue, reducing the goroutine count for monitors that didn’t need their own thread. The result was an agent whose CPU profile is flat and predictable rather than bursty.
The lesson generalized: if your health agent runs on the same host as the workload, its resource consumption pattern matters as much as its resource consumption total. A process that uses 0.5% CPU spread evenly is invisible. A process that uses 0.5% CPU in concentrated bursts can disrupt latency-sensitive distributed GPU workloads in ways that show up as lost training time rather than a CPU alarm.
This is why per-monitor configurability matters. Not every monitor is relevant to every workload. A dedicated GPU training cluster with one pod per node and no pod churn doesn’t need IPAMD monitoring or environment scanning. We shipped the ability to disable individual monitors so customers can keep the health coverage they need without paying the overhead of coverage they don’t.
How the repair cycle works
Karpenter is the compute controller that provisions and scales EKS Auto Mode nodes. It already owns the lifecycle of every node it launched, and consuming our NodeConditions for repair is a natural extension of that ownership. There’s no separate repair backend, no sidecar controller, no webhook chain. The same system that created the node is the one that replaces it.
Karpenter’s AWS cloud provider declares repair policies: each one pairs a condition type with a status that means “replace this node.” The policies include toleration windows that prevent reacting to transient blips:
Accelerated hardware faults: 10 minutes. These are unambiguous (a GPU is either present or absent) and expensive to leave running (a training job on a degraded node wastes GPU-hours).
Everything else (kernel, runtime, networking, storage, kubelet NotReady): 30 minutes. Enough time for a transient network blip or a temporary runtime hiccup to resolve on its own.
The flow:
The agent detects a terminal fault and flips the matching condition to False with a reason code.
Karpenter’s health controller sees the transition and starts a timer.
If the condition clears before the window expires, the timer resets silently. The node was never touched.
Past the toleration window, a safety gate checks fleet health. Karpenter will not repair more than 20% of nodes in a NodePool simultaneously. If a correlated event (a bad AMI rollout, a control-plane hiccup, a zonal impairment) trips conditions across many nodes at once, the system holds. Auto-repair also stands down while an Amazon Application Recovery Controller zonal shift is active, so deliberate traffic movement away from an impaired Availability Zone is not mistaken for a fleet of broken nodes.
Inside the safety threshold, Karpenter taints the node to block new scheduling, gracefully drains running pods (respecting PodDisruptionBudgets), terminates the instance, and provisions a replacement sized for the displaced workload.
The replacement node comes up with a fresh agent monitoring it from boot. No operator in the path. In our testing, the full cycle from fault injection to replacement node running workloads took under 12 minutes. Detection landed in under a second (critical GPU faults use DCGM’s push-based policy channel, not polling). Then 10 minutes of toleration, and roughly 90 seconds for the replacement to launch and register.
The part that surprised us: detection and diagnosis are not the same problem
Auto-repair handles the common case: broken node gets replaced, workload keeps running. But “why did that node fail?” is a different question, and one we initially tried to answer inside the detection path. That was a mistake.
Detection answers “is this node healthy?” It runs continuously with minimal overhead, and it needs to be fast: a condition flip that takes 5 minutes to produce is 5 minutes of degraded workload. Diagnosis answers “what went wrong?” It needs to collect detailed artifacts: full journald output, containerd state, network configuration, dmesg, GPU driver logs. In our testing, that collection completes in about 7 seconds and produces a compressed log bundle. Baking it into the detection hot path would have slowed down the thing customers care most about: how fast the system reacts.
We built them as separate concerns sharing an agent binary. The NodeDiagnostic CRD lets you request a full log bundle from any node through kubectl, without SSH. On EKS Auto Mode, where nodes are Amazon Elastic Compute Cloud (Amazon EC2) managed instances with no shell access by design, this is the only way to investigate after a GPU failure or any other node-level fault.
The experience is one command:
kubectl ekslogs <node-name>
The plugin creates a NodeDiagnostic resource. The agent on the target node detects it via a watch, collects system state into a compressed tarball, and stores it temporarily (available for 10 minutes). The plugin then downloads it through the kubelet’s Node Log Query API (KEP-2258, GA in Kubernetes 1.36). No SSH, no security groups, no key pairs.
This separation means detection doesn’t slow down to collect evidence, diagnosis doesn’t need to be always-on (saving node resources), and you can diagnose a node that auto-repair has already flagged but hasn’t yet terminated. The 10-minute window for accelerated hardware faults gives you exactly enough time to grab the logs before the node is gone. If you’re interested in further improvements, engage with us on EKS public roadmap.
What this means if you’re running EKS
On EKS Auto Mode, all of this is on by default. Auto Mode fully manages your cluster infrastructure (compute, networking, storage, patching, and security hardening) so you focus on applications, not cluster operations. The agent runs as a systemd service in the node image (not a DaemonSet you manage), Karpenter consumes its conditions as part of the compute lifecycle it already owns, and kubectl ekslogs gives you diagnostic access without SSH. There is nothing to install, configure, or operate. For GPU workloads, this means your expensive accelerator nodes are automatically monitored, classified, and replaced without any operator intervention.
On managed node groups or self-managed Karpenter, you can assemble the same loop: install the Node Monitoring Agent as an EKS add-on and opt each node group into auto-repair. The architecture is the same, just not pre-assembled.
The failure modes we hit when running it at scale, and the fixes that come out of them, flow back to anyone using it. If you’re building a node-health system or running ours and hitting an edge case, come build with us!
Login
