OpenAI on Wednesday launched GPT-Live, a pair of new voice models that fundamentally redesign how people talk to ChatGPT — replacing the company's existing Advanced Voice Mode with an architecture that can listen and speak simultaneously, much like an actual human conversation.

The two models, GPT-Live-1 and GPT-Live-1 mini, are rolling out globally starting today across iOS, Android, and ChatGPT.com. GPT-Live-1 becomes the default voice model for paid ChatGPT users on the Go, Plus, and Pro tiers, while GPT-Live-1 mini serves free-tier users. OpenAI also plans to bring the models to the API, and developers can sign up to be notified.

The release marks the third generation of ChatGPT's voice technology in roughly two years — and OpenAI's clearest bid yet to turn its chatbot into something that feels less like querying a search engine and more like talking to a colleague.

Why full-duplex voice changes everything about talking to AI

The defining technical advance in GPT-Live is what OpenAI calls a "full-duplex architecture." In telecommunications, full-duplex means both parties on a phone call can talk and listen at the same time. Applied to AI, it means the model continuously processes your incoming audio even while it generates its own spoken response — no more waiting for a clean silence gap to figure out when you've finished a thought.

"Instead of processing a sequence of separate messages, GPT-Live continuously processes input while generating output," OpenAI wrote in its research blog. "The model can therefore make interaction decisions many times per second: whether to speak, continue listening, pause, interrupt, or invoke a tool."

In practice, that translates to a voice assistant that can insert conversational acknowledgments — "mhmm," "yeah," "got it" — while you're still talking, pick up on a natural pause without jumping in prematurely, and handle rapid interruptions without derailing the entire exchange. 

OpenAI's previous Advanced Voice Mode, launched to paid users in September 2024, processed and generated audio within a single model but still operated on rigid turn-by-turn exchanges. As OpenAI acknowledged in the announcement, "because turn detection is based on silence, even a brief pause or background noise could be mistaken for the end of turn — causing the model to interrupt at unnatural times."

That brittleness created a product that, while impressive in demos, could be deeply frustrating in extended real-world use. Background chatter in a coffee shop could trigger a response. A thinking pause might get swallowed. The experience felt, as one researcher put it on X shortly after the announcement, like "walkie-talkie turn taking." GPT-Live is designed to end that era.

How OpenAI split voice and intelligence into two separate layers

GPT-Live introduces a second structural change that may prove just as consequential for enterprise adoption: it decouples the voice interaction layer from the reasoning layer.

When a user asks a straightforward question, GPT-Live handles it directly. But when the query demands web search, deeper reasoning, or more complex agentic work, GPT-Live delegates the task to a frontier model running in the background — at launch, GPT-5.5, the large language model OpenAI released in April — and continues talking with the user while the computation happens asynchronously.

"While it works, GPT-Live can keep talking with you and maintain the flow of conversation," OpenAI explains. "As we release new frontier models, we'll continuously update the model used by GPT-Live."

This delegation model is a meaningful architectural bet. Rather than building a single monolithic voice model that tries to be both conversationally fluid and deeply intelligent, OpenAI has split the problem in two: a voice-native model optimized for real-time interaction, and a separate reasoning engine that can be swapped out as the state of the art improves. 

It is, in effect, a modular design — one that allows OpenAI to upgrade the intelligence of its voice assistant without retraining the voice model itself. The implications for enterprise and developer workflows are significant. A voice agent built on this architecture could maintain a natural conversation with a customer while simultaneously querying databases, searching the web, or performing multi-step reasoning — tasks that would have introduced several seconds of dead air under the old pipeline.

The three generations of ChatGPT voice, from clunky pipeline to continuous stream

To understand how far voice AI has come, it helps to trace the three generations that led to GPT-Live.

The original ChatGPT Voice, launched in 2023, used a cascaded pipeline — a speech-to-text model (Whisper) transcribed what you said, a large language model (GPT-4) generated a text response, and a text-to-speech model converted that response back into audio. Each handoff introduced latency and lost information. 

As OpenAI noted, "the complexity came at a cost: information could be lost across models, and responses were slow and stilted." That cascaded approach was the industry standard, and its limitations were well-documented. As the blog OpenHelm noted in an October 2024 analysis of OpenAI's Realtime API, the old pipeline stacked up to roughly 1,700 milliseconds of latency — nearly two full seconds of dead air before the first word of a response. Managing the state between the three separate APIs consumed an enormous amount of engineering effort.

OpenAI's Advanced Voice Mode, which began its limited rollout to paid ChatGPT Plus users in July 2024 before expanding more broadly in September 2024, collapsed that three-model pipeline into a single model that processed audio natively. As TechCrunch reported at the time, the rollout came with five new voices — Arbor, Maple, Sol, Spruce, and Vale — alongside improved accent handling and smoother conversations. 

The feature also launched on the web in November 2024, extending it beyond mobile. But Advanced Voice Mode still operated through discrete, alternating turns — and it launched into the shadow of a PR debacle that OpenAI is still working to leave behind.

The Scarlett Johansson controversy still shadows OpenAI's voice ambitions

Advanced Voice Mode arrived in the wake of one of OpenAI's most damaging self-inflicted crises. During the GPT-4o launch in May 2024, the company showcased a voice called "Sky" that many listeners immediately noted sounded strikingly similar to Scarlett Johansson, who famously voiced an AI companion in the 2013 film Her.

Johansson said she had declined OpenAI CEO Sam Altman's offer to voice the system, then was "shocked, angered and in disbelief" when the product launched with a voice her own friends couldn't distinguish from hers, as NBC News reported. Altman had tweeted just the word "her" the day the product launched.

OpenAI pulled the voice and apologized, but the incident drew public scrutiny from SAG-AFTRA and members of Congress, and crystallized broader concerns about AI companies moving fast with creative IP.

The Hollywood labor union said the issue underscored "why we're strongly championing federal legislation that would protect their voices and likenesses ... from unauthorized digital replication," as NBC News reported. Forbes contributor Paul Tassi wrote at the time that Altman, "by holding up Her on a pedestal of something to strive for, has missed the point of that film" — in which the protagonist's relationship with his AI companion ultimately does him more harm than good.

GPT-Live appears designed, in part, to move past those controversies. OpenAI says it has "remastered the nine distinct voices in ChatGPT for GPT-Live" and notes the system "is designed for conversation, not voice impersonation," with "safeguards to prevent it from imitating a real person's voice."

What 150 million weekly voice users will actually notice today

OpenAI disclosed that more than 150 million people talk to ChatGPT using voice and dictation features each week — a notable slice of the platform's 900 million total weekly active users. The voice experience has grown into a substantial product in its own right, used for language practice, bedtime stories, commute-time chat, and hands-free everyday help.

The new product features reflect that usage. GPT-Live introduces rich visual cards that surface during voice conversations — weather forecasts, stock data, sports scores, and maps — giving users something to glance at without breaking the flow of speech.

Users can now choose between three reasoning levels for answers: Instant for quick responses, Medium for moderate thinking, and High for more complex work. And if you take a moment to think, "ChatGPT Voice now waits instead of jumping in and interrupting," OpenAI wrote. "If you ask it to stay quiet and listen, it will. And when there's background noise, like passing traffic or nearby conversations, ChatGPT is better at focusing on your voice instead of getting distracted."

Early reactions from users with preview access were cautiously positive. "I had early access to sol. it is a phenomenal model," wrote one user on X, adding it is “much better at frontend, long context knowledge work, and its vibes are much better.” Another observer cut to the heart of the matter: "The smarts are not new here, GPT-Live hands hard questions to GPT-5.5. What is new is the feel: full-duplex voice that listens while it talks."

New voice-specific safety tests reveal where the risks still live

The GPT-Live system card, published alongside the announcement, reveals a safety strategy built around the particular risks of real-time voice interaction — a domain where the speed and intimacy of conversation create hazards that text-based chat does not.

OpenAI expanded its safety evaluations to include audio-native tests, using both real user voice samples (from those who opted in) and synthetically generated prompts targeting edge cases across categories like self-harm, sexual content, illicit behavior, emotional reliance, mental health, and hate speech.

On the synthetic evaluations — which OpenAI described as deliberately adversarial — GPT-Live-1 showed substantial improvements over Advanced Voice Mode. In illicit behavior, for instance, the safety score rose from 0.63 to 0.97. On self-harm, it climbed from 0.72 to 0.98. Hate speech achieved a perfect 1.00, up from 0.87.

On the production-prompt evaluations — which used real user audio and reflected more ambiguous, borderline scenarios — the picture was more mixed. GPT-Live-1 matched or improved on Advanced Voice Mode in most categories but showed a slight regression on emotional reliance (from 0.88 to 0.82), though OpenAI noted the change was not statistically significant.

The company built real-time safeguards that can intervene while the model is speaking — steering toward safer responses, surfacing crisis resources, or ending the voice conversation entirely in higher-risk situations. It also designed additional protections for teen users and adapted self-harm support flows for voice, including crisis helpline integration.

Perhaps most notably, OpenAI said it is "rolling out longer-term measurement and post-launch monitoring focused on emotional reliance" — an acknowledgment that the very naturalness GPT-Live strives for creates its own category of risk.

Google, ByteDance, and Nvidia are already in the full-duplex race

While OpenAI was refining its safety guardrails, its rivals were shipping full-duplex systems of their own. Google's Gemini Live, which supports full-duplex conversation alongside camera and screen sharing — capabilities GPT-Live notably lacks at launch — is already available in the Gemini app. Google released Gemini 3.1 Flash Live in March as its highest-quality real-time audio model, targeting low-latency voice interactions for developers.

ByteDance launched Seeduplex in April, claiming to be the first production-scale full-duplex speech AI deployed at scale, inside its Doubao app. Seeduplex reported roughly a 50 percent reduction in false-response and false-interruption rates compared to ByteDance's previous half-duplex system. And Nvidia's PersonaPlex, released in January, brought customizable voice and role control to full-duplex models, breaking what had been a constraint where natural-sounding models were locked into a single fixed voice.

The competitive picture is clear: full-duplex voice interaction is quickly becoming table stakes for consumer AI products, not a differentiator. OpenAI's advantage lies in the scale of its existing user base, its integration with GPT-5.5's reasoning capabilities, and the breadth of the ChatGPT ecosystem.

But the window in which any one company has a monopoly on natural-sounding voice AI has already closed. OpenAI also acknowledged several gaps. GPT-Live does not support voice with video or screen sharing at launch. Language support is limited, with the company noting that "for certain languages, the model may have a non-native accent or gaps in fluency." And API access is not available on day one, meaning enterprise developers cannot yet build on GPT-Live directly — a constraint that will slow the model's penetration into commercial voice-agent workflows where competitors like Google, ElevenLabs, and Deepgram already have developer-facing products.

The end of the chat box may be closer than anyone expected

GPT-Live is essentially OpenAI's most significant bet yet on voice as the primary interface for AI — not just a convenience feature bolted onto a text chatbot, but a purpose-built interaction layer that sits between the user and the company's most powerful models.

"Over time, we believe this research will also unlock the ability to use voice for increasingly complex, longer-running, and more agentic work," OpenAI wrote. That ambition — using natural voice as the front end for autonomous AI agents that can perform multi-step tasks — is the logical endpoint of the full-duplex plus delegation architecture.

Imagine telling your phone to book a flight, negotiate with your insurance company, or debug a production server, all through a conversation that feels as natural as talking to an assistant who also happens to have the intelligence of a frontier AI model.

Two years ago, talking to ChatGPT meant dictating into a microphone and waiting nearly two seconds for a stilted reply. One year ago, it meant a smoother exchange that still felt like a polite, slightly awkward phone call with someone who insisted on waiting for you to finish every sentence. Today, it means something closer to a real conversation — imperfect, still constrained in some languages and missing video, but unmistakably closer. OpenAI once got into trouble for wanting to recreate the movie Her. With GPT-Live, the company may finally be reckoning with the harder question the film actually posed: not whether AI can sound human enough to talk to, but what happens to us when it does.

Thomas Dohmke, who stepped down as GitHub’s CEO last year to become a founder again, is opening a preview of a distributed Git network on Wednesday that is designed to keep fleets of AI coding agents from overwhelming a single central server — and one that may soon compete directly with GitHub’s core service.

Entire, Dohmke’s post-GitHub startup, is launching a preview of this on Wednesday (but for now, it is behind a waitlist). With this, developers can mirror an existing GitHub repository onto Entire’s own infrastructure in one step.

“In the era of agents, centralized Git hosting has become a fundamental constraint, as the strain of billions of agents and developers hammering a central server shows up in the form of rate limits, high latency, or even outages,” says Dohmke in today’s announcement. “Today, we begin to return Git to its original promise, with a distributed, and soon fully decentralized and open-source network of interconnected nodes around the world. By doing so, we enable any developer or agent to host their code in-region, pushing, pulling, and cloning close to where they operate, fast and without bottlenecks, while still part of a global, collaborative network.”

The key here is that the code stays on GitHub, as Entire stresses, but coding agents can work with the Entire mirror and, as the company notes, “build without rate limits.”

Entire’s mirror is meant to absorb the constant flow of traffic that a fleet of agents can generate. That traffic, after all, is part of the reason GitHub is often buckling under pressure these days and startups like Entire have an opening.

Centralized Git hosting, Dohmke says in an interview with The New Stack, has become “a fundamental constraint” now that billions of agent and developer operations land on the same servers, showing up as rate limits, latency, and outages.

Given GitHub’s recent availability issues, it’s no surprise that startups are trying to get into this space. Entire is one — and it has the pedigree — but in June, Cursor also announced Origin, its own Git forge rebuilt for swarms of agents that are cloning and committing against a single repository in parallel.

Entire is starting with active regions in the United States, the European Union, and Australia, but the team says that now that is has spun up its first few regions, it will add more soon.

‘Git as a database’

To build its network, Entire rewrote the server part of git. GitHub, GitLab, and Bitbucket all wrap the server-side of the Git binary and build their infrastructure around it. Entire started from scratch.

“We see Git really as a database,” Dohmke says. The open source Git project has two halves, he explains: the client that an agent uses to talk to a repository, and the server a host runs to manage storage. Rather than build on that stock server, like most companies would do, “we made the decision of not going that route, and instead implemented our own Git backend.”

That only makes sense if Entire’s version has significantly better performance than the stock Git server, of course. Entire says its benchmarks have pushed the network to a sustained rate of 570,000 clones per hour, 586 pushes per second, and roughly 470 combined clone-and-push operations per second.

Pushing to a native Entire branch can run up to 25 times faster than pushing through to GitHub, Dohmke says.

Entire it will open-source both the git backend and the benchmark suite.

The foundation layer, now real

When Dohmke first described Entire’s plans to The New Stack in February, he described a three-layer platform that included a Git-compatible database at the bottom, a semantic reasoning layer in the middle, and an interface on top. Even then, he said that the database, unlike a centralized Git host, could be a globally distributed network of nodes.

But in February, Dohmke also said Entire wouldn’t necessarily end up competing with GitHub, and that code repositories would stay central to the pitch.

Pressed on whether that still holds now that Entire hosts its own copy of the GitHub repo, he calls the mirror complementary, in part because Entire can offer enterprises the ability to keep their code in a local region to fulfill local regulations. He also notes that GitHub has a huge ecosystem and an extended feature set.

“I think the question for the buyer really is, is it not better for me from an availability and reliability perspective, that I have both of these products, so if one of them is down — there’s always going to be single points of failure and human errors — then I have my mirror on the other side,” he says. “But we certainly will, in deals, compete for the dollar spent at a much smaller scale compared to the multi-billion-dollar business that is GitHub today.”

For now, that keeps the two complementary. Dohmke argues that GitHub remains the “source of truth,” or “cold storage,” while the working copy lives on Entire. But he also says that Entire will launch native repositories in the coming months, and those wouldn’t need GitHub underneath at all. All of this will be open-sourced as well.

Entire raised its $60 million seed round in February, when it had 15 employees. Felicis led it, with Microsoft’s venture arm among the backers. The company is now past 40 people and aiming for 60 by the end of the year.

Entire beyond Git: the semantic memory layer

Entire is building its middle layer — the semantic reasoning layer — in parallel with the Git platform.

The semantic layer now integrates with every major coding agent, including Claude Code, Codex, Cursor, Factory AI, and GitHub Copilot, and records each session, prompt, and tool call in the repository alongside the code.

Having this data is useful for agents, and it was the first core service the company launched. Now, it is also building more services on top of that history.

The company is adding Entire Blame, for example, which shows not just who last touched a line but the agent session and prompt behind it. There is also Entire Review, which fans out several agents for an intent-aware review, and the company is adding a code and semantic search feature that lets agents (and developers) search across code changes and the reasoning that produced them.

“Session logs are now the second most important artifact in software development, and they belong in the repository alongside the code,” Dohmke says.

The post Entire is building a Git network for agents appeared first on The New Stack.

While the household-name frontier models race forward with version numbers and context windows of at least a million tokens, a new breed of upstart data science specialists is pushing the context window into double figures. 

Subquadratic debuted a 12-million-token window in May of this year, and Silicon Valley and South Africa-based Refiant launched its 10-million-token context-window model, Protea, on Wednesday. It’s a move that may signal the long-context AI race is now on.

Model inefficiency & workarounds are commonplace

But more context windows alone are not enough. This is because even the most capable models have a few hundred thousand tokens in working memory, which can force workarounds to compensate for what the model can’t access.

Refiant co-founder Dr. Viroshan Naicker tells The New Stack that he believes modern LLMs “fail to be organically efficient at an elemental level” and that his organization’s approach mimics how systems in nature, from ant colonies to beehives, find efficient solutions to complex problems.

“This is no case of pseudoscientific puff; nature is the most computationally efficient system that we know, and many algorithms used in science are nature-inspired,” Naicker says. “This is a road well-traveled in science. There are multiple teams globally working in this particular (nature-inspired) direction, trying to bridge the gap between AI inference as we know it and the energy efficiency of natural systems.”

“Fish and birds coordinate their movements to converge on the mathematically shortest, most efficient routes — honeybees, fireflies and bacteria are also programmed to use degrees of swarm-style optimization.”

What can the birds & the bees teach us about AI?

Did Naicker just mention ant colonies and honeybees?

Yes, because Refiant uses swarm-style optimization. It’s seen in ant colonies, which initially move randomly until a food source is detected, after which they leave a pheromone trail for other ants to optimize their journeys. Fish and birds also coordinate their movements to converge on the mathematically shortest, most efficient routes. Honeybees, fireflies, and bacteria are also programmed to use degrees of swarm-style optimization.

Naiker, along with his co-founders, Siddharth Gutta and Mathew Haswell, form a team with experience spanning quantum mathematics, traditional finance, and commercial scaling. Applying swarm-style optimization to data in Protea means inference is performed through a combination of compression and context management.

“From our perspective, we are also advancing a technology which provides context-specific inference models grounded in data,” Naicker clarifies. “We think this has value for reducing model hallucinations, replacing RAG, and constructing better, more reliable, agentic workflows. This adds a layer of trust in sensitive application scenarios, rather like an added insurance, rather than taking it away.”

Just how much is 10 million tokens?

The Refiant team describes 10 million tokens as equivalent to 7.5 million words in a single conversation (and we know from Anthropic’s own benchmarks this year that Claude has a 1-million context window), or five years of a user’s emails, 83 novels, or 830 podcast episodes, all held in active memory at the same time.

The team claims Protea is capable of working on entire enterprise codebases or decades of clinical trial data — datasets that previously had to be broken apart and fed to models in fragments — so they can be processed in a single pass with full fidelity. Engineers on Protea also submit that they can successfully tackle the “lost in the middle” problem — a limitation of million-plus-token windows, where models stay accurate at the start and end of the context but lose the thread of everything buried in between.

Refiant first applied these techniques to model compression, shrinking OpenAI’s GPT-OSS-120B so it could run on a MacBook Pro with 18GB of RAM. 

“Rather than publishing benchmarks, we’re inviting users to run the models and try them out.”

The Protea series is open and live, and Refiant is inviting teams to stress-test the context window across different industries and use cases. But should we trust sensitive hould enterprise data archives to a completely unproven startup founded only one year ago?

Bring-your-own-cloud, a possible progression

“We adhere to data management best practices, processes and compliance requirements,” Naiker confirms. “This is reasonable for a startup at our particular stage. Privacy and data sovereignty are important values for us, and we are actively exploring edge, self-hosted, and bring-your-own-cloud data models.”

But a 10 million-token context window is big. Won’t that fall short when Protea starts to suffer from massive latency spikes when processing a full dataset? Naiker agrees that “latency is a core issue with long-context inference models,” but in the tests his company has run, it has delivered inference at a reasonable latency, even with large token windows.

“We have internal reports and tests that validate the technology, including Ruler, MRCR and Babilong, but we aren’t asking anyone to take our word on this. Rather than publishing benchmarks, we’re inviting users to run the models and try them out,” adds Naiker.

What comes next, a 100-million context window?

Although the technology industry is littered with apocryphal statements and Bill Gates almost certainly never said “64K ought to be enough for anyone” in real life, we have to ask ourselves today whether we’ll be laughing about those “silly little” 10 million token context windows by the end of the decade.

It may not take that long. Internally, Refiant maintains that it has already demonstrated a working prototype with a 100-million-context window and is exploring how best to benchmark and productionize it at that scale in the future.

Coming next, then, as Dr. Evil from Austin Powers would say, the one-hundred-billion-context window, right?

The post “Nature is the most computationally efficient system we know”: How Refiant used swarm optimization to build a 10-million-token AI model appeared first on The New Stack.

Vercel CEO Guillermo Rauch and Coinbase CEO Brian Armstrong run very different companies, but they’re making the same architectural bet. Instead of building around a single AI provider, both are designing production systems that can route work across multiple models.

Rauch and Armstrong aren’t making this decision in a vacuum. Frontier models have become much closer in capability for everyday engineering work, open-weight alternatives have improved dramatically, and the price gap keeps widening. That makes it much easier to justify routing work across several models instead of committing to one. 

Trillion tokens, zero loyalty

In an interview with TechCrunch, Rauch said that Vercel now routes more than a trillion tokens a day across millions of deployments, and that the company is actively moving away from one-lab partnerships. Rauch’s point highlights that the model has become just one interchangeable component in a larger inference pipeline.

That’s a significant position from the CEO of a company that serves as deployment infrastructure for a huge share of the frontend ecosystem. Rauch is calling single-lab partnerships obsolete.

Rauch is calling single-lab partnerships obsolete.

Cheaper defaults, smarter routing

Armstrong is making the same bet, and the financial results state his case. Coinbase cut its internal AI spend by nearly half while overall token usage continued to grow, without imposing usage caps on engineers.

Their playbook basically runs on three core levers.

First, it’s an internal LLM gateway. Coinbase deliberately defaults its engineers to lower-cost open-weight models, specifically Z.ai’s GLM 5.2 and Moonshot AI’s Kimi 2.7. Engineers can still pull down a stronger model if a specific job absolutely demands it, but the pricing gap makes the default choice obvious. GLM 5.2 costs roughly $1.40 per million input tokens and $4.40 per million output tokens.

Compare that to Anthropic’s Opus 4.8, which sits around $5 for input and $25 for output. You are looking at a three- to six-times cost reduction per token. And it holds its own on major coding benchmarks, scoring 62.1 on SWE-bench Pro, compared to GPT-5.5’s 58.6. Plus, because Coinbase self-hosts these models, zero code or query data ever leaves their environment.

The second lever is task-based routing. Armstrong makes a highly practical point here, suggesting teams want a frontier model to do the heavy lifting for complex planning, but for pure execution tasks, where cheaper models perform just as well, there is zero reason to pay top dollar.

The third piece is aggressive caching. By keeping a conversation locked to the same model as long as the cached context is valid, Coinbase managed to push its cache hit rate from a measly 5% up to 60%. That 12x jump is a massive cost driver.

Gateways as control planes

If you want to understand Armstrong’s broader mindset, listen to his recent chat on the Sourcery podcast. He casually mentioned that Coinbase now operates with roughly 1,200 full-time AI agents, a number they calculate by normalizing compute hours to a standard 40- to 60-hour workweek. At that scale, he argues that human developers have absolutely no business manually choosing which model to use. The infrastructure has to automate that decision entirely.

Human developers have absolutely no business manually choosing which model to use.

Because foundation models are becoming so easy to swap in and out, the engineering focus is shifting to the surrounding infrastructure. Like a centralized control plane, a gateway intercepts every prompt and makes a dynamic, split-second decision about whether a workload actually requires the expensive reasoning capabilities of a frontier model or a cheaper, faster alternative can handle it. The infrastructure makes that call based on the cache state, the complexity of the task, and real-time pricing.

Teams need visibility into latency, uptime, token consumption, and cost across all providers because using multiple model providers changes observability requirements. Without that data, it’s difficult to know whether routing decisions are actually improving performance or reducing costs.

Test before you trust

Evaluation becomes just as important. Lower-cost models need to be continuously tested against the workloads that matter to an organization before they are deployed to production traffic. Public benchmarks are a useful starting point, but are no substitute for measuring how a model performs on your own code, data, and workflows.

Trying to pick the single best AI provider is a losing game.

What’s striking is that Vercel and Coinbase arrived at remarkably similar architectures despite solving different problems. Both assume that today’s best model probably won’t stay on top for long. If that’s true, the competitive advantage shifts away from the model itself and toward the infrastructure that decides which one to use. 

The post Coinbase runs 1,200 agents and just slashed its AI bill in half appeared first on The New Stack.

Your organization constantly needs more information about system performance, usage, and data while in production — or better yet, before it heads to prod. The challenge of telemetry increases with the complexity of your stack and agentic sprawl. Because “it works in the testing environment” becomes moot in the face of non-deterministic agents.

After all, AI agents span multiple environments, and that leaves traditional log-metric-trace models insufficient to handle the volume of the agentic AI era. The situation can lead companies to think that the best option is to throw everything into the locked box of proprietary tooling, but that creates another problem: Information is siloed within each layer, fragmenting data and taking you further from realizing real AI ROI.

Unified context across fragmented workflows

The OpenTelemetry framework and the OpenSearch distributed search and analytics engine make for a powerful, open-source pairing that gives organizations of all sizes unified context across their fragmented workflows. In fact, OTel has crossed the 95% adoption threshold for new cloud-native instrumentation projects and has already become the default choice for Greenfield projects.

OpenSearch, sponsored by Amazon Web Services, is gaining traction with AI engineers, as it recognizes that observability and AI must be united. This year’s OpenSearch roadmap specifically focuses on making it the primary retrieval interface for AI agents and an essential piece of any retrieval-augmented generation and agentic AI stack. 

Join us on July 22

Just because open source doesn’t have a direct cost doesn’t mean it’s free. That’s why Dotan Horovits and Rekha Thottan of AWS are going to perform a live troubleshooting simulation using correlated logs, metrics, and traces, followed by a demo of how agentic traces flow through Otel pipelines. Also learn how the open-source evaluation framework Agent Health can provide a structured pre-production benchmark to flag unpredictable agentic behavior before release. 

Join us live on July 22 to learn along and ask questions to learn how your organization can adopt these open-source standards in the second half of this year — across agentic workloads and traditional infrastructure, at scale.

Register for the webinar here

The post Watch AWS engineers troubleshoot agentic AI with OpenTelemetry and OpenSearch appeared first on The New Stack.

AI trash-talkers love to rip on the technology for failing to produce meaningful business results, often pointing to studies like that from MIT NANDA, which reveals a 95% failure rate for enterprise AI solutions, or that from IDC, which states “only 9% of [Europe, Middle East, and Africa] organizations have been able to deliver measurable business outcomes from most of their AI-related projects over the past two years.” 

What many AI skeptics fail to account for is the experiential nature of AI prototypes; not all these projects are actually meant to move beyond the testing phase. Still, a 5% success rate is embarrassing. 

What’s the holdup? 

Two things. First, most organizations build AI prototypes on sand; that is, the data infrastructure on which they build early applications can’t support later moves to production. Meanwhile, the operational teams responsible for managing those applications in production often lack the human power to keep up with engineering’s growing output. 

4 reasons prototyping infrastructure  ≠ production infrastructure

When asked why so many AI prototypes don’t make it to production, Phillip Merrick, co-founder, CPO, and chairman, pgEdge, tells The New Stack that data infrastructure is largely to blame. 

Specifically, he explains that prototype environments don’t meet the requirements of large enterprises for production, naming four main ways they fall flat.

First, Merrick says prototyping environments lack the deployment flexibility organizations need to move from prototype to production. 

Vendor-managed cloud platforms, he acknowledges, may seem like an obvious choice for prototyping, as they allow teams to get up and running quickly. Still, he warns they lack the technical chops to support AI applications in production, especially in security, compliance, and governance. Particularly for organizations in healthcare, finance, or other regulated industries, vendor-managed cloud platforms often lack the stringent controls found in self-managed cloud or on-prem environments. 

“You’ve got to be able to choose where that AI prototype is ultimately going to be put into production.”

In this way, flexibility and security go hand in hand. Merrick asserts. “You’ve got to be able to choose where that AI prototype is ultimately going to be put into production.” 

Similarly, when it comes time to shift to production, Merrick says vendor-managed cloud platforms can introduce data sovereignty challenges at both the enterprise and regional levels. 

“Your data layer is obviously where you enforce this,” notes Merrick. But he says the environments most teams use for prototyping muddy the waters: “If it’s on a vendor-managed platform in who knows what cloud, what region, then you’ve lost data sovereignty.” 

Lastly, Merrick brings attention to reliability, explaining that AI prototypes can’t move into production without assurance of high availability. For example, when it comes time to upgrade the database or swap hardware, can it be done without downtime? 

“In the vendor-managed cloud world, the answer to that is almost always no,” says Merrick, reiterating his point that moving AI apps from prototype to production requires enterprise-grade infrastructure. 

So why are developers prototyping where they can’t productionize?

If data infrastructure selection is what’s holding developers back from moving AI prototypes into production, then why do they keep starting on the wrong foot? 

As Merrick explains, many are attracted to the ease of use of vendor-managed cloud platforms. “These prototyping environments admittedly make it very easy to get started,” he says. But prototyping shortcuts, it seems, don’t pay off in the long run, as someone else is ultimately on the hook for making those prototypes production-ready. 

Still, Merrick doesn’t blame developers for looking for the easy way out. Rather, he says there’s a disconnect between the prototyping playground and the production battleground that prevents developers from understanding what it will take to productionize prototypes down the pike. 

---

More pgEdge articles in The New Stack

• Why developers are betting on Postgres for AI
• Moving AI apps from prototype to production requires enterprise-grade postgres infrastructure
• Reinventing PostgreSQL for the Next Generation of Apps
• How distributed Postgres solves the cloud’s high-availability problem
• Scaling AI inference at the edge with distributed PostgreSQL

---

Years ago, he says, tooling decisions were primarily made top-down without developer input: “Then, starting 15–20 years ago, the developers won back, quite rightly, the power in being able to choose their own tools.” 

The problem now, Merrick claims, is that developers make tooling decisions exclusively for prototyping environments, without anticipating production needs. Internal divisions mean developers are often only responsible for building prototypes before passing the baton to an entirely separate operations team for production: 

“The upshot is you really don’t have, in some organizations, this throughline of understanding [of] what the production requirements are all the way back to the developer making the initial choices.” 

For Merrick, this disconnect is where AI projects start to fall apart, as teams are left trying to move AI prototypes from accessible-but-inadequate vendor-managed cloud platforms to enterprise-grade data infrastructure that meets requirements for deployment flexibility, security, data sovereignty, and high availability. 

“But if you make the right data infrastructure choice, you won’t have that disconnect,” he says, “because you’ll have this throughline from prototype to production.” 

He names Postgres as the data infrastructure that best helps developers bridge this divide, calling it “the Swiss army knife of databases” due to its extensibility, fully open-source nature, and ability to address diverse data management problems, from unstructured data to vector embeddings to geospatial data. 

Where and how Postgres is run matters too, Merrick points out, again drawing attention to the limits of many vendor-managed cloud environments that often lack the governance controls to meet data requirements and/or the deployment flexibility to shift to compliant on-premises or BYO cloud environments. 

But picking the right data infrastructure only solves half the problem

Merrick says there’s another part of the equation most organizations are overlooking: people, or more precisely, database administrators (DBAs) and their growing workloads. 

Per Stack Overflow’s 2025 Developer Survey, 84% of respondents use AI tools, up from 76% the year prior. Meanwhile, Supabase says over 60% of databases on its platform have been launched “by some sort of AI tool.” As Merrick points out, this explosion of productivity comes with a catch: there aren’t enough DBAs to keep up.

“You’ve had this massive, massive step shift in developer productivity,” he explains. “But you have to have some way of managing that on the production side; these databases can’t go unmonitored.” Operations and administration teams were already struggling to keep track of existing Postgres databases before agentic engineering added even more, he says: “Who’s going to manage them?” 

He says it’s time for agentic operations to catch up with agentic engineering. 

AI DBA agents can give humans “superpowers” 

If it seems Merrick is proposing organizations look to fully autonomous DBA agents to take over, he says the industry isn’t there yet: 

“The world is not ready for fully autonomous databases administered by AI DBA agents. But there is a massive resource shortage and productivity problem, and DBAs can only manage so many databases,” he explains. Meanwhile, new “AI applications require so many more databases to put in production.” 

“The world is not ready for fully autonomous databases administered by AI DBA agents.”

So how can organizations increase their operational capacity? 

Merrick says DBAs should look to new AI DBA agents, not to take over but to give them “superpowers” to monitor and manage more databases with less manual slog. 

pgEdge’s Ellie is one example. Part of the pgEdge AI DBA Workbench, Ellie is an AI agent that has 21 MCP tools and can run EXPLAIN ANALYZE, inspect schemas, query historical metrics, and walk through multi-step diagnostic workflows. When a database falters, Ellie finds the problem, diagnoses it, and provides a solution in the form of working SQL code for the human DBA to review. “When you’ve reviewed it and agree that it’s the right course of action, you literally press the play button, and the agent plays that SQL code into the database, and you solve your problem,” explains Merrick.

In this way, Ellie should bring more capacity to operations teams, where Merrick insists organizations are starved for DBA expertise. To his point, some industry predictions say 41% of today’s database professionals intend to leave the industry in the next decade, half moving into retirement and the rest seeking other work. 

“An agent … can actually respond to those alerts far more quickly and productively than a human can.”

Without AI agents, Merrick argues, DBA work is tedious, laborious, and time-consuming. As he explains it, a database may have been humming along just fine, but when there’s a snag, trouble can manifest across multiple applications; it’s then up to the DBA to comb through monitoring data to observe and diagnose the problem, essentially scouring for a needle in a haystack. 

“An agent,” he says, “can actually respond to those alerts far more quickly and productively than a human can.” 

Better infrastructure AND people: It takes two to improve AI prototype success rates 

In nearly any context, AI raises questions about quality over quantity, and enterprise AI projects are no exception. Agentic engineering means developers can now produce more, but all those prototypes don’t just fly directly into production. Limitations in both infrastructure and operational human power are creating obstacles that cause many AI prototypes to fail. 

For Merrick, easing the transition from prototype to production requires not only great AI tooling but production-ready data infrastructure, paired with agentic operations that can keep up with the agentic engineering boom. 

The post Why most AI projects fail: It’s infrastructure and people  appeared first on The New Stack.

Palantir CEO Alex Karp went on CNBC’s Squawk Box last week to discuss a new partnership with Nvidia to deploy open-weight AI models in sovereign government environments. But viewers got a nearly 20-minute broadside against the entire frontier AI model industry, calling it “effing insane” and accusing companies like OpenAI and Anthropic of overcharging enterprises while harvesting their proprietary data.

Days later, Mistral CEO Arthur Mensch made a strikingly similar case on LinkedIn, warning that closed AI providers are gaining “immense leverage” over enterprise customers as organizations connect proprietary workflows to hosted models. He suggests open-weight models, open data systems, and enterprises building their own training flywheels.

The two executives are approaching this from opposite ends of the market, yet their convergence on the same message within the same week underscores architectural control.

Two pitches, one argument

Karp runs a company that sells an application and ontology layer designed to sit between enterprises and the models. The Palantir-Nvidia deal pairs Nvidia’s open Nemotron models with Palantir’s Sovereign AI Operating System, built on AIP, Foundry, Ontology, and Apollo, enabling government agencies and critical infrastructure operators to deploy, fine-tune, and audit AI models within their own air-gapped environments.

When CNBC’s Becky Quick told Karp he sounded angry, he pushed back, saying, “This is the voice of American business that is being channeled through me,” and urged the panelists to call any CEO privately to verify.

“This is the voice of American business that is being channeled through me.”

Mensch’s company sells open-weight models, and he has a custom training platform called Forge, which frames the problem differently but reached the same conclusion. He argues in the post that closed providers have a track record of going after their most successful customers once they learn what those customers are building. His program runs from open models to open data stores, strict access controls, and a continuous training flywheel that improves systems on internal interactions.

Lock-in gets an upgrade 

If you’ve been building software at scale for any length of time, you already know that when new technology arrives, enterprises can’t help but rush to adopt it. But the dependency problem becomes impossible to ignore.

We saw the same problem with cloud computing when companies went all-in on a single hyperscaler’s proprietary services, only to later discover that the cost of switching providers could exceed the cost of staying, even when staying meant overpaying. It’s one of the reasons the industry spent years building abstraction layers, portability tooling, and multi-cloud strategies in response.

Foundation models are raising the same questions, but there’s a twist. When an enterprise connects a model to its internal data, including customer records, proprietary processes, and domain-specific knowledge, the dependency becomes informational. Karp’s argument is that model quality is converging across providers, but the operational leverage accrues to whoever controls the deployment layer and the data flowing through it.

Mensch’s claim has a concrete referent that enterprise architects will recognize. In 2025, Anthropic cut off model access to coding startup Windsurf while building its competing product, Claude Code. The Brookings Institution has separately warned that model providers increasingly compete with their own customers as they chase application-layer revenue.

When access disappears overnight 

When the U.S. government ordered Anthropic to suspend access to its most advanced models for foreign nationals, the company cut access across the board, including to enterprise customers in Europe who had built workflows on top of those models.

Access has since been restored, but for CIOs and enterprise architects who had treated model APIs as stable infrastructure, it was the same as if a cloud provider pulled compute resources without warning. It’s probably why the incident sent European policymakers into overdrive. Mensch, whose company had open-weight alternatives ready, seized the moment.  

Enterprise teams need a plan for when a critical dependency is modified, repriced, or revoked by a provider whose incentives may not always align with their own.

Enterprise teams need a plan for when a critical dependency is modified, repriced, or revoked by a provider whose incentives may not always align with their own.

Architecture shifts toward portability 

Enterprise AI architecture is now essentially this: don’t marry a single provider, build for portability, keep your most sensitive data and logic under your own control.

In practice, this is showing up in three ways.

Firstly, we’re seeing a portfolio approach. A powerful closed model is maintained for complex reasoning and customer-facing work, while an open-weight model is used for repetitive, high-volume tasks. For businesses handling sensitive information, the appeal is that an open model can be run entirely on their own infrastructure, so data never has to leave the organization.

The second is the rise of the model-routing layer, abstraction frameworks that let organizations swap models without rewriting their applications. Palantir’s ontology pitch sits here as does the emerging crop of agent orchestration tools that treat the LLM as a pluggable component behind a standardized interface.

The third is the open-weight movement itself. Nvidia shipped Nemotron 3 Ultra in June under a permissive Linux Foundation license. Meta’s Llama continues to expand. Mistral’s Forge platform lets enterprises train custom models on their own data. And Mistral is teasing an upcoming open-weight model this summer, with early access opening in July.

Follow the commercial incentives

It would be naive to ignore the commercial interests at play. Karp’s Palantir sells the deployment and governance layer; it benefits directly if enterprises treat models as interchangeable commodities. Mensch’s Mistral sells open-weight models and a training platform, and it benefits directly if enterprises distrust closed providers. Zoho’s Sridhar Vembu, who endorsed Karp’s position publicly last week, has his own reasons for wanting enterprises to own their AI infrastructure rather than rent it from Silicon Valley.

But the fact that multiple executives across different market segments are saying companies need to own their data, maintain deployment flexibility, and not hand their competitive advantage to a provider who might become their competitor suggests the argument is resonating.

What developers should watch

If your most sensitive data is flowing through a third-party API with terms of service that can change, you’ve made a governance decision that your compliance team may not have fully evaluated.

For the engineering teams actually building on foundation models, the takeaway is that if your most sensitive data is flowing through a third-party API with terms of service that can change, you’ve made a governance decision that your compliance team may not have fully evaluated.

The engineering choice, to abstract the model layer, to evaluate open-weight options alongside closed APIs, to think about deployment portability the same way you think about cloud portability, is increasingly strategic.

The post Palantir’s Alex Karp and Mistral’s Arthur Mensch agree: AI lock-in is coming for enterprises appeared first on The New Stack.

In April, Andrej Karpathy published a GitHub gist file called “LLM Wiki,” a brief text document designed to help one build a personal knowledge base using LLMs. It’s based on the premise that an AI agent will keep what it knows as linked Markdown files it can read and rewrite, because a language model does not get bored maintaining cross-references and can touch fifteen files in a single pass. It was only a few thousand words with no product attached.

Two months later, Google turned that instinct into a published standard called the Open Knowledge Format. The OKF packages organizational knowledge, metrics, tables, and runbooks as plain Markdown that any agent can read without a proprietary account. Google is careful to call it v0.1 — a starting point rather than a finished standard.

Garry Tan, the Y Combinator president, got there first in a different lane. His gstack, an MIT-licensed Claude Code setup that crossed 66,000 GitHub stars within weeks, comprises 23 specialist roles, each a Markdown file. No runtime; no code; just prose that runs across ten different coding agents.

Markdown has become the substrate agents read and write

Three approaches, three different needs, one common solution. Karpathy sought agent memory, Google aimed for enterprise context in BigQuery agents, and Tan wanted a way to summon an engineering team from a terminal. All three turned to the same basic resource: a folder of Markdown files versioned in git.

Developers had already established this practice. CLAUDE.md and AGENTS.md are present in millions of repositories as the initial files an agent loads. OKF and gstack are the evolved forms of this convention – one focused on what the agent knows, the other on how it behaves.

This is the Git and JSON playbook tied to the agent’s knowledge. The formats that survived are the ones you could start using without changing anything. You can simply cat the file, clone the repo, and any tool you already use can parse it. MCP remains important as the interface an agent connects to. Markdown is becoming the format that carries the content.

The lock-in moved from the model to the files

The significant factor to observe here is the competitive advantage, not technical specifics. For two years, the belief was that owning the best model meant controlling the developer.

This perspective is now shifting. Replacing Claude with GLM or Codex, gstack continues to operate because the core intelligence evolved, but the documentation did not.

The moat is shifting from the model to the Markdown a team owns and accumulates over time.

The moat is shifting from the model to the Markdown a team owns and accumulates over time. A company’s OKF bundle, including its runbooks, metric definitions, and architecture decisions, is, by design, portable across clouds, models, and frameworks.

That kind of portability is the reason vendor-neutral formats exist and why Google’s OKF deserves a closer look.

If no one develops consumers for it, it remains just a good idea that Google released on a slow Friday.

The area where I am most likely mistaken is durability. Declaring Markdown standards is easy, but making them reliable is difficult. OKF is merely a 0.1 draft with a reference implementation, not a full ecosystem. If no one develops consumers for it, it remains just a good idea that Google released on a slow Friday.

The direction remains determined by three separate bets targeting the same file format within a single quarter. Your next agent is likely to interpret its context from a Markdown folder, and the creator of that folder now possesses an advantage that the model vendor cannot easily replicate.

The post Andrej Karpathy, Google and Garry Tan agree Markdown is the answer, but they’re not solving the same problem appeared first on The New Stack.

Halfway through 2026, artificial intelligence has been at the center of every major story inside the world of software development and in just about every major story outside of it.

Last month, the Commerce Department ordered Anthropic to pull Fable 5 and Mythos 5 offline worldwide — only to lift the ban 18 days later. It wasn’t Anthropic’s only government clash; earlier in the year, the Pentagon fought the company over its refusal to give the military unrestricted model access.

Elsewhere, frontier labs planted their flag on Wall Street in May with new deployment arms and partnerships, while Anthropic and OpenAI both pursued IPOs at valuations above $800 billion. Underneath the valuations is an infrastructure buildout of chips, data centers, and deals meant to keep pace with model releases that land every few weeks.

Meanwhile, open-weight models are narrowing the gap between closed and downloadable, and the harness — the tools, memory, and orchestration around a model — matters more as agentic AI moves into the enterprise. Add tokenomics (the real cost of reasoning at scale) and non-technical execs vibe-coding their own tools, and you’ve got ten of the biggest AI moments of 2026 — so far.

Here are ten moments that have defined a pivotal first half of 2026 in the world of AI, as chosen by the editorial staff at The New Stack.

10. President Trump’s Executive Order on AI

President Donald Trump has been relatively friendly to AI, given his support from tech companies in Silicon Valley. On June 2, 2026, Trump signed an executive order aimed at hardening American systems against AI- related threats, while at the same time rejecting “overly burdensome regulation.” The order directs the Committee on National Systems Security to prioritize cyber defense and tasks the Treasury, the NSA, and CISA with establishing an AI security clearinghouse to coordinate vulnerability scanning and patching across critical infrastructure. The administration supports deregulation along with national security-driven AI oversight.

9. AI infrastructure buildout

Chipmakers and AI labs tightened ties in 2026 to keep pace with model releases. We also saw big moves, with Nvidia and SK Hynix striking a multi-year partnership spanning Vera Rubin supercomputers, Vera CPUs, and next-gen memory. Meanwhile, data center capacity has expanded globally. The buildout shows that compute, power, and hardware are now a bottleneck to AI’s growth.

8. The rise of the harness

“The harness is where the hard work is,” Harness CEO and founder Jyoti Bansal told The New Stack last month. As base models are performing closely on benchmarks, the harness becomes the differentiator. The harness is the tools, memory, orchestration, and guardrails wrapped around a model. The harness determines whether an agent stays on task, recovers from errors, and performs safely. The harness helps shift competitive advantage from model capability to system design.

7. Tokenomics

Spend is now the battleground for both AI producers and consumers. AI labs are restructuring pricing around compute consumption rather than flat subscription, and companies are looking for ways to cut their token spend. Last month, the Linux Foundation launched the Tokenomics Foundation with support from Google, Microsoft, IBM, JPMorgan Chase, KPMG, Oracle, and Salesforce. The organization is tasked with establishing open standards, benchmarks, and best practices across the entire AI token economy.

6. Agentic AI goes mainstream

A year ago, agents were largely proof-of-concept demos that were unreliable in production. In 2026, they became infrastructure. For instance, ChatGPT’s browsing agent, Claude’s tool use and multistep coding runs, and Google’s autonomous information agents now run continuously in the background rather than on command. Meanwhile, enterprises are adding agents into real workflows such as monitoring, code review, procurement, and customer support. However, this shift may carry security risks as agents gain access to data and systems.

5. The Pentagon goes to war with Anthropic

In February, Department of War Secretary Pete Hegseth summoned Anthropic CEO Dario Amodei to his office to demand that the military be granted unrestricted use of the company’s technology. But Amodei held his ground and refused to allow the military to use Anthropic technology for mass surveillance on citizens or for autonomous weapons. Following that, President Trump ordered federal agencies to phase out the use of Anthropic, and Hegseth designated the company a “supply chain risk,” which is a label previously used for foreign adversaries – basically blocking the company. Anthropic sued in federal court, claiming the government’s move was unfounded and retaliatory. The company received a preliminary injunction from a San Francisco court, holding that the government’s actions constituted unconstitutional First Amendment retaliation. 

Meanwhile, amid the initial fallout from Anthropic’s battle with the Pentagon, OpenAI made its own deal with the military.

4. AI titans plant their flags on Wall Street

Within 72 hours in May, Anthropic and OpenAI each launched enterprise deployment arms, announced major financial services partnerships, and shipped agent tooling targeting Wall Street workflows. The message was the same — the next phase of frontier AI is not about models. It’s about deployment. 

Anthropic’s new services firm — backed by Blackstone and Hellman & Friedman alongside General Atlantic, Apollo, Goldman Sachs, and Sequoia Capital — targets mid-sized enterprises that the large consulting and systems integration firms don’t prioritize. These include community banks, regional health systems, and mid-market manufacturers. Applied AI engineers from Anthropic embed directly with clients alongside the new firm’s own engineering staff, doing workflow discovery, building custom Claude-powered solutions, and supporting clients long-term.

OpenAI’s Deployment Company — “DeployCo” — operates one market segment up, targeting large enterprises with the same forward-deployed engineering model. Its acquisition of applied AI consulting firm Tomoro brings roughly 150 experienced Forward Deployed Engineers (FDEs) from day one, backed by more than $4 billion in initial investment and a partner roster that includes McKinsey, Bain & Company, and Capgemini.

Meanwhile, both companies are considering IPOs with valuations exceeding $800 billion.

And both companies are betting on the same thesis: that the deployment gap — the widening distance between what frontier AI can do and what enterprises have actually shipped — is the next major revenue opportunity. And both moved on it in the same week. 

3. Open-weight models are coming

Chinese labs continued to close the gap with Western frontier labs in 2026. Alibaba’s Qwen, Zai’s GLM and Moonshot’s Kimi, delivering open-weight releases that rivaled closed models on standard benchmarks. Zai’s GLM-5.2, released June 13, beat Anthropic’s Claude Opus 4.8 in some benchmarks – showing the highest marks for open-weight models. GLM-5,2 is also one-fifth the price of comparable closed models. 

“The industry is all focused on which lab has the smartest model, but that focus looks to the past,” David Mytton, CEO of Arcjet, told The New Stack. “GLM-5.2’s capability indicates that usage of open-source models is about to explode. “This all seems obvious in retrospect: models became more capable at the start of the year, agents started taking real actions (particularly since most of the work happens following the chat prompt), and legal restrictions on using frontier models are causing people to look elsewhere. This will cause all sorts of security issues because managing so many model capabilities will become difficult.”

Paul Sawers, contributing writer at The New Stack, tells us for this countdown: “Budget open-weight model panels are now matching frontier proprietary benchmarks at a fraction of the cost, undercutting the case for paying top dollar for a single closed model. Example of startup ditching Anthropic for DeepSeek.”

Indeed, the European AI agent startup Lindy AI migrated 100% of its production traffic from Anthropic to DeepSeek, citing millions of dollars in savings,

2. CEOs vibe coding their own tools

While software developers seem to have wholly adopted AI coding assistants, the business side of organizations — the C-suite and other executives high up in the chain of command — is adopting these tools to “vibe code” a variety of agents and productivity applications.

The trend runs from simple workflow automations to full production systems serving hundreds of users. The tools are Claude, Cursor, and, increasingly, the AI features embedded in the platforms these executives already run. The motivations range from impatience with IT queues to genuine curiosity about what the technology can do. And the results are more varied than the enthusiasm surrounding them might suggest.

Woodson Martin, CEO of OutSystems, took a more structured approach to his own vibe coding experiment. He built a personal mobile app wrapper on top of MCP services his team had created — and he built it twice in parallel, once using OutSystems’ own AI coding tool, Mentor, and once using Claude, connecting to the same backend both times.

“I was tired of explaining it to somebody who was supposed to build it for me,” Martin told The New Stack in April. “I was just like, ‘I’ll do this myself.’”

The app is a personal chief-of-staff system that consolidates customer account intelligence — buying signals, website activity, internal data — into a pre-meeting briefing he can pull up on his phone. It replaced what had been a 45-minute PowerPoint session plus multiple prep meetings from his sales team. 

1. The government cracks down (and later relents) on Anthropic Fable 5 and Mythos 5

The Fable/Mythos takedown illustrated how unpredictable AI policy has become. Anthropic launched Fable 5, its only Mythos-tier model, on Jun 9, 2026, with the fuller Mythos 5 reserved for a small set of trusted customers under what the company called Project Glasswing. The rollout lasted only three days.

On June 12, Commerce Secretary Howard Lutnick sent Anthropic CEO Dario Amodei a directive ordering the immediate worldwide suspension of both models for all foreign nationals, including Anthropic’s own non-citizen employees.

Apparently, the trigger was a jailbreak that Amazon researchers found, which could expose the models’ cybersecurity capabilities. This raised concerns at Commerce. Anthropic said it didn’t have a way to restrict access by nationality in real time, so it disabled Fable 5 and Mythos 5 globally. The Commerce Department partially opened Mythos 5 to select government-approved organizations in the following weeks, but the freeze didn’t lift until June 30 — after an 18-day period of limbo.

Meanwhile, Anthropic added extra cybersecurity safeguards to Fable 5 and began restoring global access on July 1.

Having mentioned Anthropic’s run-in with the Pentagon, Frederic Lardinois of The New Stack tells us: “While the two situations are not directly linked, it’s hard not to read the Fable controls as an extension of this existing animosity between Anthropic and the Trump administration — and in part, this seems personal as well.”

What’s next

What’s in store for the second half of 2026? We’ll be tracking how much AI-generated code actually makes it into production — and the tools designed to close that gap; the expanding autonomy of agentic AI and the guardrails keeping pace with it; the fast-moving regulatory environment around the frontier labs; and enterprise adoption of open-weight models.

We’re also watching AI’s spread among knowledge workers, the ever-longer stretches agents run unsupervised, and the hiring and performance of forward deployed engineers.

Whatever happens this year, The New Stack will cover it. And if you haven’t already, subscribe to The New Stack Daily newsletter for timely, thoughtful updates on how AI is reshaping software development.

The post 10 moments that defined AI’s turbulent first half of 2026 appeared first on The New Stack.

Finding the most capable model at the lowest cost has always been the goal. But as agentic AI evolves, a new problem is frustrating engineers: token consumption is becoming too high across AI systems. Every agent operation consumes tokens, but in very different ways. For instance, a moderately complex agent request can consume 20,000 to 60,000 tokens across its reasoning chain, but a nontrivial engineering task can burn 150,000 to 200,000 tokens per problem.

Every agent operation consumes tokens, but in very different ways.

Developers are realizing that selecting the right model is important, but the bigger issue is limiting unnecessary token movement throughout an agent’s workflow. That is why teams are starting to consider how to accomplish the same tasks with fewer tokens.

Compounding costs across agents

The costs add up quickly. A task that takes about 50,000 tokens with one agent can easily consume several hundred thousand with multiple specialized agents together. That’s because each one needs enough context to do its job. It’s not unusual for an agent to process 30,000 tokens of context just to return a 500-token response, and those exchanges add up over the course of a workflow. Each handoff effectively pays a tax in input tokens that compounds with every loop iteration.

Each handoff effectively pays a tax in input tokens that compounds with every loop iteration.

This is especially noticeable among multi-agent architectures. When one agent delegates to another, it must encode its current state and task instructions into the downstream agent’s context window. The receiving agent processes all of that, produces a result, and passes it back. Then the orchestrating agent reingests it alongside everything else it’s tracking. Every exchange adds another layer of overhead.

Building more token-efficient architectures

A growing collection of strategies addresses this problem. Three practical solutions stand out:

Compress context, preserve reasoning

The most direct solution is to reduce the amount of context an agent carries from step to step. Rather than accumulating an ever-growing interaction history and replaying it with every task, systems can summarize earlier portions of a conversation or working memory before passing them forward.

One way to do that is by narrowing the agent’s field of view. Instead of handing it an entire codebase or document collection, the system surfaces only what’s relevant to the task at hand. Go too far, though, and the agent can lose important context that it will need later.

To make this work, the system needs a compact memory layer of key facts and decisions alongside the compressed context. The agent needs to recall the reasoning chain without having to reread it each time.

Route tasks to cheaper models

Hierarchical routing lets engineers parse a JSON response, format a log entry, or check whether a file exists without using the same model used to architect a system design. It assigns each subtask to the smallest model that can reliably do the job. A lightweight model, for instance, handles routine classification, extraction, and formatting steps, while a more suitable model handles decisions that truly require deeper reasoning.

So if 60 to 70 percent of an agent’s steps are routine operations, then a smaller model can handle the work at a fraction of the cost, substantially reducing the overall token spend for the workflow.

Cache reasoning, skip redundancy

That’s where semantic caching comes in. Instead of solving the same problem twice, it compares the meaning of a new request using embeddings. If the match is close enough, the agent can reuse earlier work instead of generating a new reasoning chain.

The savings make a difference, especially in scenarios involving customer support systems that answer similar questions all day, or even document-processing pipelines that handle thousands of nearly identical files. In such scenarios, reusing prior reasoning can significantly reduce the number of tokens an organization consumes.

Measuring what matters

But these workflows only deliver value if teams see their impact, and many organizations are still figuring out how to measure them effectively. Poorly designed agent loops or inefficient multi-agent handoffs can dominate costs in ways that are invisible when you’re only looking at per-request pricing.

It’s also easy to focus too much on tokens. Running an agentic application also means paying for GPUs, memory, vector databases, and the tooling needed to monitor everything in production. Saving tokens helps, but it won’t solve the whole problem if the rest of the stack is still expensive.

The next phase of AI infrastructure

The next generation is increasingly focused on building systems based on better architectural decisions. That means context management, model invocation, task decomposition, and intermediate work reuse are becoming just as important as inference pricing or even benchmark scores.

Model capabilities will continue to improve, and inference costs will likely fall. But if autonomous agents become the dominant way organizations build AI applications, the systems that scale most effectively may not be the ones with the cheapest models, but those that waste the fewest tokens.

The systems that scale most effectively may not be the ones with the cheapest models, but those that waste the fewest tokens.

The post Why cheaper models alone won’t save your AI budget appeared first on The New Stack.

Cyberattacks used to be the biggest security issue surrounding AI infrastructure, but that could be changing. A recent cargo theft outside Chicago suggests another vulnerability — and it’s one that has nothing to do with malware or prompt injection.

Just last week, the Cook County Sheriff’s Office recovered two stolen trailers containing roughly $1.3 million in data center equipment and copper wiring, taken from separate shipments originating hundreds of miles away. One trailer held about $300,000 worth of copper wire — reported stolen in Pine Hill, Alabama — destined for data center construction. The other carried roughly $1 million in data center infrastructure equipment, stolen out of Jacksonville, Florida. Both ended up at the same truck yard in Elk Grove Township, outside Chicago.

Viewed in the context of the AI boom, it highlights that the physical supply chain itself is becoming a new target for bad actors.

Viewed in the context of the AI boom, it highlights that the physical supply chain itself is becoming a new target for bad actors.  

A new high-value cargo

We’re all familiar with typical bottlenecks like GPU shortages, power constraints and cooling capacity, which have plagued the AI era since its inception. But we forget that building an AI data center requires an enormous volume of specialized hardware moving through freight networks. These include servers, networking gear, fiber, switchgear, cooling systems, power distribution equipment and thousands of pounds of copper. Each represents capital investment and potential deployment delays.

As hyperscalers accelerate the construction of data centers, the exposure of these items between the factory and data center creates a risk category that the industry as largely ignored.

When one delay cascades

Large GPU clusters depend on the synchronized delivery of dozens of interconnected systems. A training cluster is a tightly coupled system of servers, switches, optics, power distribution, and cooling that must be installed together. Missing networking hardware can idle racks, delayed power equipment can postpone an entire deployment and stolen copper can stall electrical work. So when one component category disappears, the delay cascades across everything.

So when one component category disappears, the delay cascades across everything.

Cargo theft by the numbers

Infrastructure resilience increasingly depends on whether critical hardware arrives at the construction site at all — and on schedule. Verisk CargoNet reported that U.S. and Canadian cargo theft losses jumped roughly 60% in 2025 to nearly $725 million, even as the total number of incidents held essentially flat — a sign that thieves are becoming more selective about high-value freight. Metal theft rose 77%, driven largely by demand for copper, while organized groups shifted toward enterprise computing hardware. CargoNet expects that focus on high-value technology — RAM modules, storage drives and enterprise computing equipment — to carry into 2026. For broader context, the Department of Homeland Security has estimated that cargo theft overall costs as much as $35 billion a year.

The Chicago incident fits squarely inside that trend.

Beyond firewalls and malware

Obviously, cargo theft isn’t an engineer’s problem. But organizations building AI infrastructure may need to broaden their thinking about deploying AI capacity on aggressive timelines.

Cloud providers, colocation operators and hardware vendors have already invested heavily in defending infrastructure from digital threats. As AI infrastructure becomes more valuable, protecting the physical systems behind it may deserve similar attention.

The next supply-chain conversation

The AI boom has already forced the industry to rethink electricity, cooling, networking and semiconductor manufacturing. Physical logistics may be next.

It starts long before the equipment reaches the data center.

If the value of AI infrastructure continues to climb into the billions of dollars, the industry’s definition of “infrastructure security” is likely to expand beyond firewalls and identity management. It starts long before the equipment reaches the data center.

The post The $1.3 million theft that exposed AI’s blind spot appeared first on The New Stack.

Microsoft’s latest AI services announcement suggests the era of standardizing on a single model may be ending. This week, the company launched a $2.5 billion AI adoption business designed to help enterprises customize AI deployments and use multiple models rather than lock themselves into a single provider — part of a larger shift toward systems that route each request to the model best suited to the task.

Simply put, the company that arguably has the deepest single-model partnership in the industry is now selling model swappability as the product.

Betting $2.5 billion on flexibility

Microsoft said Thursday it is creating a new operating entity, Microsoft Frontier Company, to help corporate customers select AI technologies that actually work for their businesses and produce a return on investment, Reuters reported. The unit launches with $2.5 billion in funding from Microsoft and will work with customers including Unilever and Novo Nordisk.

The new firm will help customers choose and integrate AI tools — from Microsoft and external providers — with each customer’s internal data. Customers will own the results of that work rather than handing it back to Microsoft. The move puts Microsoft alongside Palantir, which is doing similar work with large customers using Nvidia’s open-source models, and Amazon Web Services, which recently launched a $1 billion embedded-engineering unit of its own.

What’s most telling is the reasoning. Judson Althoff, CEO of Microsoft Commercial Business, told Reuters the new firm grew partly out of Microsoft’s own experience watching models like DeepSeek and Google’s Gemini catch up to OpenAI. Referring to the original Copilot, he said, “we made a mistake by binding it to OpenAI models only.” Customers, Althoff said, care more about the combination of their data and the models than about any particular model — and they need the ability to swap models quickly as the state of the art shifts.

“We made a mistake by binding it to OpenAI models only.”

One model no longer fits

Consider a typical customer service application that might need to summarize a support ticket, analyze a 300-page contract, generate an email, transcribe a meeting and review source code. Those aren’t necessarily the same problem. A model like Google’s Gemini, with a context window of a million tokens or more, may be the right choice for the contract. A small, fast model like OpenAI’s GPT-5.4 mini or Anthropic’s Claude Haiku may handle ticket summaries at a fraction of the cost. The transcription may go to a purpose-built model like Whisper. And if regulators require customer data to stay on-premises, an open-weight model like Meta’s Llama or Mistral is often the preferred choice.

Instead of choosing a single foundation model, developers increasingly choose several — and the application decides which one handles each request.

AI gateways become core infrastructure

The model is just one component of the stack, so the decision to route the request has to live somewhere.  That’s why developers are forgoing hard-coding an application to a single model and building systems that can choose among several. The routing logic might prioritize cost for one request, speed for another, or keep sensitive workloads on a local model. That way, if one provider experiences an outage, traffic can be routed elsewhere without changing the application itself.

The company that arguably has the deepest single-model partnership in the industry is now selling model swappability as the product.

That changes what developers build

Once companies stop relying on a single model, the challenge shifts to building the systems that decide which model to use for each request.

That means developers need tools to route requests, compare model performance, monitor reliability, control costs, enforce security policies and switch to another model if one goes down, which is a very different engineering problem, especially since deciding which model should respond to a request occurs every time someone uses your application. At enterprise scale, those decisions happen millions of times a day, so they have to be fast, reliable and easy to manage.

The ecosystem is already responding

Open-source proxies like LiteLLM and gateways like Portkey normalize APIs across providers. Orchestration frameworks such as LangChain and LangGraph assume the presence of multiple models from the start. The Model Context Protocol (MCP) is making tool integrations portable across models rather than bound to one vendor. And the cloud providers themselves — Amazon Bedrock, Azure AI Foundry, Google Vertex AI — now expose many models behind a single API.

Orchestration is the new moat

Core models will keep improving. But as performance converges for many business tasks, orchestration becomes the challenge. Microsoft’s announcement is one indication that the largest vendors believe enterprises are heading in that direction and are willing to spend billions to be the ones holding the routing layer.

Instead of treating the model as the platform, enterprises are consistently treating it as a replaceable component behind an orchestration layer.

The cloud era taught developers not to tie applications too tightly to one server; containerization made infrastructure portable. Now the same philosophy is being applied to AI. Instead of treating the model as the platform, enterprises are consistently treating it as a replaceable component behind an orchestration layer.

The post Microsoft just admitted its biggest AI mistake — and spent $2.5 billion fixing it appeared first on The New Stack.

When Subquadratic launched earlier this year, it could build a sparse-attention model that could handle a 12-million token context window and be significantly faster than today’s large language models. But it didn’t launch the model widely and it didn’t publish benchmarks.

Given the company’s large claims, that created quite a bit of skepticism. In June, Subquadratic published its first model card and benchmarks for its small model, SubQ 1.1, supplied third-party verification from data firm Appen, and started talking about its first design partners who now have access to its model.

So far, however, few people have actually used its model. To talk about the company, why its model isn’t widely available yet, and what it has in store for the near future, we met up with Subquadratic co-founder and CTO Alex Whedon.

“We’re not a sparse attention company either.” — Alex Whedon, Subquadratic.

One thing Whedon definitely wanted to clear up is that the company’s current model may be based on sparse attention, but that isn’t its full mission.

“We’re not a sparse attention company either,” Whedon tells The New Stack. “We’ve been working on non-attention architectures for quite a while as well. We think that we will be the first people to leapfrog ourselves in terms of the next model architecture.”

We’ll get back to that.

What the model card shows

It’s the company’s SubQ 1.1 Small model that people are talking about now. This model is built on Subquadratic Sparse Attention (SSA), an attention mechanism the company says scales close to linearly with context length instead of quadratically.

“In the case of Subquadratic Sparse Attention specifically, which is one of a couple model architectures we worked with, the idea is that not all of the token relationships matter,” Whedon explains. “Token relationship compute is why you see this quadratic scaling law.” This means there are almost a million possible two-token relationships in a 1,000-token input in a full attention matrix.

For SubQ 1.1 Small, the strongest results are in long-context retrieval, which makes sense, given that this is where the architecture should have its biggest edge.

On the needle-in-a-haystack test, SubQ 1.1 Small scores near-perfect from 1 million tokens out to 12 million, even though it was trained mostly at 1 million. It hits 99.12 percent on Nvidia’s harder RULER test, which asks the model to trace and aggregate facts across a 128,000-token context rather than just find one.

On general capability, it lands just below the mid-tier frontier models, at 85.4 on GPQA Diamond against 87.5 for Sonnet 4.6. On the LiveCodeBench coding benchmark, it scores 89.7, below Opus 4.8 and GPT-5.5, but slightly better than Sonnet 4.6.

Efficiency is where the model shines, though. The company says that at 1 million tokens, SubQ uses 64.5x less compute than dense attention and runs 56x faster than FlashAttention-2 on a single attention layer. At the full 12-million-token window, it puts the attention compute reduction at close to 1,000x.

“Even in full dense attention, the relative importance of over 99 percent of tokens is very low, attention scores are below 0.1,” Whedon says. “We actually show this in our model card. So clearly we’re just wasting compute most of the time, and in fact we’re maybe making the modeling task harder, because we’re introducing noise.”

“Transformers are a brute-force approach to the problem of text modeling,” he says. “You could say, ‘I’m going to compare every single individual token to every other possible individual token.’ That’s what transformers do. Very brute force, very naive. It just assumes that the first needs to look at the second, the third, the 50th, and the 5,000th. That’s not how humans read text.”

SSA also differs from retrieval-augmented generation, which drops chunks of text before the model sees them. “Every token of the text is being seen by the model,” he says. “It’s just not being redundantly compared to every other token of the text.”

On capability, SubQ 1.1 Small lands roughly in Sonnet 4.6 territory, sometimes a bit above, sometimes below. But its edge, the company says, is size and cost.

“What we posted publicly was fewer than 100 billion parameters,” Whedon says about the size of the model. “I would venture to say that our model is smaller than any of the models offered by OpenAI or Anthropic. But our next model will not be.”

Smaller, cheaper, built for enterprises

Subquadratic is also making the pitch that its model’s capabilities will be especially interesting for enterprises.

“We think that’s a pretty interesting enterprise offering,” he says. “We’ve seen a lot of people in the enterprise space talking about using the mid-tier models as opposed to the frontier for large data-processing tasks, which is exactly where we’re trying to plug in.”

Given that a lot of enterprise problems start with searching through large heaps of data, this makes sense. You can pack a lot of documents into a 12-million token context window, after all. Most of today’s models break down well before the user fills their million-token windows, but with its near-perfect retrieval scores, SubQ may be a good answer for these problems.

As Whedon noted, the model’s first users are design partners, not the public. “We’re giving access to the model to design partners now, and these are mostly enterprises, largely with eight- to nine-figure spend,” Whedon says. “This is a core market that we really care about. It has been since day one.” A limited individual-access release will follow before any general availability.

The launch led with claims instead of benchmarks by choice.

“We were announcing mostly research,” he says. “We could have maybe messaged the launch a little bit differently. There was some debate about how we were going to message it.”

Built on an existing model

One question from May hasn’t gone away, though. The model card states that Subquadratic “started with an existing open-weight frontier model by replacing its dense attention with Subquadratic Sparse Attention (SSA),” and then ran roughly one trillion tokens of long-context continued pretraining on books, documents, and repository-scale code.

That confirms what some of the skeptics suspected at launch, when OpenAI researcher Will Depue wrote that SubQ was “almost surely a sparse attention finetune of Kimi or DeepSeek.” What’s new here then is the SSA mechanism and the long-context training recipe, not a model trained from scratch. The company has not said which open-weight model it started from.

The biggest lever on long-context retrieval was pretraining on very long sequences, Whedon says, something SSA’s efficiency made cheap enough to run as routine.

“Nobody’s talking about multimillion-token pretraining,” he says.

Why hybrids don’t go far enough

There have, of course, been attempts to improve on quadratic scaling, but Whedon thinks most of those attempts only go — almost literally — halfway. Hybrid models such as Nvidia’s Mamba-based Nemotrons, Qwen’s Gated DeltaNet layers, and the various linear-retention designs swap out some of the attention layers, but they don’t go all the way.

“If 80 percent of the layers are not quadratically scaling, then your maximum payoff is like a 5x increase as you scale toward infinity,” he says. “We see a 60x increase at 1 million tokens, almost 1,000x at 12 million. That is the type of payout that you only get if you actually change the scaling law, as opposed to a scalar win.”

He actually credits DeepSeek’s own sparse attention mechanism with making his company’s pitch easier.

“They showed that you could dynamically select relationships without a significant quality trade-off,” Whedon says. “However, they did so by redundantly using a smaller but still full-attention model that ends up using the vast majority of the compute at scale.”

Subquadratic ran its own benchmark against GLM 5.2. “At 1 million tokens, 58 percent of the prefill latency comes from that selection mechanism,” Whedon says. “So that selection mechanism, which is supposed to be seen as cheap, actually dominates the compute, because it’s a quadratically scaling component.”

Beyond sparse attention

It’s also why Whedon pushes back on the “sparse attention company” label. Subquadratic has been working on what he calls “zero attention,” architectures that drop the attention mechanism altogether.

“Attention is kind of similar to RAG in that you have queries, keys, and values that represent information about the tokens that you’re processing,” Whedon says. “There’s this discreteness of representation, where everything is represented within these nice little boxes. That’s super convenient. It’s easy to build a brute-force solution around it. But it also means your ability to compress information is limited. If you had a more continuous, abstract way of representing the information, then you could compress it further, which means you can make smaller models, or you could just scale things up again to create another leap in intelligence.”

He traces the idea to world models and to Yann LeCun’s work. “The stuff we’re doing takes a lot of inspiration from world models, not the video modality in this case, but some of the things LeCun is talking about,” he says. “Rethinking how to represent long-range dependencies, how to keep a long-range state, how to rethink the objective function.” He stops there. “That’s probably all I could say for now.”

Subquadratic has also marketed only one of the three kinds of efficiency it says it is chasing. “We care about compute, sample, and memory efficiency,” Whedon says. “We’ve done a lot of work on all three, but have only really talked about the compute efficiency publicly.”

The near-term plan

The near term plan for Subquadratic, however, is more modest. “Over time, yes,” Whedon says, when asked whether Subquadratic could rival OpenAI and Anthropic on raw quality in the long run. “In the shorter term, we have to be strategic. If we try to boil the ocean on much less capital, it’s not going to go well for us.”

The next model, he says, will likely be a mid-tier size rather than a frontier-class one (think SubQ 1.2 Medium), that he expects to outperform most of the competition in its tier.

How the team will bring the model to market, though, remains to be seen. I wouldn’t be surprised if the team launched its model on one of the hyperscaler’s large model platforms, but Whedon remained tight-lipped about the company’s plans.

The fact that we met with the Miami-based Whedon in San Francisco, though, gives you a bit of a hint of what the team is currently up to.

The post What comes after attention? This startup says it already knows. appeared first on The New Stack.

Z.ai, the Beijing-based artificial intelligence lab formerly known as Zhipu AI, on Wednesday officially launched ZCode, a free desktop application it describes as an "Agentic Development Environment" purpose-built for its flagship GLM-5.2 large language model. The move marks the company's most aggressive push yet into the fast-growing AI-powered coding tool market, where it now competes directly with Cursor, Claude Code, GitHub Copilot, and Google's Antigravity.

"Introducing ZCode, the official development environment for GLM-5.2," the company wrote on X, noting the tool is available on macOS, Windows, and Linux, supports bring-your-own-key (BYOK) configurations for third-party models, and offers a 1.5x usage-quota bonus for subscribers to its GLM Coding Plan.

Read one way, ZCode is simply another entrant in a crowded market. Read another, it is a single product that crystallizes three of the most consequential trends in enterprise software today: the race-to-the-bottom pricing of frontier AI models, the geopolitical balkanization of the AI stack, and the rapid maturation of agentic coding agents into what Gartner now estimates is a roughly $10 billion market.

An AI coding tool designed to think in projects, not prompts

Unlike traditional IDEs that bolt on AI through a chat sidebar or autocomplete extension, ZCode is best understood as an agent-first development environment. Its core design is built around long-horizon tasks: the user describes an outcome, the agent plans the work, edits files, runs checks, reviews progress, and continues across multiple iterations until the goal is met.

ZCode organizes the development experience around the ZCode Agent, deeply tuned for GLM-5.2, with emphasis on deep integration: the model, tools, and execution workflow are tuned together so the Agent fits continuous, multi-step real-world development tasks. The environment supports continuous follow-up across devices: desktop, mobile Remote, and Feishu / WeChat Bot can all keep the same workspace task moving. Sensitive commands, file changes, and high-permission actions go through confirmation before execution.

That remote-control feature — the ability to steer a running coding agent from WeChat, Feishu, or Telegram on a phone — is a differentiator that speaks directly to the Chinese developer market, where those messaging platforms dominate professional communication. You can keep checking progress and adding instructions while long-running work continues, from any device with these messaging apps.

The tool is free to download. Revenue flows through Z.ai's GLM Coding Plan subscription tiers, which start at $16.20 per month for a "Lite" plan and scale to $144 per month for "Max" — prices that undercut Anthropic's Claude Code and Cursor's comparable tiers by significant margins.

Through July 31, ZCode is offering a promotional 1.5x effective quota bonus for Coding Plan subscribers, with off-peak token consumption charged at a 0.67x coefficient. The platform also supports multiple AI models and agents, including Claude Code, Codex, Gemini, and OpenCode — a pragmatic concession to the reality that no single model wins every task.

GLM-5.2, the open-source model trained entirely on Chinese chips, powers the whole experience

ZCode's value proposition is inseparable from GLM-5.2, the model it was designed to showcase. Z.ai released GLM-5.2 on June 16, first to its Coding Plan subscribers and subsequently as open-source weights under the MIT license on Hugging Face — a sequencing decision that prioritized distribution over the traditional benchmark-led launch.

The model's specifications are formidable. GLM-5.2 is a 744-billion-parameter mixture-of-experts architecture with 40 billion active parameters, a genuine one-million-token context window — five times the 200K limit on its predecessor — and training on 28.5 trillion tokens. It ranked second globally on Code Arena as of mid-June, trailing only Anthropic's Claude Fable 5, making it one of the highest-performing publicly available models for coding tasks.

Critically, the model was built entirely without American chips. As Decrypt reported, GLM-5.2 "runs entirely on Huawei silicon." Stability AI founder Emad Mostaque estimated total training costs at roughly $25 million, with 80 percent spent on post-training — a figure that, if accurate, would make GLM-5.2 extraordinarily cheap relative to Western frontier models.

On benchmarks, GLM-5.2 performs within striking distance of the best proprietary systems. It trails Anthropic's Claude Opus 4.8 by just one percentage point on FrontierSWE, a benchmark measuring multi-hour autonomous engineering projects, while edging out OpenAI's GPT-5.5.

Its API pricing — $1.40 per million input tokens and $4.40 per million output — are a cost reduction of up to 82 percent compared to Anthropic's Claude Opus 4.8 at $5 and $25, respectively. Because ZCode is a first-party tool from the same company that makes the model, it requires no manual endpoint configuration — the model is wired in.

The Anthropic export ban gave Chinese AI its biggest opening yet

ZCode's arrival cannot be separated from the geopolitical drama that has roiled the AI industry over the past three weeks. On June 12, the U.S. government, citing national security authorities, issued an export control directive suspending all access to Anthropic's Fable 5 and Mythos 5 models by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees. Enterprise clients in finance, healthcare, SaaS, and critical infrastructure found their core intelligence services abruptly disabled, without exception, prior warning, or effective recourse.

While the Trump administration lifted those controls just yesterday — Anthropic confirmed on June 30 that the Department of Commerce had rescinded the directive — the episode sent shockwaves through the developer community and accelerated interest in open-source, self-hostable alternatives. The government's crackdown on Anthropic coincided with a swift rise in Chinese open-source models that are proving to be almost as capable and significantly cheaper than some of the most powerful U.S. models.

Z.ai's timing was surgical. On the same day the Trump administration ordered Anthropic's most advanced models blocked for foreign nationals, Zhipu announced the open-source release of GLM-5.2 with no usage restrictions. The South China Morning Post reported that GLM-5.2 would be available to all users of Zhipu's new GLM Coding Plan subscription, "priced at just a tenth of Anthropic's premium Claude Code and Claude Max tiers."

The market responded accordingly. Zhipu AI's market capitalization crossed HK$1 trillion (US$128 billion) on June 22, driven by a 42 percent intraday share surge. JPMorgan raised its 2026–2030 revenue forecast for Zhipu by between 7 and 16 percent following the launch, projecting an over 534 percent revenue surge for 2026 and expecting the AI firm to turn a profit by 2028.

Why vendor lock-in now carries a geopolitical risk that no SLA can cover

The Fable 5 episode did more than embarrass Anthropic. It introduced a new risk category into enterprise AI procurement: sovereign access risk. When a government can disable a commercially deployed AI model overnight, the traditional evaluation criteria of developer experience, benchmark scores, and pricing become secondary to a more fundamental question: Will this tool still work tomorrow?

The event exposed the inadequacy of standard enterprise contract language. An investigation by FifthRow found that almost all standard Data Processing Addenda, SaaS agreements, and procurement SLAs "relied on vague 'force majeure' or 'compliance with law' catch-alls, not on precise, actionable regulatory suspension or kill-switch clauses."

ZCode's BYOK architecture and GLM-5.2's MIT-licensed open weights offer a partial answer. A development team can download the model, host it on its own infrastructure, and run ZCode against it without ever touching Z.ai's cloud — eliminating both American export-control risk and Chinese data-sovereignty concerns in a single move. The catch is that anyone using Z.ai's cloud API remains subject to Chinese law, a consideration that evaporates only with pure self-hosting.

Gartner analysts have warned that governance, pricing, support, workflows, commercial maturity, and market durability matter as much as developer experience and model capabilities when evaluating coding agent vendors for enterprise-wide adoption. By that measure, ZCode faces a steep climb. It is not open source itself; Linux support remains in beta; and security reviewers have flagged the need for careful evaluation of its credential handling, particularly for remote development over SSH and messaging-platform-triggered tasks — an agent that can be summoned from WeChat involves access paths that should be mapped before trusting it with anything sensitive.

Inside the $10 billion race where model labs are becoming full-stack IDE companies

ZCode enters one of the most crowded and fastest-moving markets in enterprise software. Enterprise AI coding agents are capturing a growing share of enterprise software engineering spend, with the market estimated at roughly $9.8 billion to $11.0 billion annualized as of April 2026, according to Gartner. A defining shift this year, the analyst firm noted, is "the movement of frontier model providers into direct competition with application-layer vendors" — precisely the pattern ZCode embodies.

Gartner codified this evolution in May when it renamed its annual Magic Quadrant from "AI Code Assistants" to "Enterprise AI Coding Agents," defining the category as "autonomous or semiautonomous software engineering solutions that perceive context, translate human intent into multistep plans, and execute and verify those steps across code, tests and related engineering artifacts." The 2026 Magic Quadrant names Anthropic, Cursor, GitHub, and OpenAI as Leaders. Z.ai was not among the 12 vendors evaluated — an absence that underscores both the company's nascent enterprise sales presence outside China and the Western-centric lens through which the analyst community still views the market.

The competitive landscape is daunting. Cursor is the $2 billion ARR IDE that feels like VS Code with a supercharger. Claude Code reached approximately $2.5 billion in annualized revenue by early 2026. Google relaunched Antigravity 2.0 at I/O in May, and Cognition retired the Windsurf brand, relaunching the IDE as Devin Desktop with the Agent Command Center as the default surface.

Against these entrenched players, ZCode's pitch rests on three pillars: deep first-party integration with GLM-5.2 that no third-party editor can replicate, aggressive pricing that starts at a fraction of Western competitors, and MIT-licensed open weights that allow enterprises to self-host — eliminating the regulatory kill-switch risk that the Fable ban made viscerally real.

Z.ai's real challenge is turning a $128 billion valuation into a global developer tools business

Z.ai controls the model (GLM-5.2), the subscription layer (the GLM Coding Plan), and the IDE (ZCode) — a tightly coupled stack that optimizes for performance but concentrates switching costs. For the company, the business logic is clear. Its most reliable revenue stream has been on-premises deployments for Chinese government agencies, state-owned banks, and energy conglomerates. In full-year 2025, on-premises deployment revenue reached RMB 534 million, growing over 100 percent year-over-year and accounting for 73.7 percent of total revenue with a gross margin of 48.8 percent. ZCode and the GLM Coding Plan represent the company's bid to build a comparable revenue engine in cloud-based developer tools — globally, not just in China.

The early signals are encouraging for Z.ai, if anecdotal. Community reception on X was enthusiastic, with one early user calling the tool "super stable" and others clamoring for more Coding Plan capacity. "Bro, can't snag your family's Coding Plan? When are you gonna stock up on more cards?" one user wrote in Chinese, suggesting demand is already outstripping supply.

But the hard questions loom large. Can a Chinese AI company build trust with Western enterprise buyers amid escalating technology tensions? Can ZCode's ecosystem mature fast enough to compete with Cursor's polished UX, Claude Code's deep agent primitives, and GitHub Copilot's unmatched distribution? And can Z.ai sustain a company valued at $128 billion while still losing money? 

What is no longer in question is the competitive dynamic itself. Three weeks ago, a U.S. government directive proved that access to the world's best coding model can vanish overnight. Today, a Chinese lab is shipping a free IDE, an open-source model trained on zero American chips, and a subscription plan that costs less per month than a single lunch in Manhattan. The AI coding agent market did not just become global this summer. It became a market where the fallback option might be better than the thing it's falling back from — and that changes the calculus for every engineering leader choosing a toolchain in the second half of 2026.

OpenClaw finally dropped its iOS and Android apps this week, meaning you can now ditch the Telegram and WhatsApp methods to talk directly to your personal AI agent. But what’s arguably more exciting is that the app isn’t actually running the AI on your phone. It’s just hooking up to an agent you’ve already got running somewhere else. Your phone now acts as a window into that agent, complete with voice, notifications, and camera access.

It’s a nice design choice, and exactly where personal AI agents are headed.

Phones become authenticated endpoints

The phone is basically becoming a really smart remote control for OpenClaw. Instead of cramming an increasingly powerful agent onto a phone with battery and memory constraints, developers are treating the phone as one more screen for an agent that lives elsewhere. The agent keeps working whether your phone is in your hand or charging in the other room.

Within this model, the phone approves actions, pings you with notifications, lets you talk to the agent, and shares your camera when the agent needs eyes on something.

Persistent runtimes replace mobile constraints

But OpenClaw isn’t the first to do this. Anthropic’s Claude Cowork with Dispatch follows a remarkably similar pattern. Users assign work from their phones, but execution occurs on a persistent desktop runtime. The mobile app acts as a companion for starting tasks, monitoring progress, and receiving results rather than becoming the agent itself.

OpenAI is moving in a similar direction as well. With Codex, developers increasingly interact with long-running coding agents that continue working independently and can be checked on from multiple clients, instead of treating the phone as the place where the agent runs.

Different companies, different products, but a similar architectural bet to keep the agent running in a persistent runtime and give people lightweight clients to interact with it.

When multiple teams independently converge on the same architectural pattern, it’s often an early signal that the industry has found a model that solves a real engineering problem.

The engineering problems are totally different now

This shift changes what developers spend their time thinking about. Building mobile apps used to mean worrying about battery life, memory limits, offline mode, and squeezing the best performance out of a phone. If the agent is running somewhere else, most of those concerns fade into the background.

Now, a new set of questions comes to mind, such as how a phone securely connects to a long-running agent? How do you manage permissions across multiple devices? What happens if every client disconnects but the agent keeps working?

Agent identity beyond login screens

There’s a downstream effect here. Once the phone is just one of several trusted endpoints talking to your agent, you need a much more robust approach to identity. You’re not logging a user into an app anymore. You’re authenticating devices into an ongoing relationship with a persistent agent.

As that agent gains the ability to read your files, send emails, call APIs, and control external tools, authentication becomes load-bearing infrastructure.

Distributed agents reshape developer tooling

Zooming out a bit and looking at the bigger picture highlights how personal AI agents increasingly resemble distributed systems rather than mobile apps. The intelligence lives in a persistent runtime while the phone is one authenticated endpoint among several.

For developers, the mobile app is only part of the job. They also have to build the components that keep an agent running, connect it to a user’s devices, and ensure those connections remain secure.

The agent keeps running independently, while the phone is simply another place to check in, approve actions, or start a conversation.

Looking at OpenClaw alongside Anthropic and OpenAI, it’s hard not to notice the same pattern. The agent keeps running independently, while the phone is simply another place to check in, approve actions or start a conversation. That architecture solves many practical problems, which may explain why several companies are heading in the same direction.

The post OpenClaw’s new app doesn’t run AI on your phone. That’s the whole point. appeared first on The New Stack.

AI has changed the web right before our eyes. With Google’s AI Overviews doing the heavy lifting, publications that once owned the first page of search results are being replaced by summaries. Readers get their answer without ever clicking through. Much of the traffic has simply stopped.

Cloudflare on Wednesday announced a slew of updates for publishers who are facing this new reality. From new crawler classifications and analytics dashboards to Answer Engine Optimization tools and an expansion of its Pay Per Crawl program, it’s clear the company is trying to become the economic pipes of the AI web.

The shift from ‘keep out’ to ‘let’s make a deal’

A year ago, Cloudflare’s pitch was practically defensive, asserting that website owners should be able to block AI crawlers. And while that still holds true, the company has pivoted to discussing building “rails” for an “agentic economy.” And it makes sense. If AI agents are already browsing the web, collecting content, and in some cases buying things, someone needs to handle the business end of how the sites they visit are compensated. Cloudflare thinks that someone should be Cloudflare.

Paying for value, not visits

Roughly a year ago, Cloudflare launched Pay Per Crawl, which let publishers set a price for AI companies to pay when they fetched a page. Now the company is pushing toward Pay Per Use, which means publishers get paid when their content actually appears in an AI-generated answer.

To backtrack, under the old model, an AI crawler pays to visit your site, whether or not it does anything useful with what it finds. Cloudflare says it’s already testing this with Ceramic.ai and You.com, each running slightly different versions of the concept.

Instead of charging for access — which is basically a toll booth — publishers are charging for value.

The economics here flip. Instead of charging for access — which is basically a toll booth — publishers are charging for value. That’s closer to how affiliate marketing or licensing deals work, and it’s a much harder problem to solve. It requires knowing which content contributed to which answer, which means an attribution infrastructure that doesn’t really exist at scale yet.

Crawlers need clearer labels

But here’s where things get more technical — and a bit political.

Cloudflare wants AI companies to stop lumping all their crawlers together. Right now, a single bot from a major AI company might be fetching pages for search indexing, model training, and agent tasks all at once. That makes it impossible for site owners to say yes to one use and no to another.

Starting September 15, Cloudflare plans to change the defaults for new and free-tier sites. AI search crawling stays on, but training and agent access get blocked on ad-supported pages unless the site owner opts in. Mixed-use crawlers that refuse to separate their traffic get blocked entirely.

The company is clearly taking a shot at Google here, noting that Google’s bundled approach gives it access to roughly twice as much content as AI-native competitors. Separating crawlers’ intent has become a prerequisite for any kind of functioning market between publishers and AI companies. Because if site owners can’t distinguish between “index my page for search” and “train your model on my writing,” they’ll increasingly just block everything.

Separating crawlers’ intent has become a prerequisite for any kind of functioning market between publishers and AI companies.

Optimizing for AI answers

Cloudflare is also rolling out a dashboard designed for business teams, called Attribution Business Insights. Think of it as the AI equivalent of knowing your Google Search Console numbers, except the “search engine” is now ChatGPT or Perplexity or whatever agent your reader happened to ask.

On top of that, Cloudflare is introducing Answer Engine Optimization (AEO). The idea is that ranking in Google is no longer enough; for publishers to succeed, they also need to understand how and where their content gets cited in AI-generated responses. That’s a different optimization problem than SEO, and right now almost nobody has good tooling for it.

Infrastructure as competitive advantage

Cloudflare already sits between websites and the internet. It sees the traffic and knows what changed on a page and what didn’t. It can tell a crawler to come back later because nothing’s new, which, by the way, it says would eliminate over 50% of current AI crawl traffic.

That puts Cloudflare in a unique position. It’s already part of the path between AI companies and the web. Now it wants to become the layer that manages access, attribution, and eventually payments between them. If AI agents become the primary way people find information online, Cloudflare is betting the next battle won’t be over who builds the smartest model but instead over who builds the infrastructure everyone else relies on.

If AI agents become the primary way people find information online, Cloudflare is betting the next battle won’t be over who builds the smartest model — it’ll be over who builds the infrastructure everyone else relies on.

The post Cloudflare wants to build the economic layer of the AI web appeared first on The New Stack.

For two years, the AI infrastructure race has been dominated by one question: Who has the fastest GPU? Jim Keller thinks that’s becoming the wrong question.

In a recent interview with EE Times, the Tenstorrent CEO argues that the riskiest move an organization can make right now is optimizing its AI infrastructure for the models it’s running today. It’s not because those models are bad — but because they won’t be the models it’s running in 18 months. Keller invoked Rent’s Rule and Amdahl’s Law to argue that memory, networking, and system-level balance now matter more than peak floating-point performance.

Not because those models are bad — but because they won’t be the models it’s running in 18 months.

It sounds like the start of Keller’s product pitch, but there’s real weight behind it, because AI has evolved faster than the infrastructure underneath it. And the companies that spent hundreds of millions building around one generation of models are now staring down the cost of doing it all over again.

That fear is called lock-in, and it’s reshaping how the biggest players in AI think about hardware.

Workloads outgrew the GPU

In 2023 and 2024, AI infrastructure was a relatively simple procurement problem: Train large language models, serve them to users, and buy as many GPUs as Nvidia can ship. The workloads were predictable and GPUs handled them well.

Then AI outgrew the infrastructure it had been built for.

Reasoning models spend more time working through problems instead of jumping straight to an answer. Agents bounce between APIs, databases and code before completing a task. Multimodal models mix text with images, audio and video. None of those workloads stress hardware in quite the same way — and that’s forcing infrastructure teams to rethink assumptions that made perfect sense just two years ago.

AI outgrew the infrastructure it had been built for.

No single chip architecture handles all of that equally well. And the organizations building AI infrastructure are starting to realize that the question isn’t just which accelerator is fastest — it’s how do we build systems that won’t need to be torn apart every time AI takes another leap?

Nvidia is already selling one answer

Look at what Jensen Huang has been talking about, and it’s not GPUs anymore.

At GTC 2026, Nvidia unveiled the Vera Rubin platform — seven chips designed to operate as a single system: the Rubin GPU, Vera CPU, NVLink 6 switch, ConnectX-9 networking, BlueField-4 DPU, and more. The Vera CPU exists

for the CPU-intensive work of agentic AI — tool calls, code execution, orchestration. Nvidia calls these deployments “AI factories,” and the language is deliberate. They’re selling complete infrastructure, not individual accelerators.

When the company with 70% market share stops leading with GPU benchmarks and starts talking about system-level co-design, it tells you where the center of gravity in this market is moving.

That reframing matters. When the company with 70% market share stops leading with GPU benchmarks and starts talking about system-level co-design, it tells you where the center of gravity in this market is moving. Compute still matters. But Nvidia is conceding — through its product architecture if not its marketing — that raw accelerator performance alone won’t be enough for what’s coming.

Hyperscalers design their own silicon

AMD sees the same problem, even if it’s taking a different route. Helios brings together CPUs, GPUs and networking into one rack-scale platform, reflecting a broader shift away from treating the GPU as the center of the universe. The pitch isn’t “our accelerator is faster.” It’s that the infrastructure surrounding the chip increasingly matters just as much as the chip itself.

The hyperscalers have been making this argument with their wallets for even longer. Google has spent a decade co-designing its TPU silicon, interconnects and software framework — its seventh-generation Ironwood chip is now generally available — giving it unusual control over the full stack. Amazon went the opposite direction, building separate chips for separate jobs: Trainium for training, Inferentia for inference, with Trainium3 now in production and serving customers like Anthropic. Microsoft’s Maia 200 targets inference costs, while the company simultaneously deploys Nvidia’s Vera Rubin NVL72 for training and experimentation — arguably the most pragmatic dual-track strategy in the market.

Behind all of them sits Broadcom, whose AI semiconductor revenue recently crossed $10 billion in a single quarter, driven by staggering demand for custom accelerators and data center switches. Broadcom designs custom accelerators for Google, Meta and others while supplying the Tomahawk and Jericho switch silicon that connects those accelerators at data center scale. Custom ASIC shipments are projected to grow roughly 45% year over year in 2026 — triple the growth rate of merchant GPUs.

Then there are the companies that decided building a better GPU wasn’t the answer.

Cerebras questioned the need for thousands of interconnected chips, opting instead for a wafer-scale processor that keeps far more of the workload on a single piece of silicon. Groq took the opposite approach, optimizing almost entirely for inference. SambaNova focused on enterprise AI, building systems where efficiently serving multiple models matters more than posting the fastest benchmark.

Adaptability beats raw speed

The first wave of generative AI rewarded whoever could buy the most compute. That made sense when most organizations were solving the same problem. Today, AI workloads are changing so quickly that infrastructure teams are starting to optimize for something different: adaptability.

Keller’s example illustrates the point. Tenstorrent’s BlackHole architecture uses standard Ethernet instead of proprietary interconnects, allowing its hardware to slot alongside existing GPU deployments rather than replacing them. Keller told EE Times that one customer used Tenstorrent’s Galaxy servers to increase token throughput on GPUs they already owned instead of rebuilding their infrastructure from scratch.

Whether Tenstorrent’s approach becomes the industry standard is almost beside the point.

The bigger idea is already spreading. Across the industry, companies are spending less time asking how to build the fastest AI hardware and more time asking how to build hardware that won’t have to be replaced every time AI takes another leap forward.

The question that matters now

No one knows what AI workloads will look like three or five years from now. That’s the problem.

Infrastructure refresh cycles are measured in years. AI models seem to reinvent themselves every few months. Building around today’s workloads is starting to look like a risky bet when tomorrow’s could demand something different.

Infrastructure refresh cycles are measured in years. AI models seem to reinvent themselves every few months.

Every company is responding in its own way. They have different strategies but are still asking the same question: How do you build infrastructure that outlasts the AI running on it?

That may prove to be a more important engineering challenge than building the next record-breaking accelerator. And it’s the challenge driving companies from Nvidia and AMD to Google, Amazon, Broadcom and Tenstorrent.

The post The infrastructure lock-in costing AI companies hundreds of millions appeared first on The New Stack.

Amazon EKS runs hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Operating at that scale has taught us something that has shaped how we build the service and that we think is useful to anyone running Kubernetes at scale: most availability problems do not stem from a component failing. They come from a component reacting to a problem in a way that makes it worse. A cache that goes stale and serves wrong answers. A health check that restarts the very process keeping a cluster alive.

What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage. This post is the story of how we keep the EKS-managed Kubernetes control plane on the right side of that line at ever-growing scale: the foundational changes we made and why, and what operating at fleet scale taught us about building systems that tolerate faults rather than spreading them. These are the reasons our most demanding customers confidently run their mission-critical workloads on EKS.

How AI and analytics workloads reshaped what “scale” means

Kubernetes was built for a particular rhythm of work. Pods came and went at predictable rates, and controllers had seconds or minutes to reconcile. The system’s design reflected that pace: strong data consistency, ordered watches, and consensus-replicated storage that puts correctness first. It worked beautifully for what it was designed to do, and it still does.

“What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage.”

But the workloads evolved faster than anyone anticipated. Foundation model training runs scale-up training jobs on thousands of GPU nodes in minutes. Real-time inference services scale from a warm baseline to thousands of replicas, then drop back within the hour. Apache Spark analytics pipelines burst from zero to tens of thousands of executor pods, chew through a dataset, and vanish. 

Emerging agentic AI workloads add yet another dimension: autonomous agents that spin up, fan out, execute tasks, and tear down in seconds or less. These workloads share a trait that distinguishes them from traditional microservices: they generate enormous volumes of state transitions within compressed time windows and are deeply intolerant of delays. This velocity of state change pushed us to reinvent some of the mechanics to support a scale that was previously impossible, and to contribute what we could upstream.

How EKS reimagined Kubernetes storage foundation

Every Kubernetes cluster depends on etcd as its source of truth. Every application, every service endpoint, every scheduling decision is stored there. If etcd loses data, the cluster forgets everything it knows. Protecting that state is the most important job for a managed Kubernetes service.

Operating etcd for one cluster is well understood. Operating it for a fleet of millions is a different problem entirely. Hardware fails, networks blip, and disks degrade, so something has to handle those events without a human in the loop. And the operations etcd needs most, like replacing a failed member or recovering after a zonal event, are exactly the ones where a person acting under pressure can make a mistake that causes permanent data loss.

From the beginning, we built an operator agent that runs alongside every etcd instance and automates its entire lifecycle. The agent has two jobs. First, backup and recovery: it takes point-in-time snapshots and stores them durably outside the cluster. If too many instances are lost at once and the survivors cannot form a majority, the agent automatically detects the condition and rebuilds from the latest snapshot. Second, membership management: when an instance fails, the agent removes the terminated member and adds its replacement in an order that protects quorum and prevents split-brain.

A recent, more fundamental change was replacing etcd’s consensus mechanism, Raft, with a purpose-built journal that provides durable, ordered storage independently of etcd. In traditional etcd, a majority of members must agree on every write before it is committed. If two of three are unhealthy, the cluster becomes unavailable. 

By offloading durability to the journal, etcd peers no longer negotiate quorum among themselves. Writes commit as soon as the journal acknowledges persistence, and that entire class of etcd quorum-loss failures disappeared. Since the journal handles persistence, etcd no longer needs to fsync writes to local disk, so its data store has moved to an in-memory filesystem. What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path. For a deeper look at this architecture, read “Under the hood: Amazon EKS ultra scale clusters.”

“What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path.”

For ultra-scale clusters, we went further and partitioned etcd into resource-specific shards. Each partition operates independently with its own storage budget and throughput capacity. The primary value is failure isolation. In a monolithic deployment, if the events keyspace exceeds its quota because a misbehaving controller creates objects faster than garbage collection can remove them, it blocks writes to everything, including node leases. 

Suddenly, healthy nodes appear unhealthy because their lease renewals are being rejected. With partitioned etcd, the events partition hits its quota, but the leases partition continues operating normally. Nodes remain healthy. The scheduler keeps running.

What replacing etcd’s consensus mechanism unlocked

Removing the quorum requirement allowed us to make a change we had wanted for a long time: running etcd on the same host as the API server. In the traditional layout, every read and write crosses the network between separate machines. Each trip is fast on its own, but at thousands per second, the travel time adds up. With collocation, the API server talks to its local etcd over a loopback interface, and pod scheduling and controller reconciliation get measurably faster. For workloads where job controller queue depth is the binding constraint, shaving milliseconds off each API call means many more jobs are processed per second before the queue starts growing.

This is where the operator agent paid off. When etcd runs on the same host, an etcd member comes and goes whenever a control-plane host is replaced, which happens routinely. That only works if membership management is completely safe and automatic, which is exactly what the agent was already doing. We did not have to build a colocation from scratch; we built it on top of infrastructure that had been managing etcd membership safely since day one.

Collocation also taught us a lesson worth passing on: the convenient path needs a failover in case it breaks. The local etcd is the fast path, but if it becomes impaired, the API server fails over to another etcd member that is actively serving other API servers from the same journal. When you optimize for the common case, design just as deliberately for the moment that optimization is not available.

Fixing bottlenecks across the stack

At extreme scale, you have to address bottlenecks across the entire Kubernetes stack, and most of them are not bugs in the traditional sense. They are design choices that were correct at the scale Kubernetes originally targeted and break down only when the numbers get large. Rather than working around them internally, we fix them upstream so the entire community benefits.

One example involved the watch cache, the in-memory layer that distributes state changes from etcd to every controller watching for updates. When a controller starts, it requests a full snapshot of the current state via a mechanism called WatchList, and the existing implementation holds a shared read lock for the duration of the response build. 

At hundreds of thousands of objects, that work runs long enough to starve the writer that needs exclusive access, so the cache’s resource version cannot advance. Consistent reads see a stale cache and fail over to etcd, while the response building churns through hundreds of thousands of allocations under the lock. We identified this as a limitation in the watch-cache’s locking model and are working with the community to refactor the underlying data structures and interfaces to eliminate the contention.

The same shape appears elsewhere. In the Horizontal Pod Autoscaler, a single mutex protecting the scaling state becomes a serialization point at high HPA counts, where workers spend nearly all their time blocked rather than doing useful work. A redesigned data store (PR #139142) restores parallelism and raises reconciliation throughput by orders of magnitude. In the scheduler, we identified a bottleneck (issue #138426): every scheduling cycle rebuilds a set of in-use persistent volumes by scanning every node in the cluster, even for pods that do not use storage at all. The fix computes that information lazily, and only for pods that actually need it, restoring throughput at scale.

Each of these started from a real production workload hitting a cliff, and we are working on the fixes upstream so the improvements reach every Kubernetes user.

From engineering to guarantees: EKS Provisioned Control Plane

The engineering described above made the EKS control plane more resilient and performant. But customers had a different problem: they could observe that the control plane kept up today, but they could not reserve its capacity the way they reserve compute or GPU capacity. 

A team planning a thousand-node training run could secure the instances weeks in advance, yet had no equivalent mechanism for the orchestration layer that would coordinate them. EKS Provisioned Control Plane fills that gap. It exposes the control plane’s performance as dimensions you size explicitly, backed by the same kind of commitment you expect from the rest of your infrastructure.

You choose a scaling tier that maps to concrete, measurable capabilities: API request concurrency, pod scheduling rate, and cluster database size. The tiers range from XL through 8XL. At the top end, 8XL on Kubernetes 1.34 provides 16,000 concurrent API request seats, 400 pods-per-second scheduling rate, and 16 GB of cluster database storage, all backed by a 99.99% availability SLA measured in one-minute intervals.

Tiers are not static. You step up before a GPU training run or a large sales event, step back down during quiet periods, or grow permanently as your platform matures. Configuration happens through the console, CLI, eksctl, CloudFormation, or Terraform on any cluster, without recreation or downtime. 

For AI workloads, orchestration capacity is planned alongside GPU capacity, available when the compute comes online. For analytics platforms submitting hundreds of jobs per minute, the control plane is ready for the burst before it arrives. And for organizations that need environmental consistency across staging, production, and disaster recovery, the same tier guarantees consistent performance characteristics everywhere.


Taking the same foundation to the edge

Some workloads cannot move to the cloud, whether due to data sovereignty requirements, latency constraints, or unreliable connectivity to the Region. Running Kubernetes in these disconnected environments introduces unique challenges: etcd must remain durable on hardware with only a few machines, the cluster must self-heal without reaching the cloud, and observability must survive network partitions that last days. 

With the updated architecture for EKS local clusters on instance store Outposts, we brought edge clusters onto the same management plane and software stack as EKS clusters in the cloud.

The control plane lives in an EKS-managed account on the Outpost rather than in the customer’s account, so customers never manage control plane instances, etcd backups, or logging agents themselves, and they cannot accidentally break the thing keeping their cluster alive. The same machine images, container images, and operator agent run in both places, with edge-specific behaviors selected by configuration. 

Because it is the same stack, new Kubernetes and EKS platform versions arrive in lockstep with their cloud release, and features like EKS add-ons, Pod Identity, and access entries work the same way they do in a Region.

The hardest part was keeping etcd healthy on hardware with only a few machines that may be cut off from the cloud for days at a time. We solved it by extending the same agent. It keeps a spare copy of the data continuously up to date and promotes it the instant a machine fails, so the cluster heals itself with no human involvement and no connection to the cloud. 

Observability survives the disconnect, too: the metrics agent continues collecting and writing to local disk, shedding the least critical data first when space runs short, so the signals that matter most are the last to go. When the link returns, the buffered data is flushed back with its original timestamps.

All of this only works because the system was designed from the start to operate without anyone logged in. That same design is what makes it possible to deploy changes safely across the entire fleet.

Operating safely at fleet scale

Every one of these changes was deployed to a running fleet of hundreds of thousands of clusters. The journal migration and collocation required transitioning each cluster individually. Every migration follows a strict sequence: validate pre-conditions, create a point-in-time snapshot, perform the switchover, validate post-conditions. If any step fails, the system rolls back automatically. 

Rollouts proceed cell by cell, zone by zone, region by region, with automated monitoring comparing latency, error rates, and throughput between updated and non-updated clusters. Any statistically significant deviation triggers an automatic halt.

What made all of this possible is that EKS is built to operate without human intervention at the individual cluster level. Through Zero Operator Access, the architecture prevents AWS personnel from having technical pathways to access customer content in the managed control plane. A system designed to work without human access must be observable, recoverable, and automatable from the start, and that same discipline is what enables operating at extreme scale.

Three operational lessons shaped how we approach this work.

The first is that a healthy leader is not the same as a working one. The control plane’s controllers run in an active-passive configuration, and early on, we treated an unhealthy standby as if cluster operations had halted. They had not; what matters is whether a leader exists. But the harder lesson: a leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth. If the queue fills while the leader looks healthy, the system is falling behind in ways no liveness probe will catch.

“A leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth.”

The second is that maintenance ordering matters as much as the maintenance itself. etcd defragmentation is blocking, and the pause grows with database size. When it hit the leader, every write stalled. We taught the agent to move leadership to a healthy node before defragmenting, so the disruptive work always lands on a follower while writes keep flowing.

The third is that liveness is not readiness. A process can be alive but not ready while it warms caches, and routing based solely on liveness sends requests to an instance that cannot handle them. Equally, readiness flapping during graceful draining should never trigger a restart. We keep the two signals strictly separate: one decides recovery; the other decides routing.

None of this work is visible from the outside, and that is the point. The largest clusters taught us lessons that made every cluster faster. The riskiest migrations produced safety machinery that protects every upgrade. The upstream fixes we contributed for workloads at the edge of what Kubernetes can handle flow back to every user of the project.

“None of this work is visible from the outside, and that is the point.”

When you deploy on EKS and your pods come up in seconds, even during a burst, even when something behind the scenes goes wrong, that speed is not accidental. It is the accumulated result of years of operating at scales where small problems can become big ones fast, and engineering the system to contain them before they do.

To explore the architectures referenced in this post, see EKS Provisioned Control Plane and local Amazon EKS clusters on AWS Outposts.

The post Operating Kubernetes at scale: a few stories from running Amazon EKS appeared first on The New Stack.

Building with AI has, until recently, meant calling up someone else’s model. But wiring an app to an API from OpenAI, Anthropic, or Google is a poor fit for government and operators of critical infrastructure. When data legally or operationally cannot leave a secured network, a hosted endpoint in someone else’s cloud is a non-starter, regardless of whether the workload is intelligence analysis, grid operations, or patient records.

“The most revealing aspect here is that Palantir didn’t ship a model, but the apparatus for deploying and owning one.”

That’s the lens through which to read Palantir’s announcement on Monday. The company introduced an “intelligent engine,” built on Nvidia AI and Nemotron open models, for running, customizing, and continuously improving AI inside air-gapped and other sovereign environments while keeping data and model weights in the customer’s hands. The most revealing aspect here is that Palantir didn’t ship a model, but the apparatus for deploying and owning one.

From calling AI to operating it

Most organizations will run a hybrid of hosted and self-hosted models for years, but a second pattern is hardening alongside this workflow. Rather than treating the model as an external service, the application communicates with an internal AI platform that routes requests to one or more models running on the organization’s GPUs. Data never leaves the perimeter, which hands security and compliance teams direct control over governance, auditing, and retention.

So we’re seeing the question shift from which model we should call to which models we should own and operate ourselves. And, just as importantly, what does it cost us to run them?

What’s actually in the box

This is where the announcement gets more concrete than the press release lets on, and where it’s worth knowing what Nemotron is.

Nemotron is Nvidia’s family of open-weight models, released in 2026 in three sizes — Nano (~31.6B parameters), Super (120B), and Ultra (550B). All three use a hybrid Mamba-Transformer mixture-of-experts design that activates only about a tenth of their parameters per token, so they run far cheaper than their headline sizes suggest, with context windows up to a million tokens.

Nvidia publishes the weights, training data, and recipes under a permissive license, and the models are deployable via open runtimes such as vLLM, SGLang, and llama.cpp, Ollama — or as Nvidia NIM microservices, the containerized, TensorRT-LLM-optimized path that ships inside the Nvidia AI Enterprise suite. Palantir, notably, was already on Nvidia’s published list of early Nemotron adopters, so this is less a cold start than a formalization.

An important note for builders weighing the move: Nemotron doesn’t top the raw-capability leaderboards; open families like DeepSeek, Qwen, and Kimi K2 generally score higher on absolute benchmarks. Nemotron’s pitch is efficiency on Nvidia silicon and genuine openness, which is exactly the axis that matters when the deployment target is your own hardware behind an air gap rather than a hosted endpoint.

Palantir’s contribution is the layer that turns “download the weights” into “run this in a classified environment and keep improving it.” The company describes three engineering surfaces: deployment (getting base and customized models into air-gapped and classified networks), context (prompts, workflow structure, and model behavior in production), and model (changing the weights themselves on proprietary data and mission outcomes). Underneath sits Palantir’s existing stack — AIP, Ontology, Foundry, and Apollo — handling data authorization, enforced isolation, and auditability.

The “self-improving” claim is a telemetry loop: The engine captures usage and trace data, then uses it to post-train and align the model toward the tasks where it adds value. This is the part that demands the most operational discipline, because a feedback loop with no evaluation harness is a recipe to drift.

The catch: Owning the stack means owning the stack

The sovereignty pitch is genuinely attractive, and it has a price tag the announcement doesn’t dwell on. Owning your models means owning everything around them. That’s GPU capital expenditure and the power and cooling to match; an inference stack you keep patched and performant; a model lifecycle — fine-tuning, evaluation, rollback — that you staff and run yourself; and a security burden that doesn’t go away just because the box is disconnected.

“Owning your models means owning everything around them.”

Ultra-class models, in particular, are not modest: running a 550B model, even at ~10% activation, requires multi-GPU server nodes, not a spare rack.

For an agency that legally cannot use a hosted API, that cost is simply the cost of doing the work, and the calculus is easy. For a commercial enterprise that could use a hosted endpoint, the math is a real trade — control and data residency on one side, capex and operational headcount on the other. The right answer is workload-specific, and anyone selling it as obvious is selling something.

Read the announcement for what it is

It’s also worth being clear about what this news is not. It’s a packaging-and-positioning announcement between two companies with aligned incentives: NVIDIA sells more GPUs and more AI Enterprise licenses, Palantir sells more platform, and “sovereign AI” is the banner both have been marching under all year. There’s no named agency customer, no contract value, and no benchmarks. Palantir CEO Alex Karp says many U.S. clients already use these models and frames the appeal as avoiding proprietary insight being baked into the weights of closed models; NVIDIA’s Jensen Huang calls open source foundational to national security. Both quotes are doing strategic work. The substance here is the integration and the go-to-market, not a technical breakthrough.

Why it still matters

Strip away the government framing, and the same pattern is spreading through finance, healthcare, manufacturing — anywhere data residency and compliance are design constraints rather than afterthoughts. The reasons it’s newly practical are concrete: open models good enough for production, MoE architectures that cut the compute bill, and an inference tooling layer (NIM, TensorRT-LLM, vLLM) mature enough that air-gapped serving is an engineering project rather than a research one.

“The model becomes a component. The engine becomes the product.”

The takeaway here for developers is that as open models converge on “good enough,” differentiation shifts to everything around them — deployment, routing, governance, evaluation, security, and lifecycle management. The model becomes a component. The engine becomes the product.

Most organizations will keep calling hosted APIs for plenty of work. But for a growing set of them, the interesting question is how much of the stack they’re prepared to own — and whether they’ve counted the cost of owning it.

The post Palantir and Nvidia want to change who owns government AI appeared first on The New Stack.

Workday, the payroll and HR data platform, has been pursuing AI and agents for a while, but while other businesses may allow a little room for error, getting a payroll run in Workday 99% right is not exactly good enough.

Gabe Monroy, Workday’s chief technology officer, tells The New Stack that enterprise AI has to clear this bar before anyone will let it near their HR and finance data.

“There aren’t many systems that are more critical — or less forgiving — than ones that are dealing with people and money,” he says. There’s no tolerance for “well, it works most of the time,” Monroy says in an interview.

At its DevCon developer conference in early June, Workday laid out its plans to clear this correctness bar. The company introduced Agent-Ready Tools, a set of connectors that let agents act across the platform over the Model Context Protocol (MCP), a Developer Agent that lets people build apps and agents on Workday in plain language, and Agent Passport, which tests and verifies agents before they go into production and keeps monitoring them after, with Cisco as the first attestation partner.

Guardrails belong in the inference engine

Monroy spent most of his career in infrastructure and the developer space: at Deis, Microsoft, DigitalOcean, and, most recently, Google. At Google, he focused on building infrastructure for large AI labs to run inference at scale. For someone so focused on infrastructure, coming to Workday might seem a bit like an odd move, but Monroy argues that, at this point, LLM safety is — or at least should be — part of the core infrastructure for enterprises.

“The stakes are higher … in the world of people and money”

“The stakes are higher in the world of Workday and in the world of people and money, and that’s something that I was really excited about tackling at Workday specifically — and I do look at it as a core infrastructure,” he says. “A lot of what I’ve been doing in my recent past has been building infrastructure for large AI labs to do inferencing at scale, and what you pick up pretty quickly is that inferencing is probabilistic.

“[Inferencing] involves prefill and decode, and a whole bunch of really technical machinery in place to stream tokens out to end users, but what is nowhere in that stack today is the concept of native LLM-level enforced guardrails — guardrails that are part of the core inference.”

In his view, making it safe for enterprises to operate inference at scale has to be done at the inference engine layer. Agent gateways and similar add-ons that wrap the model from the outside sit at the wrong layer, he argues.

For a system of record like Workday, the system must strictly enforce guardrails on who the user is, what their budget authority is, and where they sit in the org chart. Those constraints, Monroy says, can be wired deep into the inference rather than checked after the fact. “These are things that we have the opportunity to wire deeply into the inferences at a very, very low level, in a way that produces much safer outcomes,” he says.

Workday’s recent acquisition of Pipedream partially plays into this. With Pipedream, an agent can reach out to third-party systems outside Workday to, for example, pull a policy document from Google Drive, and the platform can then verify that this specific agent has all necessary access rights to do so.

“Bring it to our shop”

All of those agents need to be managed, of course, and while many SaaS companies are currently building out their own agent platforms and agent orchestration services, seemingly all offering the same services, Monroy argues that orchestration should happen close to the data sources.

“If you’re trying to run an agent interaction that is integrating with people and money, that orchestration loop should probably happen closer to Workday, ideally on the Workday,” he says. “I do think there is something to be said for inference engines having proximity to the underlying system, because when it comes to a low-level inference engine and runtime perspective, there are some differentiated things you can do at the inference engine level that are only possible due to proximity to.”

“I do think there is something to be said for inference engines having proximity to the underlying system…”

He likened it to car repair. If somebody hands you a toolbox, you may or may not be able to fix your car’s problem. “Maybe you’ll get the outcome, but I tell you what, if you really want your car repair done right, bring it to our shop. We got the hydraulic lift, we got the trained engineers who know how to do this stuff, and we got the tools — but our tools are the best in the industry.”

For the part of an agent’s orchestration loop that has to run next to HR and finance data, “you really should be running that in our shop.”

For more general-purpose workflows, though, he believes there is room for general-purpose platforms. Not everything needs to run on Workday, after all, and Workday does offer MCP servers that third-party tools can tap into to access its data and tools as well.

To some degree, every system-of-record vendor is making some version of the proximity-and-context case right now. Owning the context for AI agents to work with is a major moat for these companies, after all, even as APIs and MCP servers allow agents to pull in data across vendors.

Meanwhile, there is also a group of other companies positioning themselves as neutral parties benefiting from the fact that these other vendors are now opening their platforms to third-party agents. It seems unlikely that, in the long run, enterprises will want to manage multiple agent platforms, but at this point, it is still far from clear in which direction this pendulum will swing.

Happy to lose the tooling fight

If proximity is where Workday thinks it can win, there is one area where the company doesn’t want to compete: developer tools.

“We’re not going to differentiate Workday on building better tools for developers,” Monroy says. “We’re going to differentiate on safety, on trust, on the inference engine.” It’s telling that the demos at DevCon ran on Claude Code, Cursor, OpenAI’s Codex, and Google’s Antigravity rather than on anything Workday-branded. “On the tooling front, I’m happy to have Claude Code and Codex and others,” he says.

It’s no secret that developer loyalty to AI tools doesn’t really exist at this point. Developers definitely don’t want a one-off tool from a single vendor, and Monroy notes that he isn’t going to ask them to adopt one. Instead, Workday wants to own skills, which he calls “the underlying lingua franca that we’re using to converge across the system.”

Workday is happy to be the system of record for its vertical and to let others fight over the tools.

The post “Bring it to our shop”: Workday’s pitch for keeping AI agents close to your most valuable data appeared first on The New Stack.