Moonshot AI became the latest AI company to discover that launching a popular model is only half the battle. Less than two days after releasing Kimi K3, the company stopped accepting new subscribers after demand exhausted its available GPU capacity. Existing users will keep access while Moonshot expands its infrastructure and reopens subscriptions in batches.

Inference demand outpaces supply

The incident emphasizes how demand is outpacing available infrastructure. As AI models take on longer, more coding and agentic workloads, companies are finding they need more inference capacity than they anticipated.

“Kimi K3 has received far more love than we expected,” the official Moonshot account writes on X. “Over the past 48 hours, demand has pushed close to the limits of our current capacity. We’re adding capacity as fast as we can and will reopen new subscription spots in batches.”

Kimi K3 has received far more love than we expected, and our GPUs are feeling it.

Over the past 48 hours, demand has pushed close to the limits of our current capacity. To protect the experience of existing subscribers, we're temporarily pausing new subscriptions and…

— Kimi.ai (@Kimi_Moonshot) July 19, 2026

“We’re adding capacity as fast as we can and will reopen new subscription spots in batches.”

For infrastructure engineers and developers, the resulting capacity crunch is a bold indication of why companies from OpenAI to Anthropic to Moonshot are rationing access instead of selling unlimited usage.

Open weights, closed capacity

At 2.8 trillion parameters, Kimi K3 is one of the largest open-weight models slated for release — Moonshot has scheduled the public weight drop for July 27. In Arena.ai’s Frontend Code Arena, K3 topped both OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5. On the broader Artificial Analysis Intelligence Index, it trails both, scoring 57 to Fable 5’s 60 and Sol’s 59. That doesn’t make it any easier to run.

Open weights let anyone deploy the model, but whoever hosts it still has to pay the inference bill. Coding activities tend to tie up GPU resources far longer than a typical chatbot interaction, making it harder to keep latency low as more developers pile on.

“Agent tasks are not one-off question answering; they continuously generate, read, and process tokens during ongoing tasks,” wrote Citigroup semiconductor analyst Peter Lee in a research note.

Lee argued that as developers build longer agentic workflows, lower inference costs are quickly “re-converted into higher total resource consumption,” shifting the bottleneck from compute to server memory.

Moonshot’s subscription pause is a sign that keeping enough inference capacity online once developers start using it at scale might be just as hard as building the model.

“Agent tasks are not one-off question answering; they continuously generate, read, and process tokens during ongoing tasks.”

China’s chip constraints compound crunch

For a company like Moonshot, this general industry bottleneck is compounded by regional infrastructure realities. Unlike traditional software companies, AI developers typically rent much of this computing power from cloud providers such as Alibaba Cloud, Tencent Cloud, and Huawei Cloud rather than owning extensive data-center infrastructure themselves.

The capacity crunch illustrates the mounting challenge facing Chinese AI developers as US export controls continue to restrict access to leading chip provider Nvidia’s most advanced AI chips. As a result, companies such as Moonshot depend on a combination of older chips and domestically produced alternatives. These constraints have forced Chinese developers to concentrate strongly on software tuning and more efficient use of computing resources to narrow the performance gap with US rivals.

Token economics under pressure

The scramble for computing power has fueled a data-center construction boom across China. Alibaba has committed more than $53 billion to AI and cloud infrastructure over three years, while ByteDance is reportedly considering spending as much as $70 billion this year on AI data centers and related infrastructure.

AI companies typically charge customers based on the number of tokens, or units of text, processed by a model, making token prices a key measure of operating costs.

According to Bernstein Research, Moonshot charges $3 per million input tokens and $15 per million output tokens for Kimi K3. This makes it about 40% cheaper than Anthropic’s Opus 4.8 and roughly 70% cheaper than Claude Fable 5.

“A world where there are only [two to three] dominant frontier labs with 90 percent inference margins is net negative for every other layer while being awesome for those [two to three] labs,” wrote Atreides Management founder Gavin Baker on X. Baker argued that models like Kimi K3, Grok 4.5, and Muse 1.1 could shift value away from the model layer and toward chipmakers, cloud providers, and the software companies building the infrastructure that serves AI models.

For developers, Moonshot’s subscription freeze serves as an architectural warning. The era of assuming infinite, cheap API access is ending.

“A world where there are only [two to three] dominant frontier labs with 90 percent inference margins is net negative for every other layer while being awesome for those [two to three] labs.”

The post Moonshot launched Kimi K3. Then demand shut down subscriptions in 48 hours. appeared first on The New Stack.

Microsoft and Mistral are deepening their partnership with a multibillion-dollar agreement focused on enterprise AI infrastructure. The goal is to give organizations more flexibility over where they run frontier models and how those deployments are managed, particularly in regions with tight data residency and sovereignty requirements.

Under the deal, announced Tuesday, Microsoft will use Mistral’s expanding European compute infrastructure to increase regional capacity. Mistral, meanwhile, plans to deploy thousands of NVIDIA Vera Rubin GPUs to power everything from model training to multi-agent workloads. 

For engineering teams in regulated industries, the announcement emphasizes where enterprise AI is heading. Many organizations no longer want the public cloud to be the only place they can run AI models. They’re asking for deployments that fit into the infrastructure they already have, including on-premises and air-gapped environments.

Sovereign compute meets agentic AI 

At the compute layer, the agreement gives Microsoft access to European-operated infrastructure outside its traditional first-party data centers and leased facilities. The move builds on Microsoft’s European Digital Commitments, announced in 2025, which focus on keeping customer data in Europe and helping organizations meet regional regulatory requirements.

Crucially, Mistral will power this infrastructure with NVIDIA’s next-generation Vera Rubin rack-scale platform. Because sovereign environments support multi-step agentic workflows, this hardware pairing is essential. NVIDIA claims the platform delivers up to 10x the agent throughput at scale compared to its Grace Blackwell generation.

“Europe should have access to the world’s most capable AI without compromising control over their data, operations or digital future.”

Even though exact financial terms, capacity allocations, and rollout timelines have not been made public, the agreement positions Mistral as both an AI model supplier and an independent European compute vendor for Microsoft’s enterprise ecosystem. For European companies wary of the US CLOUD Act, the appeal is obvious — though whether a US-headquartered provider can fully place data beyond its reach remains legally contested.

“Europe should have access to the world’s most capable AI without compromising control over their data, operations or digital future,” said Microsoft Vice Chair and President Brad Smith in the announcement. 

The model layer

At the model and API tier, Microsoft announced that Mistral Medium 3.5 and Mistral OCR 4 are now live in Microsoft Foundry, with Medium 3.5 also integrated into Microsoft Copilot Studio.

Among the models coming to Azure AI Foundry is Mistral Medium 3.5, an open-weight model with 128 billion parameters and a 256,000-token context window, which could indicate it’s better suited for applications that need to work across large documents or extended conversations.

Microsoft is also adding Mistral OCR 4, which is designed for document-heavy AI workflows. The model can process documents in 170 languages and preserve information about page layout, including bounding boxes, document structure, and confidence levels.

The model can process documents in 170 languages and preserve information about page layout, including bounding boxes, document structure, and confidence levels

Air-gapped deployments go mainstream 

For developers, the primary technical win is architectural symmetry. Engineers can build, test, and fine-tune applications inside Microsoft Foundry and migrate those workloads across public Azure, Azure Local, or sovereign infrastructure operated by Mistral without having to refactor underlying workflows.

This hybrid portability targets industries bound by strict compliance regulations. The deployment options also include fully air-gapped environments, allowing organizations to run AI workloads on networks that remain completely isolated from the public internet.

Deployment flexibility drives decisions

The partnership indicates that deployment flexibility is becoming part of the buying decision. Many organizations want the freedom to run AI in the cloud, on-premises, or in regional infrastructure without being locked into a single approach. And that’s where Microsoft’s expanded relationship with Mistral fits so well.

The companies are betting that giving customers more deployment options will weigh just as much as offering competitive models. If the strategy works, Microsoft will be providing the control plane for how the world’s most regulated industries run it. 

If the strategy works, Microsoft will be providing the control plane for how the world’s most regulated industries run it. 

The post Microsoft is building an AI stack it doesn’t fully own — on purpose appeared first on The New Stack.

The fast pace of AI research means organizations now have a wide range of models to choose from that can power AI agents to solve real business problems. But choosing a model doesn’t guarantee you effective agents or even good performance. For that, you need to run your agents in an environment that provides them with tools, state, security, and scale, with fast startup times and good integration with your existing business systems.

Picking the right agent runtime environment is like picking an enterprise app server but for AI systems — and agents have very different needs from traditional applications.

Gartner predicts more than 40% of agentic projects will be canceled by 2027; not because models aren’t powerful enough to be useful but because of unclear business value, inadequate risk controls and ballooning costs.

Gartner predicts more than 40% of agentic projects will be canceled by 2027; not because models aren’t powerful enough to be useful but because of unclear business value, inadequate risk controls and ballooning costs.  Agents that deliver in proof-of-concept systems will fail in production if the runtime stack powering them can’t keep up and keep them under control. 

Agentic compute is different

It’s easy to think of an AI agent as just another microservice that takes unstructured input, runs APIs or queries, and returns messages. But infrastructure designed for traditional enterprise applications with predictable business logic or even cloud-native stateless workloads doesn’t fit agents with their bursty, long-running, stateful, non-deterministic, code-writing, tool-invoking behaviors that might be triggered by a system event or an email — not just a chat session.

Model inference needs GPUs for speed, but agents also need reliable, durable compute that supports stateful sessions for long-running processes, along with strong security and real-time visibility.

You still need to think about familiar issues like hosting, scaling, identity, and security, but all that is complicated by the unpredictable, multi-stage workflow of the agent reasoning loop.

An agent pulls in input from multiple sources, reasons over its context about the execution plan for accomplishing the goal, calls other tools or writes its own code, iterates over the results of those calls, and maybe builds on them or switches to another approach that requires another reasoning loop and eventually delivers output. That might be updating a system or sending an email rather than just displaying an answer.

Model performance is only one part of making that useful. Architecting a successful agent system that can run at enterprise scale requires considering the agent runtime, the application layer that agents call, and the tools, APIs, and MCP servers they consume.

---

More on Microsoft Azure

• How to secure Kubernetes in the age of AI workloads
• In the AI age, Java is more relevant than ever

---

You have to be able to integrate with business logic and existing systems, manage the usual quotas, rate limits and SLAs for APIs so agents don’t overload them — and you have to do all that while keeping up with AI developments that are moving too fast for you to build the infrastructure primitives you need to rely on from scratch every time.

Requirements of modern agentic infrastructure

Instead, you should look for flexible infrastructure that fits the way the agent works. As with any technology, if you build on an existing platform like Azure Container Apps, you can save effort in areas where your business can differentiate. And while AI agents have flaws (from hallucinations to high token costs) that aren’t fully solved, you can pick an agent runtime environment that makes it easier to get useful results despite them.

Agentic compute needs fast startup and resume. Whether it’s a human typing into a chat prompt or system events automatically launching multiple agents, agent infrastructure needs to spin up quickly. If it takes a few seconds to spin up a container, the reasoning loop can’t start till that’s done.

It also needs to scale up and down responsively, without costing you anything when it’s not running.

Because agent workloads are long-running and event-driven, agents need to start up, do some work, go idle for hours or even days, and then resume instantly with their memory, context, caches, connections, identity, and security intact. That means persisting and restoring state so long-running agents don’t have to pay the same startup tax over and over again. Whether the last stage of the agent’s reasoning loop was successful or a failure, it has to take another approach; you don’t want it to do the same work again.

Rather than building your own custom microVM stack, the new Azure Container Apps Sandboxes provide a temporary, secure, and stateful compute environment that spins up, executes code, snapshots disk and memory, then automatically idles and resumes just as fast. This is the stateful equivalent of Dynamic Sessions, with sub-second startup from pre-warmed pools; you can burst out to hundreds (and eventually thousands) of concurrent sandboxes when you need them, then scale back down to zero.

An agent runtime needs to be secure by default because agents are only useful when they take action — and by definition, they’re likely to do unexpected things, write and execute their own untrusted code, and keep trying to achieve their goal (sometimes even when there’s a policy that should stop certain behaviors).

To give agents secure, auditable access to the resources they need, run them and the code they generate in a sandbox, rather than on a developer laptop with production credentials and admin rights. Identity, access control, and execution boundaries have to be enforced at the runtime layer, not bolted on in a hidden prompt.

ACA Sandboxes have egress and access policies, so you can control outbound calls and limit what URLs they can access. If you can’t use managed identity for all the services an agent needs to connect to you, you can protect secrets by injecting API keys through an external egress gateway instead of hard-coding them. Because the gateway is external, outbound call decisions are controlled by policy outside the sandbox, not by agent code that will relentlessly try any method to get through.

If agents are useful, you’re going to run a lot of them, often simultaneously. AI agents need strong isolation for each task, so every untrusted code execution happens in its own sandbox and no data leaks between tasks.

You can manage them as groups, but each ACA Sandbox has its own secure boundary, so details from one customer support agent won’t end up in a chat with a different customer. If the untrusted code does turn out to be problematic, isolated sandboxes at least contain the blast radius.

Manage the tools agents call

You don’t want to rely on fragile glue code or ad hoc orchestration that you have to rewrite any time systems change for the calls agents make to tools, APIs, cloud services, and other workflows. You also don’t want to rely on manual cleanup of resources no longer needed. A runtime with built-in agent tool execution makes agents more robust. It takes the drudgery out of connecting the agent environment to your other systems, enabling them to interact securely, in isolation, and at scale.

“It’s not just about where the agent is running and what the capabilities of the agent are, but also what actions the agent can take.”

“It’s not just about where the agent is running and what the capabilities of the agent are, but also what actions the agent can take,” points out Vyom Nagrani, who runs the team of PMs responsible for both Azure Container Apps and the Azure SRE Agent that’s built on ACA Sandboxes. Sandboxes have access to a connector framework with over 1,400 enterprise-grade connectors, enabling them to take actions not only on Azure and Microsoft services but also on third-party tools.

“The runtime provides those connectors, and it provides a managed way of authenticating against all of these third-party systems, so now you can build an agent which can talk to many, many, many different systems. It’s not boxed into one authentication boundary,” Nagrani tells The New Stack.

MCP servers are a built-in capability of the connector framework. “You can take any of these connectors; you can take any REST API and expose it as an MCP server, which then the agent can consume.” Or if you want to build a custom MCP server, you can host that in a sandbox too. ACA Sandboxes can be both where agents run and, if that’s appropriate, where the tools they use are hosted.

If you’re building a user interface to wrap your agents, the application layer that makes calls to the agents can run in Azure Container Apps Express, a new service now in public preview, Nagrani says. 

“It’s a simplified app hosting stack for the app layer that responds to HTTP traffic and serves web traffic: that’s where the human interactivity comes in.”

Putting it all together

ACA Sandboxes offers an agent runtime environment that answers the key questions architects need to consider: where agents and the ephemeral compute they need access to run; where the application layer that calls agents runs; and how agents get access to all the tools, APIs, services, MCP servers, and existing business logic they need to orchestrate.

Whether you’re a platform engineer, a software vendor or a startup building a new AI platform, treating agent runtimes as the new application server and MCP servers as the new APIs requires agent infrastructure that supports agent workloads effectively, allowing governance to shift left into the runtime layer where it can scale with the ever-increasing numbers of agents. 

How real platforms build on agent runtimes

Azure Container Apps is already a strong platform for running agents. Auger, a startup launched by the former CEO of Amazon’s global consumer business to help mid-size enterprises get their complex supply chains out of Excel spreadsheets, used it to build a multi-agent system that can give real-time answers about shipments and forecasts in a world where mines or critical shipping lanes might be closed at any time by war or weather. 

Backend agents pull the unstructured data with all the details to answer those questions from different siloes, building ETL pipelines and creating an ontology of the supply chain ecosystem for each customer that includes functions and actions — all of which need to be audited and reversible. Frontend agents use that ontology to answer questions such as, “What happens if I build a new warehouse here or switch to a supplier in this country?”

ACA Sandboxes are built on Azure Container Apps, giving organizations an easier way to create agent services with strong isolation, dynamic scaling, fast startup, persistent state, and broad tool and service integrations.

That’s just what the Foundry team was looking for when they started work on the Microsoft Foundry Agent Service managed agent runtime. ACA Sandboxes gave them a platform that delivers fast start/resume, built-in tool execution (including for untrusted code), persistent state for long-running agents, strong per-agent isolation, and secure-by-default operations. Agent identity, “on‑behalf‑of” authentication to existing business services, and strict isolation are critical capabilities they didn’t have to build themselves.

“It’s the same enterprise-grade infrastructure behind Microsoft’s own agentic products, now available for all Azure customers to build on.”

The Foundry Agent Service adds a layer of visibility and observability into agent actions and conversations, allowing customers to monitor agent performance and see where the system is doing well and where it needs improvement.

“Azure Container Apps Sandboxes package the hard parts of an agent runtime into a first-class Azure resource — sub-second start and resume, built-in execution of tools and untrusted code, and snapshot-based state that lets long-running agents pick up exactly where they left off. Every agent gets its own hardware-isolated environment with secure-by-default operations, so builders can focus on what their agents do, not on the plumbing underneath,” Nagrani points out.

“It’s the same enterprise-grade infrastructure behind Microsoft’s own agentic products, now available for all Azure customers to build on.”

The post The rise of the agent runtime: The compute platform behind production agents appeared first on The New Stack.

Public AI assistants have become so commonplace that software vendors are increasingly adding AI search, conversational experiences, and AI agents to their own applications. From eCommerce and customer support to enterprise software, AI is rapidly becoming the primary interface to many applications.

Companies that build products around proprietary information are particularly well positioned to benefit from this shift. Whether they provide financial intelligence, market intelligence, legal research, scientific publishing, or business information, their products help professionals make better decisions by transforming trusted information into actionable insight. 

AI allows these organizations to deliver that expertise through entirely new user experiences. Increasingly, they compete not only on the quality of their proprietary information, but on how intelligently they retrieve, understand, and transform it into customer value.

“Increasingly, they compete not only on the quality of their proprietary information, but on how intelligently they retrieve, understand, and transform it into customer value.”

A new competitive battleground is emerging. As AI becomes the primary interface to proprietary knowledge, the ability to retrieve, verify, rank, and assemble information is becoming almost as important as the proprietary information itself. 

Much of the industry’s attention has focused on increasingly capable language models, but those models are only as effective as the context they receive. Designing retrieval workflows that consistently deliver trusted, relevant, and up-to-date information is rapidly becoming one of the defining engineering challenges for AI-native applications.

Retrieval engineering: optimizing the workflow

For decades, search engineering has focused on helping people find the right information. Whether searching a website, a legal database, or a financial research platform, the challenge was to retrieve the most relevant results while balancing competing priorities such as relevance, latency, scalability, and cost. The search system’s job was to retrieve relevant information. The human’s job was to evaluate it.

AI fundamentally changes that role.

Instead of retrieving information for people to evaluate, retrieval systems increasingly assemble the context that large language models and AI agents use to investigate, reason, and act. Every retrieval decision now becomes part of an automated workflow in which relevance, freshness, latency, and trust directly influence the final answer.

“Prompt engineering influences how a language model reasons. Retrieval Engineering determines what it has to reason about.”

This shifts the engineering challenge away from individual technologies and towards the retrieval workflow itself. The goal is no longer simply finding relevant documents, but orchestrating retrieval, ranking, filtering, inference, and real-time updates so they work together efficiently. We believe this emerging discipline deserves its own name: Retrieval Engineering. Prompt engineering influences how a language model reasons. Retrieval Engineering determines what it has to reason about. Both matter, but as AI applications become increasingly autonomous, the quality of retrieval increasingly determines the quality of the outcome.

As AI applications evolve from conversational assistants to deep research systems and autonomous agents, optimizing workflows rather than individual components becomes increasingly important. A single user request may trigger dozens—or even hundreds—of retrieval operations before a response is generated.

The challenge isn’t vector search

Vector databases solved an important problem by making semantic retrieval practical at scale. But semantic retrieval is only one stage of a much larger workflow.

Production AI applications increasingly combine vector similarity with keyword search, structured filtering, business rules, personalization, machine-learned ranking, and real-time inference to assemble the context that language models depend on. The engineering challenge is no longer selecting the best retrieval technology—it is orchestrating increasingly sophisticated retrieval workflows that remain accurate, responsive, and cost-effective.

Many organizations address this by combining specialist technologies. A vector database provides semantic retrieval. A search engine handles lexical matching. Additional services provide reranking, personalization, and inference. This works well initially, but every additional component introduces another network hop, another operational dependency, and another source of latency. The problem is no longer vector search. It is engineering an efficient retrieval architecture.

From components to platforms

This shift is changing how retrieval infrastructure is designed. Instead of optimizing individual components in isolation, engineering teams increasingly need to optimize the retrieval workflow as a complete system—balancing retrieval quality, latency, freshness, scalability, and infrastructure cost.

“The problem is no longer vector search. It is engineering an efficient retrieval architecture.”

That is why AI Search Platforms are emerging. Rather than stitching together retrieval, ranking, inference, and serving from multiple independent services, they execute the workflow within a single distributed architecture. The optimization problem changes from integrating components to engineering the workflow itself.

AI has transformed the user interface. It is now transforming the retrieval infrastructure behind it. For organizations building applications around proprietary knowledge, the next competitive advantage will not come solely from larger language models or better embeddings. It will come from building retrieval workflows that consistently deliver trusted, relevant, and timely context at scale.

Retrieval Engineering is rapidly becoming one of the disciplines defining the next generation of AI-native applications.

If you’re interested in exploring these ideas in more depth—including Retrieval Engineering, AI Search Platforms, and the architectural patterns behind AI-native information platforms—we cover them in our ebook, Building AI-Native Information Platforms.

The post Is retrieval engineering becoming AI’s next bottleneck? appeared first on The New Stack.