There’s a pattern I’ve watched repeat for two years. A team builds an agent, hits reliability problems, upgrades the model, sees marginal improvement, and hits the same reliability problems in a slightly different form. The diagnosis is always the same: the model wasn’t smart enough. The fix is always to try a new, smarter model. The result is always the same: still broken. 

This isn’t a model problem. It never was.

Andrej Karpathy figured this out months ago, and in a post on X, he noted a shift in how he was spending his AI compute: “a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge.” He wasn’t running a smarter model. He was building better infrastructure:  raw sources indexed into a directory, an LLM incrementally compiling them into a structured wiki with summaries, backlinks, and concept articles, tools handed to the agent as CLIs, outputs filed back into the base to enhance future queries. The model was constant. The infrastructure around it was the variable.

“This isn’t a model problem. It never was.”

I keep coming back to the framing. The model runs on context. The quality of execution depends on the quality of the context it receives, the precision of the actions it’s permitted to take, and the feedback loops that let the system learn from what it got wrong. None of that lives in the model. It lives in the infrastructure underneath. And it turns out, most teams haven’t built it.

The missing compile setup

Karpathy’s setup is built around one extra step. Raw data comes in. An LLM “compiles” it into a structured, queryable form. Then, agents operate over that compiled version with tools. That compilation step is where the work happens, and it’s what turns a mess of internal data into something an agent can reason over instead of guessing at.

Too many production agent systems still skip this step. They wire the model directly to raw data — databases, APIs, document stores — and expect it to compile at query time, within the context window, under latency pressure. What comes back is pattern-matched guesswork over a window full of noise.

The teams that got this right built the compilation step explicitly. Not a generic knowledge base: a structured representation of how their specific organization works. How things are named internally. What actual decision paths look like. What previous similar runs produced, and what was decided. The organizational equivalent of Karpathy’s wiki — built from operations, not from documentation.

This is harder to build than switching models and harder to maintain. It’s also the difference between an agent that operates in your organization’s actual reality and one that confidently operates in a hallucinated version of it.

The tool retrieval problem

Karpathy notes that once his wiki reached meaningful scale — around 100 articles and 400,000 words the system remained usable because the LLM maintained indexes and summaries, and because he began adding tools, including a small search engine exposed to the LLM over a CLI. The point is not that context disappears. It is that retrieval, indexing, and tool interfaces become part of the system you have to engineer.

Production agent systems hit this wall hard. Consider a mid-size engineering org: GitHub, Jira, Confluence, a handful of cloud providers, monitoring and alerting tools, a CI/CD platform, internal deployment tooling. Each integration is a family of tools with its own schema, naming conventions, and expected invocation patterns. Dropping all of that into a single context window is slow and expensive, and it produces poor tool selection. The model pattern-matches across noise.

Standard vector retrieval compounds the problem. It matches the semantic similarity between a user query and stored tool descriptions. It works when vocabulary aligns. It breaks when it doesn’t: a developer asks “why did the deploy fail,” the right tool is something like get_pipeline_run_logs, and the vector match between those two phrases is poor. The agent selects the plausible tool instead of the correct one.

The fix is to guess the answer first. Given the query, what would a working tool call look like? The system writes that hypothetical call, then matches against it instead of the raw question. “Why did the deploy fail” and “fetch pipeline logs” don’t look alike as text. But once you’re matching the shape of the right action instead of the words in the request, they line up.

This is a translation layer, from intent to action, and that translation is where most agent failures start. It’s an engineering problem, not a model problem. I’ve seen teams run this in production after watching vector retrieval fall apart at scale, and they say the same thing: switching from semantic similarity to hypothetical-invocation matching gave them more reliable tool selection than upgrading the model.

The guardrails gap

The capability-without-constraint failure mode shows up across contexts, and the pattern is consistent: an agent with broad tool access, executing correctly from its own perspective, takes an action nobody intended. Not because it hallucinated. Because the boundary between what it could do and what it should do wasn’t enforced.

Three cases illustrate the shape of the problem.

Start with GTG-1002, still the most detailed public account I’ve seen. In November 2025, Anthropic disclosed that a Chinese state-sponsored group had manipulated Claude Code in a cyber-espionage campaign against roughly 30 organizations, infiltrating in a few cases. Anthropic reported that AI did roughly 80-90% of the tactical work, with humans stepping in only at strategic decision points, and that, at peak, the AI was firing off thousands of requests, sometimes several a second. No human team could keep pace. This wasn’t a model-quality failure. It was a failure of execution boundaries. Once the attacker fragmented the work and slipped past the safeguards, the system could take high-risk actions faster than any human could supervise.

Prompt injection is a different vector, same structural failure. Researchers have repeatedly demonstrated agents executing injected instructions from content they retrieved:  a malicious payload in a web page or document that redirects the agent’s next tool call. The model does what the injected instruction says because nothing in the execution layer distinguishes “instruction from user” from “instruction found in retrieved content.” The agent is working correctly. The architecture isn’t.

The third pattern is quieter and more common: over-permissioned agents operating across multiple write-capable systems. An agent with access to a CRM, an email client, and a calendar does something unexpected — updates records, sends a draft, books a meeting — because a workflow reached a branch that wasn’t anticipated and there was no scoped permission model to prevent it. Nobody intended this. Nobody wrote a rule against it. The agent had access, so it acted.

“An agent with access to a CRM, an email client, and a calendar does something unexpected, because a workflow reached a branch that wasn’t anticipated and there was no scoped permission model to prevent it.”

What these have in common is that the model isn’t the failure point. The failure is the absence of an execution layer that defines, per action, what is permitted and enforces it regardless of what the model decides to do.

The architecture that addresses this intercepts every tool call before execution, masks sensitive data before the LLM processes it, blocks specific tool combinations at the execution layer rather than the prompt layer, enforces per-agent rate limits and role-based access, and generates a full audit trail with explicit reasoning for every invocation. Human checkpoints must be designed into high-stakes paths, and not as fallbacks, but as architecture.

We built our execution isolation layer at Mate around exactly this pattern. Every tool call gets validated, scoped, and logged before reaching the integration. The model never has direct write access to a downstream system;  it proposes actions to a layer that decides whether to execute them and at what scope.

Karpathy’s wiki points to the same discipline in miniature: health checks that find inconsistent data, impute missing fields, and suggest new connections. In an enterprise system, that linting layer needs permission boundaries. Some updates can be automatic; others should be routed to proposals, approvals, or review queues. The boundary between what an agent can change and what requires human judgment must be explicit.

What the engineering work actually looks like

The teams producing reliable production agents spend most of their engineering effort on infrastructure. I see four areas coming up again and again. 

The context graph first. Building and maintaining a compiled representation of org knowledge isn’t a one-time task. Schemas change. Systems get renamed. Personnel and processes shift. Teams that do this well treat the context graph as a product with an owner, an update cadence, and health checks, rather than a setup step that runs once at deploy time.

Observability second. Model-agnostic proxy layers are becoming standard: a single layer capturing full traces on every LLM call regardless of provider, enforcing per-tenant cost and rate limits, allowing model swaps without rearchitecting. Tracing for agents means capturing reasoning, not just requests: what the agent considered, which tool it selected and why, what came back, and what it did with the result. Without that, debugging is archaeology.

Continuous evaluation third. Per-agent, per-workflow datasets built from production traces rather than synthetic benchmarks. Two tracks: deterministic checks for things code can verify, like tool call correctness, rate limit compliance, scope violations, and model-as-judge for things it can’t, like reasoning coherence and response quality. When you promote a new prompt version or upgrade a model, you run it against real production data. The question isn’t “does it benchmark higher.” It’s “does it still work in this org’s actual context.”

Configuration management fourth. Prompt versions, model selections, and tool configurations need to be independently releasable and independently rollback-able. Changing which model an agent uses shouldn’t require a code deployment. Rolling back a prompt regression shouldn’t require an incident. The teams that figure this out early ship changes faster and break less. It’s the same ML engineering discipline that matured in recommendation systems five years ago, now applied to agent behavior, a practice most organizations are building from scratch.

The differentiator isn’t reasoning

Karpathy’s shift from manipulating code to manipulating knowledge describes where the hard work actually lives in agent systems. The model executes over context. How that context is structured, retrieved, and scoped determines the outcome. What constrains the model’s actions determines safety. What measures and improves the system determines reliability.

All of that is infrastructure. None of it is solved by a more capable model.

The model is commoditizing faster than most teams realize. The reasoning gap between major providers is narrow and narrowing. The infrastructure gap between teams that have built context plumbing and guardrails and teams that haven’t is wide and widening.

“The reasoning gap between major providers is narrow and narrowing. The infrastructure gap between teams that have built context plumbing and guardrails and teams that haven’t is wide and widening.”

A smarter model won’t help an agent that doesn’t know your organization. It won’t stop a prompt injection in a retrieved document. It won’t scope-limit an over-permissioned workflow. It won’t tell you that your tool retrieval accuracy dropped three weeks ago because someone renamed an integration.

The infrastructure does that. Build that first.

The post The bottleneck for AI agents isn’t the model anymore. It’s the context layer. appeared first on The New Stack.

Platform engineering has won the argument. Some 90% of organizations have adopted at least one internal platform; golden paths are orthodoxy, and environment requests that once took days now close in hours. By the standard the discipline set for itself, that is victory.

Then the most demanding customer the platform has ever had showed up, and it is not a developer. A coding agent that wants to validate its work requests an environment the way a client calls an API: in bursts, concurrently, with a lifetime measured in minutes and an expectation measured in seconds.

A 100-developer organization in which each engineer supervises a few agent sessions per day generates hundreds of environment requests before lunch. Each request needs realistic dependencies, and each is dead weight the moment its validation finishes. That is not a ticket queue. That is traffic.

The most demanding tenant the platform has ever had

The demand is not speculative. GitHub’s Octoverse counted 43.2 million pull requests merged per month, up 23% year over year, with Copilot’s coding agent alone opening more than a million pull requests in its first five months. Every one of those changes needs somewhere realistic to run before it merges.

The tenant mix is shifting underneath those numbers. Stack Overflow’s 2025 survey found that half of professional developers already use AI tools daily, and every daily user is a candidate to operate two, three, or five concurrent agent sessions. Environment demand no longer tracks headcount. It tracks headcount multiplied by agents multiplied by iterations.

“Coding agents turned environment requests into traffic: concurrent, short-lived, and relentless. The platform teams that keep up will be the ones that stop provisioning environments and start serving them.”

Platform teams can see what is coming. The latest State of Platform Engineering report found that 94% of organizations consider AI critical to platform engineering’s future, and its central theme is the shift from cloud-native platforms to AI-native ones.

What changed is not only the volume but also the shape. Human environment demand is diurnal, negotiable, and tolerant of a morning’s delay. Agents retry, fan out, and iterate in tight loops, and demand that the shape already has a name across the platform. The name is traffic.

Duplicate everything, and the cost curve kills you

The duplication model hands every request a full copy of the stack. Price one out: a 40-service system with its databases and queues costs a few dollars an hour per copy, takes tens of minutes to assemble, and sits mostly idle during the brief window of validation it exists to support.

Multiply by concurrency, and the model collapses. Hundreds of requests a day with modest overlap means dozens of full copies running at once, and a bill that scales linearly with agent activity. The latency is wrong by an order of magnitude too, because an agent that iterates in seconds cannot wait tens of minutes for its environment to arrive.

Pre-provisioning a warm pool does not rescue the model; it only moves the waste. Agent demand is bursty, so a pool sized for the peak idles through the trough, and a pool sized for the trough queues at the peak. Paying full-copy prices for capacity you mostly do not use is the definition of the wrong cost curve.

Share everything, and the queue kills you

The shared model runs one staging environment and admits tenants in turn. Queueing theory has described this failure mode since 1961. Little’s law says the number of requests in a system equals the arrival rate multiplied by time in the system, so as arrivals approach the rate the environment can absorb, wait times stop degrading gracefully and start exploding. Agents multiply the number of arrivals by 5-10 while the completion rate remains fixed.

Shared staging also fails on isolation. One broken change contaminates the environment for every tenant behind it, so the line does not merely lengthen; it periodically resets to zero while someone hunts down the offending commit.

Teams respond to the wait the way people always respond to a slow shared resource, by batching. Changes pile into larger deployments, making each trip through the environment count, which raises the blast radius of every failure and lengthens each occupancy. The queue teaches exactly the behavior that makes the queue worse.

Both models sit at the wrong ends of the same curve, paying full cost for full isolation or zero marginal cost for zero isolation. Neither is a point from which you can operate a serving system.

Environments are a serving system now

The mental model that fits this demand curve already exists inside every platform team. It is the one used for compute. A serving system is judged on latency, concurrency, marginal cost per request, and safe multi-tenancy on shared infrastructure, and those are exactly the four requirements agent-driven demand imposes on environments. A serving system is also something its clients invoke directly, through an interface rather than a person, which is the property that matters most once those clients are agents.

Renaming the problem matters because it changes who owns it and how it gets measured. A provisioning workflow is done when the environment exists. A serving system is never done. It has dashboards, capacity plans, and error budgets, and it is expected to absorb demand spikes without a human in the loop.

“The unit of work ceases to be a ticket and becomes a request. The latency target drops from hours to seconds.”

The mindset gap shows up on every operational dimension. The unit of work ceases to be a ticket and becomes a request. The latency target drops from hours to seconds. The success metric shifts from closed tickets to p99 latency at peak concurrency.

Serve the delta, not the whole stack

One architecture meets all four serving requirements by refusing to copy anything that has not changed. Run a single high-fidelity, stable copy of the system, deployed continuously from main. When a validation request arrives, deploy only the services that changed as lightweight, ephemeral environments, and route that request’s traffic through their own versions, while everything else falls through to the shared, stable environment.

Each serving property follows from the delta. Latency lands in seconds because starting one or two services is fast. Marginal cost approaches zero because tenants share the stable environment. Concurrency is bounded by cluster capacity rather than by environment count, and isolation holds because each request sees only its own changed services, not anyone else’s.

Fidelity is not the thing you give up. A full duplicate is faithful, which is exactly why teams build one, and also why it is slow and costly to stand up and prone to drift between refreshes. Sharing one stable copy that is continuously deployed from main gives every validation request the same real, current dependencies without reproducing them per request.

Routing is the implementation detail rather than the point. Service meshes can carry the routing label, sidecar-free approaches can too, and propagating a label through a call chain is a solved problem in most modern stacks. This is the pattern Signadot enables off-the-shelf.

Agents provision their own environments

An environment that arrives in seconds and costs almost nothing is not only fast enough to keep up with agents. It is cheap and fast enough for them to operate. When requesting one is an API call rather than a ticket, provisioning becomes a step within the agent’s own loop: ask for an environment, deploy the change to it, run the checks, read the result, tear it down, and repeat in the next iteration.

Both properties are what make that possible. A workflow measured in minutes and gated on human approval can never fit within a build-test-fix cycle, because the agent would spend its run waiting in a queue it cannot influence. Near-zero marginal cost makes a discarded environment a non-event, and seconds of latency lets validation live inside the loop instead of after it. Once the environment is something an agent requests for itself, the human stops being the rate limiter, and the platform’s serving capacity takes over.

Validation throughput is what ships AI code

Agents made generation cheap and pushed the bottleneck downstream, onto whether a change can be validated as fast as it is written. Validation throughput, not lines generated, now decides how much AI-written code actually ships, and it is a property of your platform rather than any model.

“Validation throughput, not lines generated, now decides how much AI-written code actually ships.”

Treat environments as a serving system, and environment capacity becomes a dimension you plan and budget like compute or continuous integration (CI) runners. This turns agent adoption from a surprise infrastructure bill into a demand curve you can plan against. For a decade, platform engineering built self-service golden paths for people. 

The next job is self-service for developers and agents that can scale with agent-driven velocity, and we built Signadot for exactly that.

The post Platform engineering’s new job: serving environments at agent speed appeared first on The New Stack.

The post This Week’s Awesome Tech Stories From Around the Web (Through July 18) appeared first on SingularityHub.

It has been almost two years since OpenAI released o1, a model that popularized the idea of LLM-based reasoning models. DeepSeek-R1 followed about four months later, together with details of a reinforcement learning with verifiable rewards (RLVR) recipe to train such reasoning models.

Last week, OpenAI released the GPT-5.6 model family. It comes in three sizes, each with roughly five or six reasoning-effort settings.

So yes, reasoning models are here to stay. They have become a standard part of modern model releases.

In the past, I covered the methodology of reasoning models (Understanding Reasoning LLMs) as well as relevant research papers (The State of Reinforcement Learning for LLM Reasoning and The State of LLM Reasoning Model Inference). And I even wrote a whole new 440-page book on how to develop reasoning models, Build A Reasoning Model (From Scratch).

These resources have focused on turning a conventional LLM into a reasoning model. Now, in this article, I want to focus on and explain how to develop a reasoning model that has multiple effort modes, similar to what’s shown in the figure at the beginning of this article.

No worries, this article can be read as a standalone article. However, the aforementioned resources may be interesting and useful.

1. A brief definition of reasoning models

When talking about pretty much any machine learning or AI technique or subfield, the one lesson is that we usually shouldn’t take technical terms “literally”. For example, an (artificial) neural network in machine learning and AI doesn’t literally work like a biological neural network like the human brain.

Similarly, when talking about “reasoning models”, we shouldn’t expect that these models literally reason like us humans. In the context of AI and LLM research, “reasoning model” means a model that outputs an intermediate reasoning trace, which is like an intermediate response that works through a question or task step by step.

It’s probably easiest to explain this by showing an example.

2. A brief overview of training and inference scaling reasoning models

There are essentially two ways to improve (reasoning) task performance: training scaling and inference scaling.

Let’s briefly talk about training first.

2.1 Training reasoning models

In a nutshell, DeepSeek-R1 proposed training an LLM using reinforcement learning with verifiable rewards (RLVR) to turn it into a reasoning model. RLVR is a technique to provide a reward signal (0=incorrect and 1=correct) for verifiable data domains. These verifiable data domains here are math (we can use a symbolic math checker like SymPy or WolframAlpha to check results) and code (we can use a compiler or unit tests, or integrated platforms like LeetCode) to check for correctness.

Notably, the reasoning trace itself was not used for training or updating the model. Although they tried to use this intermediate response information for training, the DeepSeek-R1 paper reported that it wasn’t helpful for the model training, so it was ultimately not used. (Whether and how to incorporate intermediate reasoning traces in the training signal via process reward models is an active area of research.)

2.2 “Aha” moments

Anyway, just training on the output rewards alone, as Figure 7 shows, turned out to be sufficient for the model to learn how to reason through a problem, meaning that it would learn to write intermediate explanations, backtrack, and self-correct itself. These moments when the model realizes that it made a mistake and self-corrects itself are called “Aha” moments.

By the way, while DeepSeek-R1 is inarguably the more popular paper, and the paper that created excitement around reinforcement learning with verifiable rewards and the development of reasoning models, there is another paper, Kimi K1.5, published on exactly the same day on arXiv (22 Jan 2025). Also, the term RLVR was already coined two months earlier in Tülu 3: Pushing Frontiers in Open Language Model Post-Training.

​

One reason why the DeepSeek R1 is ultimately the more popular paper is that it demonstrated that reasoning behavior can be achieved with pure reinforcement learning (RL).

For instance, Tülu 3 and Kimi K1.5 applied reinforcement learning on top of a supervised fine-tuned (SFT) model. The DeepSeek-R1 model was also trained from an SFT checkpoint of the DeepSeek-V3 base model, and it included a DeepSeek-R1-Zero variant trained with pure RLVR. R1 Zero is a weaker model than R1, but it showed that RLVR is sufficient for teaching the model to generate and use reasoning traces.

​While R1-Zero was more of a proof-of-concept model, note that the full DeepSeek-R1 reasoning model training pipeline is usually multi-stage and a bit more complicated, as mentioned above.

​

By the way, most of today’s LLMs are effectively reasoning models, meaning they have been trained in a similar fashion to DeepSeek-R1 using a form of RLVR.

2.3 Inference scaling in a nutshell

Next to improving reasoning behavior through training, another lever for improving model performance is inference compute scaling. In short, this means that we are spending more compute after training the model, during usage, to get better answers.

This is a whole topic by itself, and you could read through my The State of LLM Reasoning Model Inference for a more detailed rundown:

I will try to summarize what’s most essential to mention as background info below.

First, training a model with RLVR is already implicitly leading to a form of inference scaling, since reasoning models usually output more tokens during inference compared to conventional LLMs, and that means we are spending more compute during inference.

Second, we can further adjust this output length via reasoning effort levels, but more on that later.

​Third, there are many additional inference scaling techniques. A popular one is self-consistency, which is often implemented as a form of majority voting where the model is queried multiple times, and the final answer is selected via majority vote.

This can be applied to conventional LLMs as well as reasoning models. Also, this method can be used on demand and in addition to reasoning training. A good example of that is DeepSeekMath-V2, where the researchers applied extreme inference-scaling on top of a reasoning model (specialized for math) to achieve state-of-the-art performance on challenging math olympiad-type problems.

But again, I will refer to my other article, The State of LLM Reasoning Model Inference for an overview of other techniques:

3. Think tokens

You may have seen the <think></think> tokens in the earlier “Aha moments” figure. I also included the corresponding figure below so you don’t have to scroll all the way up.

​

These <think> and </think> tags are cosmetic with respect to reasoning ability. They do not make the model reason, and they are not required to achieve good reasoning performance. One could train the same model without these delimiters and likely reach similar benchmark performance.

The purpose of these <think> tags or tokens is mainly to mark where the reasoning trace begins and ends so that the training pipeline or user interface can separate it from the final answer and optionally hide it from the user. (UIs like ChatGPT or Codex usually do this.)

The point here is that the <think> tokens are not giving the model the ability to “think” or reason or reason better. One could train the same models without such <think> tokens and reach similar benchmark performance.

There is also nothing special about the literal strings <think> and </think>. Another pair of delimiters could serve the same purpose.

By the way, the way this is implemented is typically by adding a formatting reward during the RLVR stage. So instead of just rewarding the model based on answer correctness, one would provide additional reward for the use of <think> tokens, which in turn encourages the model to use those.

In DeepSeek-R1, for example, the overall reward was calculated as

R_total = R_accuracy + R_format

where the format reward was a simple rule-based check that encouraged the model to place its reasoning inside:

<think>

reasoning trace

</think>.

4. Reasoning mode on and off switches

The first generation of reasoning models was dedicated reasoning models. With that, I mean that there was a DeepSeek-V3 base model and a separate DeepSeek-R1 reasoning model.

No matter what the prompt is, R1 generally outputs very verbose responses using lots of tokens, even for simple prompts. It also lacks a built-in option to turn off the reasoning mode.

Later models, like Qwen3 and others, experimented with hybrid approaches, where the same model can behave like a regular instruction fine-tuned model or a reasoning model on demand.

Note: Some model developers call this “thinking mode,” while others call it “reasoning mode.” Both terms refer to the same behavior.

In Qwen3, this is handled via the tokenizer using enable_thinking=True or enable_thinking=False. Under the hood, setting enable_thinking=False essentially adds an empty <think></think> section to the beginning of the assistant response to turn off Qwen3’s reasoning (”thinking”) mode.

How is this implemented during training, such that the model supports this toggle during inference time, as shown in the figure above?

In short, as explained in the Qwen3 technical report, this on/off behavior is introduced primarily through supervised fine-tuning (SFT) and then reinforced during general RL in their largest flagship models.

For instance, after the initial reasoning model is trained via long-chain-of-thought SFT and reasoning RL, they add a “Thinking Mode Fusion” stage. During this additional SFT stage, the model sees both thinking and non-thinking examples:

• /think: <think>{reasoning}</think>{answer}

• /no_think: <think></think>{answer}

Thinking is the default behavior, so /think can also be omitted. The subsequent general RL stage further reinforces this mode and format following.

These /think and /no_think flags are a “soft” switch. However, the enable_thinking=False setting mentioned earlier, which force-adds the empty <think></think> in the False case, acts then as a “hard” switch.

In other words, the tokenizer does not add /no_think to the query. It directly fills in the empty <think></think> section at the beginning of the assistant response. The model only sees the resulting tokens and continues directly with the answer.

Anyway, this on-and-off toggle is essentially a simplified version of the reasoning effort levels in GPT-5.6 and others, which I cover in the next section.

5. How “reasoning effort” settings work

In this section, I want to provide a brief overview of how the different reasoning effort toggles may be implemented, which have been introduced in models like GPT 5 and are present in pretty much any flagship model today.

Concretely, at the beginning of this article, I showed a figure from the Codex GPT 5.6 interface that lets users select multiple reasoning “effort” settings.

The following subsection will illustrate how these settings may be implemented. Then, in the next section, I will go over some of the more interesting research papers related to this topic.

5.1 Reasoning effort and response length and quality

Unfortunately, the implementation details of their effort settings are not shared by OpenAI, but there is some evidence out there that can be used for educated guesses.

For instance, via their open-source gpt-oss models from last year (I wrote about them in From GPT-2 to gpt-oss: Analyzing the Architectural Advances), we know that OpenAI allows us to toggle the reasoning effort setting via the system prompt (”Reasoning effort: low/medium/high”) that is prepended to each prompt.

As expected, the reasoning effort directly affects the response length and accuracy, as shown below.

Presumably, their GPT 5 models, including the recent GPT 5.6 models, use a similar approach.

By the way, note how different effort settings scale the response length in the figure above. The effort level seems directly correlated to token usage, which in turn seems correlated to accuracy. It might be possible to come up with effort settings beyond the “high” one, but I assume performance would saturate at some point. This saturation can be seen more clearly for the GPT 5.6 Sol model, which also shows that increasing reasoning budgets can become uneconomical at some point.

Another good, very recent data point that shows the relationship between reasoning effort, token usage, and benchmark performance is this week’s new open-weight Inkling release by Thinking Machine Labs.

As discussed in this section, during inference, the reasoning effort level can simply be controlled via a system prompt. (The ChatGPT UI presumably simply maps the menu choice to a system prompt.) However, this would not work for an arbitrary model and requires certain modifications to the training pipeline, which will be discussed next.

5.2 Possible effort level implementations

While the training details are not public, neither for GPT 5.6 nor the open-source gpt-oss models, typically, the reasoning effort label is included in prompts during post-training.

There are typically two ways to implement this.

First, we can implement it as part of the RLVR process and apply a different length penalty when different system prompts are used. For example, a high length penalty when “Reasoning effort: low” and a mild or no penalty when “Reasoning effort: high”.

Second, we can fine-tune the model after RLVR to follow different effort instructions via supervised fine-tuning (SFT).

For instance, after the core RLVR stage and during SFT, the prompts in the training dataset are paired with target responses that exhibit the desired amount of reasoning. (The targets may be written by humans, generated by another model, or generated and then filtered.)

During this SFT stage, the model learns the association between the effort label and the target reasoning length directly from the training examples. An RL-based implementation would instead place the effort labels and budget-aware reward inside the RLVR stage. The two approaches could also be combined, which I suspect was done for both gpt-oss and GPT 5.6 (note that effort settings in GPT 5.6 are likely just changing the system prompt for a given user query).

5.3 Inkling case study

The just-released Inkling technical report gives a small but somewhat concrete example of effort-level training.

During large-scale RL, they did two things for each sample:

1. Specified the desired effort level in the system message.

2. Adjusted the cost assigned to each generated token.

Conceptually, the reward likely looked something like this:

Here, e is the requested effort level and λ(e) controls the token penalty.

• Low effort uses a larger per-token cost, encouraging shorter reasoning traces.

• High effort uses a smaller per-token cost, allowing the model to spend more tokens.

Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly. The difference between Inkling and models such as gpt-oss and GPT-5.6 is that the effort label is a continuous number between 0 and 1 instead of ordinal labels such as low, medium, and high.

This places Inkling’s effort-level conditioning primarily in the Reasoning RL stage, not only in the later SFT stage.

They do not disclose the exact reward formula, token-cost coefficients, or whether effort conditioning was also included in SFT, though.

5.4 A short note about inference scaling versus training scaling

Before moving on to the reasoning effort papers, I want to briefly connect this section back to the earlier “2.3 Inference scaling in a nutshell” section.

Earlier, I separated scaling into training compute scaling and inference-time scaling. The GPT-5.6 interface provides a nice way to illustrate the difference, as shown below.

On the left, selecting Luna, Terra, or Sol changes the model itself. As a rough analogy, this corresponds to training compute scaling. These are separate trained models. At a fixed training recipe and dataset size, a larger model requires more training compute. It also generally requires more compute per generated token.

On the right, we keep the model fixed and only change the reasoning effort. This is inference-time scaling. The model weights stay the same, but the model is allowed to spend fewer or more tokens working on the answer.

One small terminology caveat is that selecting a different model from the menu is not training scaling at that moment. The training has already happened. It is better to think of the model menu as selecting among models that were produced at different training scales.

The Artificial Analysis results below show how these two axes interact in practice.

Each blue curve corresponds to one model, Luna, Terra, or Sol. Moving along a curve by increasing the reasoning effort is inference scaling. Moving from one model curve to another corresponds to model scaling, which I use here as a practical proxy for training scaling.

As expected, both approaches can improve the benchmark score, but they also increase the cost. More interestingly, the curves overlap. For instance, a smaller model at a higher reasoning effort can sometimes reach a similar score as a larger model at a lower reasoning effort.

By the way, the x-axis in this figure shows API cost rather than raw compute. The API cost is a useful practical measure, but it also depends on the provider’s pricing and the number of generated tokens. Also, the exact shape of these curves is benchmark-specific.

So, the model size and reasoning effort form two separate knobs. We can use a larger model, increase the reasoning effort, or combine both. Which combination is best depends on the desired accuracy, cost, and latency.

So far, the article should give you a pretty solid understanding of how reasoning effort modes work and how they are implemented. This is a fine point to wrap the article if you are short on time. Otherwise, if you want to look into some of the nitty-gritty details of some of the recent open-weight models, please read on!

6. Bonus: Different ways to implement reasoning efforts (in flagship open-weight LLMs)

[This section is fine to skip unless you are interested in some additional details]

Section 5 described two possible ways to train reasoning-effort controls, namely effort-conditioned supervised fine-tuning and reinforcement learning with different token costs. Originally, I wanted to cover research articles on alternative ways to implement reasoning budgets. However, reading through most of these articles, they seemed more like proofs-of-concept that may or may not work well in practice.

So, instead of covering those, I decided to pivot a bit and cover those recipes used by state-of-the-art and notable open-weight (flagship) LLMs. For these models, there is at least evidence that the methods work in practice.

This leaves six examples. DeepSeek V4, Nemotron 3 Ultra, Kimi K2.5, GLM-5, Qwen3, and Inkling. They have different levels of detail in their reporting, but each contributes a useful variation. (I exclude models whose reports only show an effort setting in the user interface without explaining how that behavior was trained.)

6.1 DeepSeek V4 trains separate effort specialists

Let’s start with the DeepSeek V4 technical report, which describes the use of three modes:

• Non-think produces a direct response without a reasoning trace.

• Think High is the classic approach where the model places the reasoning trace between <think> and </think> tags. This is similar to what was discussed in the DeepSeek R1 section (section 2) at the beginning of this article.

• Think Max is the same as above but adds a special system instruction. (More on that below.)

The additional system prompt instruction for Think Max starts with “Reasoning Effort: Absolute maximum with no shortcuts permitted.”

At first, this sounds like a simple prompt engineering trick, but this prompt is actually backed by a different training setup. That is, each mode uses its own context window and length penalty (unfortunately, the exact length penalty implementation is not detailed in this report). Think Max receives a longer context window and a smaller length penalty than Think High, which gives it more room to continue reasoning.

So, the system instruction selects a behavior that was created during post-training. Adding the same instruction to an arbitrary model would not have the same effect.

Unfortunately, the public, and otherwise very detailed DeepSeek V4 report does not connect the descriptions of the reasoning mode and domain specialists in enough detail to reconstruct the exact teacher assignment.

However, the report states that the final model, which supports different reasoning effort levels, was created via on-policy distillation from said teachers.

To summarize, DeepSeek V4 develops the three reasoning specialists during post-training. Starting from the base model, it applies supervised fine-tuning followed by RLVR via GRPO. The RL configuration differs for each mode. In particular, each specialist uses its own context window and length penalty, while Think Max additionally receives a special system instruction.

Then, including domain specialists, the different reasoning mode specialists are distilled into a single checkpoint that supports all three effort modes.

6.2 Nemotron 3 Ultra combines learned modes with hard budgets

The Nemotron 3 Ultra technical report describes three settings called reasoning-off, regular, and medium-effort, analogous to DeepSeek V4 in the previous section. Medium-effort is the cheaper reasoning mode compared with regular. NVIDIA introduces this mode during SFT using examples generated by GPT-OSS-120B in its medium-effort mode, and then further optimizes it during RLVR. About 2.5% of the RLVR prompts use medium-effort (this corresponds to length-based adjustments applied to their rewards).

6.2.1 Using Nemotron reasoning budgets during inference

At inference time, all three modes are selected through the chat template.

1) Regular is the default and uses enable_thinking=True, which starts the assistant response with an opening <think> tag.

2) Medium-effort uses enable_thinking=True together with medium_effort=True where the latter setting also appends {reasoning effort: efficient} to the latest user message.

By the way, to further complicate things, the regular and medium-effort modes can also be combined with a separate inference-time reasoning budget. This budget acts as an external stopping mechanism. In the released implementation, the chat client asks the model to end the reasoning trace near the chosen token limit. If the model has not emitted </think>, the client closes the reasoning block and continues generation to produce the final answer. The learned effort mode determines how the model uses its reasoning tokens, while the budget constrains how long the reasoning trace can continue. This makes it possible to pair either mode with a tighter or looser budget depending on the desired cost and accuracy.

3) Reasoning-off uses enable_thinking=False, which prefills an empty <think></think> block (similar to Qwen3 discussed in section 4) so that the model proceeds directly to the final response. Thus, these are chat-template controls rather than system prompts.

6.2.2 Reasoning budget-aware training in Nemotron

The inference controls described above are backed by two related SFT components. The first introduces medium-effort behavior using GPT-OSS-120B traces, as discussed earlier. The second prepares the model for hard reasoning budgets.

To construct this training data, the authors take regular reasoning traces, truncate them at randomly selected token budgets, and keep the original final answers. The inserted </think> token is masked from the SFT loss. As a result, the model sees examples where it has to move from an incomplete reasoning trace to the answer after the reasoning block has been closed externally.

Medium-effort training then continues during RLVR. About 2.5% of the RL prompts use the medium-effort setting across math, STEM, and coding tasks. The report notes that the mode can be calibrated through reward hyperparameters where length-based reward adjustments provide additional control over the cost-quality trade-off.

6.3 Kimi K2.5 alternates budgeted and unconstrained RL

The Kimi K2.5 technical report discusses a training method called Token Efficient RL for lower reasoning effort. (While there was a K3 announcement this week, the reasoning-effort methodology of K3 is not publicly disclosed, but it could be similar or related to K2.5.)

6.3.1 Kimi’s Toggle method

The report mentions that a fixed token budget can make a reasoning model overfit to short solutions. That means the model becomes more concise (i.e., faster and cheaper), but it may lose the ability to benefit from additional inference-time compute and can thus perform poorly.

Kimi K2.5’s method, called Toggle, alternates between two RL phases every fixed number of training iterations:

1. In the budgeted phase, correct solutions are encouraged to stay within a problem-specific token budget.

2. In the unconstrained phase, the usual maximum generation length is restored so that the model can still learn from longer solutions.

For each problem, the budget is estimated from a selected percentile of response lengths among correct rollouts in RLVR. The budget constraint is then only activated once the mean accuracy on that problem exceeds a threshold. This avoids forcing the model to shorten its reasoning before it can solve the problem reliably.

The report evaluates Toggle on K2 Thinking and finds that it reduces generated tokens by about 25 to 30% with little change in benchmark performance. The same behavior also transfers from math and coding RL tasks to GPQA and MMLU-Pro.

Toggle supplies a concrete flagship-model recipe for training a more token-efficient reasoning policy while preserving its ability to scale at test time.

6.3.2 What Toggle changes at inference

Toggle operates entirely during RL training. Both alternating phases update the same policy (i.e., LLM), and the final (unified) checkpoint has no budgeted-versus-unconstrained selector. At inference, the resulting model then runs in thinking mode by default.

Interestingly, though, Kimi K2.5 itself exposes a separate binary choice between thinking and instant modes in some APIs I checked (like vLLM or SGLang). Thinking mode is enabled by default. Instant mode disables the reasoning trace through thinking: {”type”: “disabled”} in the official API or chat_template_kwargs={”thinking”: False} when serving the model through vLLM or SGLang. However, these settings are separate from Toggle.

Also, the official Kimi report does not provide a separate training recipe for instant mode. However, K2.5’s SFT data were generated using both the earlier K2 model, which produces direct responses without long reasoning, and K2 Thinking, which produces extended reasoning traces. This likely exposes the unified checkpoint to both response formats similar to what’s done in Nemotron 3 above. At inference time, the chat template selects between them by prefilling either an open <think> tag for thinking mode or an empty <think></think> block for instant mode. But again, unfortunately, the report does not disclose the exact data mixture or whether additional mode-specific RL was used.

The newer Kimi K3 provides a more direct inference-time effort interface. The current Kimi Code documentation lists three settings called low, high, and max, with max as the default. These are passed through the reasoning_effort parameter. However, Moonshot has not yet explained how the three effort levels were created during training. Its launch post says that these details will appear in a future K3 technical report, so I’ll stay tuned for that.

6.3.3 Kimi K3 (Update)

Based on the new Kimi K3 technical report (released after I published this article), Kimi K3 has three reasoning modes: low-, high-, and max-effort.

For each training problem, the researchers first estimate an initial token budget. On a simple verifiable task, a correct response would normally receive a reward of +1 and an incorrect response 0. But here, if the response exceeds the selected token budget, its reward is instead set to -1. This gives the model a strong incentive to stay within the budget. (The exact reward setup is different for non-verifiable and agentic tasks, and Kimi also uses learned reward models and pairwise comparisons.)

To have Kimi K3 support the three reasoning effort modes, the training looked as follows. First, the researchers start by training a max-effort version (“specialist”) with a relatively generous budget. They then lower the budget to train high- and low-effort versions.

This process is repeated for three domains: general tasks, general agents, and coding agents. And each domain gets one specialist for each effort level. So, for example, one specialist learns low-effort coding behavior, while another learns max-effort agent behavior. This gives nine specialist models in total.

The behaviors of these nine specialist models are then combined into a single Kimi K3 model using multi-teacher on-policy distillation. At inference time, a natural-language thinking-effort instruction in the prompt can then be used to tell the model which effort level to use.

6.4 GLM-5 introduces turn-level and interleaved thinking through SFT

The GLM-5 technical report extends the binary on/off thinking switch introduced with GLM-4.5 to multi-turn and tool-using scenarios. It describes three related behaviors (rather than three effort levels):

• Interleaved thinking: this inserts a reasoning block before each response and tool call.

• Preserved thinking: here, the chat retains earlier reasoning blocks across turns so that the model can reuse them later.

• Turn-level thinking: this enables or disables reasoning separately for each request in a conversation.

At inference time, turn-level thinking is the actual on-off switch. In the Z.ai API, thinking is enabled by default and can be disabled for an individual request with thinking: {”type”: “disabled”}. The hosted implementation is not disclosed but the open GLM-5 chat template shows the equivalent mechanism when self-hosting with Transformers, vLLM, or SGLang.

It starts the assistant response with <|assistant|><think> when thinking is enabled and <|assistant|></think> when it is disabled. The latter closes the reasoning block immediately, so generation proceeds directly to the final answer.

The report says that these behaviors are introduced during multi-task SFT together with an updated chat template.

After SFT, GLM-5 goes through reasoning RL, agentic RL, and general RL. And a final on-policy distillation step uses checkpoints from the preceding stages as teachers. This helps the final model recover capabilities that may have weakened during the sequential RL stages.

6.5 Qwen3 uses mode fusion and inference-time truncation

Qwen3 was already covered in Section 4, so I will only summarize the parts that matter for this comparison. According to the Qwen3 technical report, its post-training pipeline has four stages. These are long-chain-of-thought SFT, reasoning RL, Thinking Mode Fusion, and general RL.

Thinking Mode Fusion is the key stage for the effort on-off switch. Here the model is trained via SFT on a mixture of thinking and non-thinking examples. The /think examples contain a reasoning trace, while /no_think examples begin with an empty <think></think> block that is accompanied by a short answer. The following general RL stage reinforces instruction and format following for both behaviors.

Qwen3 also supports a hard thinking budget. At the requested threshold, the reasoning span is stopped and a stop-thinking instruction is inserted before the model continues with its final answer. The report says that this partial-reasoning behavior was not trained explicitly. It emerged after Thinking Mode Fusion.

This gives Qwen3 a learned on-off switch plus an inference-time budget. It is similar but simpler than the DeepSeek V4 and Nemotron recipes.

6.6 Inkling conditions RL on a continuous effort value

Inkling was already discussed in Section 5.3. The short version is that its technical report mentions that they use continuous effort conditioning (values between 0.0 and 1.0) rather than fixed effort labels.

After a relatively small initial SFT stage, most of Inkling’s post-training comes from asynchronous RL with more than 30 million rollouts. The desired effort is included in the system message, and the token length penalty is adjusted according to that value during RL. As previously discussed, a higher token cost encourages a shorter response. A lower token cost gives the model more room to reason.

6.7 Overview of the known recipes

The table below summarizes what is actually documented in the six technical reports.

So, looking at the six different open-weight models, they have a shared framework. First, they introduce effort mode control through SFT and the chat template. Qwen3 explicitly mixes thinking and non-thinking examples, while GLM-5 adds interleaved, preserved, and turn-level thinking patterns.

The second shared component is a mode-conditioned RL stage, where context windows and length penalties change with the requested effort. DeepSeek V4, Nemotron 3 Ultra, and Inkling use this approach.

A third ingredient improves robustness under explicit budgets. Nemotron trains on randomly truncated traces, Qwen3 can continue from a forcibly stopped reasoning span, and Kimi alternates budgeted with unconstrained RL. These methods help preserve answer quality when the available reasoning length changes and is even cut short.

7. Conclusion

The open-weight examples in this article implement reasoning effort through several different mechanisms. Similar labels can be backed by separate specialists, mixed SFT data, mode-conditioned rewards, hard token budgets, or combinations of these methods.

It is difficult to say which approach is best. The models differ in their base checkpoints, training data, post-training compute, benchmarks, and serving goals. Their reports also omit many details needed for a controlled comparison. (Also, there may not be a one-size-fits-all, and a method that works well for an interactive assistant may be a poor fit for a long-running coding agent.)

The holy grail is of course automatic effort selection. We saw this a while back with GPT 5’s Auto mode. It’s a tricky problem to solve, and in the end, the implementation was probably more miss than hit, which is why it got removed from the UI (at least, I can’t find it anymore).

In the near future, I think reasoning effort will remain an explicit model input, which will most often be delivered through the system prompt. However agent wrapper/harness around the LLM, or an internal router may increasingly infer the appropriate mode and budget from the task state and available resources automatically (while of course still allowing a user override).

I still hope that effort selection will become more automatic. Similar to GPT 5’s auto mode, a cheap model or router could choose the mode from the request, tool state, and remaining time or token budget while still allowing a user override. The override is useful if you want to optimize for latency or cost, or maximum performance.

I realize that this was a long article, and it was perhaps not the flashiest topic. But I thought that given all the talk about LLMs, reasoning models, and agents, a look at reasoning models was something not covered before, and I hope it was a unique and somewhat useful overview!

Further resources

If you want a hands-on implementation of the core training methods behind reasoning models, my Build a Reasoning Model (From Scratch) book walks through reinforcement learning with verifiable rewards and inference-time scaling step by step, with code.

This article focused on how a trained reasoning model can support different effort modes. The book takes a step back and shows how to turn a conventional LLM into a reasoning model in the first place. It is a sequel to Build a Large Language Model (From Scratch) and starts where that book leaves off.

The print edition has now started shipping

If you liked my previous Build a Large Language Model (From Scratch) book, this is essentially a sequel implementing inference-time scaling techniques and reinforcement learning algorithms from scratch.

And if you want to support future long-form articles like this one, consider becoming a paid subscriber. It helps me keep writing these independent deep dives and sharing the accompanying code, figures, and experiments.

I keep seeing the same idea in conversations about agents: Detailed specs are old-world overhead now. Give the model a rough goal, let it explore, fix what comes back, move on. It sounds efficient but it also hides the cost. A simple prompt looks cheap and tempting because it gets implementation started right away. Then […]

People continue to be impressed by yesterday’s Kimi K3 launch. Congrats to Databricks on their $188B Series M (watch our pod on the latest Databricks narratives) and OpenRouter might get bought (watch Alex Atallah’s keynote).

On a slow news day, The most popular talk this week is Abhishek Bhardwaj’s Sandbox track keynote which recaps a year of growth since his original work on Arrakis got him hired by Greg Brockman, and now building out the cloud infra behind ChatGPT Work (upcoming episode!). Spoilers: if you think running agent sandboxes is just “run containers on Kubernetes”, 1) you havent been paying attention to our E2B, Daytona and both Modal podcasts, and 2) you might be overtuned to compute problems and are probably underestimating the importance of storage/filesystems…

If you do leading AI work in NYC, especially for AI x Finance, speaker applications for AIE NYC 2026 opened today.

AI News for 7/16/2026-7/17/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

AI Twitter Recap

Moonshot’s Kimi K3 Release, Frontier Positioning, and the China/Open-Weight Debate

• Kimi K3 is the center of gravity today: the release triggered a broad reassessment of how close Chinese open-weight models are to the frontier. Multiple posts frame K3 as the first genuinely useful Chinese model at this tier, with strong coding, agentic, and long-horizon knowledge-work performance. Community reaction ranged from Salakhutdinov congratulating Moonshot founder Zhilin Yang to practitioners simply reporting that “Kimi K3 is really, really good”. A recurring theme was that K3 narrows the gap enough to pressure US labs to ship faster, as argued by @kimmonismus and others.

• The strategic argument shifted from “compute moat” to “efficiency stack”: a notable thread argues that K3 weakens the thesis that frontier capability is gated mainly by raw FLOPs, pointing instead to MoE routing, quantization, data curation, and scarcity-driven infra design such as Moonshot’s “Mooncake” stack; see @AnikaSomaia. Related commentary emphasized that Chinese labs may be compressing the capability-per-FLOP curve rather than matching Western capex directly, with @dylan522p and @novasarc01 making the case that better post-training and harness conversion rates can shrink product gaps nonlinearly.

• There is still disagreement on how far behind K3 really is: some view it as near-frontier or even surpassing specific Western models on important slices, while others argue it remains several months behind on broader generality, efficiency, or hidden evals. See the skeptical but detailed framing from @scaling01, contrasted with more bullish takes from @kimmonismus and @theinformation. The practical consensus is narrower: K3 is now impossible to dismiss.

Benchmarks: Artificial Analysis, Arena, DeepSWE, ARC, Cyber, and FrontierCode

• Artificial Analysis and coding-agent benchmarks place K3 firmly in the top cluster: Artificial Analysis says the frontier widened from two to six labs above 51 on its Intelligence Index in roughly six weeks, with Kimi K3 at 57, behind Claude Fable 5 at 60 and ahead of Opus 4.8 at 56. On coding agents, AA later reported K3 scoring 57 on its Coding Agent Index, matching GPT-5.6 Terra and GPT-5.5, ahead of Opus 4.8, with 84% Terminal-Bench v2, 64% DeepSWE, and 23% SWE-Atlas-QnA. Cost claims were mixed: AA calls it frontier and relatively efficient; @theo counters that token efficiency and throughput often erase the headline price advantage versus GPT-5.6 Sol.

• Frontend and coding evals were especially strong for K3: Arena reported that K3 put China ahead of the US on Frontend Code Arena for the first time, and user tests echoed that K3 can outperform or match Fable on visually grounded frontend tasks, e.g. @hqmank’s globe dashboard test. On software engineering, DataCurve said K3 debuted at #3 on DeepSWE, calling it the first open-weights model with frontier-level results there.

• ARC and cyber remain useful reality checks: ARC Prize verified that Thinking Machines’ Inkling is now the highest-scoring open-weight model on both ARC-AGI-1 (79.5%) and ARC-AGI-2 (36.5%), while speculation around K3’s ARC-AGI-2 score continues via BenchPress estimates. On cyber, the UK AISI-related discussion around GLM-5.2 matching Opus 4.5 on “The Last Ones” and OpenAI’s claim that GPT-5.6 Sol is SOTA on that range underscores that open models still appear materially behind the best closed models on long-horizon cyber, even as the gap narrows.

Model Architecture, Inference, and Systems Work

• Kimi Delta Attention drew serious technical interest: a strong technical explainer by @sdrzn highlights K3’s use of Kimi Delta Attention (KDA) as a fast-weights style memory mechanism, effectively maintaining fixed-size learned per-request state rather than paying full attention costs over long contexts. The claimed payoff is up to 6x faster/cheaper throughput at 1M context and pricing that stays flatter at long context lengths. If these characteristics hold in wider deployments, this is one of the more consequential architecture-level ideas in the release.

• Serving and hardware discussions followed quickly: people were already preparing K3 deployments on heterogeneous infra, e.g. 4xH100 nodes over RoCE, while Huawei’s “950 SuperPoD” announcement added fuel to the “Chinese AI stack scaling under constraints” narrative. On the software side, vLLM + AMD support, Red Hat AI running Inkling on a DGX B200 node with vLLM, and vLLM’s own note on maintaining production quality under ~2,000 commits/month were relevant infrastructure updates.

• Kernel/perf engineering remains a differentiator: K3 was repeatedly praised for kernel-writing and performance engineering ability, with kernelbench-related examples from Moonshot staff and community comments that K3 helped design kernelbench.com itself. Separately, Simran Arora noted how hybrid linear attentions, full-model megakernels, and fast MLA/DSV4 decode kernels in AMD’s aiter are now directly feeding frontier model development.

Agents, Memory, MCP, and Workflow Scaffolding

• The value is shifting from base model access to harnesses and workflows: several posts argued that as frontier intelligence becomes cheaper and more open, the durable moat moves to orchestration, memory, tools, and domain-specific scaffolding. Good summaries came from @jmorgan and @Yuchenj_UW, the latter framing the key distinction as valuemaxxing vs tokenmaxxing.

• Memory architectures are converging around “wiki memory”: Paulius Ztin’s long post is one of the more concrete design writeups here. The proposal: agents should stop repeatedly re-deriving the same understanding from raw docs and instead build a task-specific Markdown wiki layer over unified memory, synchronized via FastMCP. In the same neighborhood, Qdrant shared production guidance on multitenant retrieval and later highlighted mem0’s view that continual learning is more a memory problem than a weight-update problem.

• MCP and skill abstractions keep maturing: notable product updates included Perplexity Agent API adding custom skills, Hermes Agent desktop and Unreal Engine companion skills from Nous, and advanced MCP usage patterns from Tadas + Anthropic’s Dom. On the research side, MemoHarness stood out: it decomposes agent harnesses into six editable control surfaces and reports 0.806 on Shell-Agent vs 0.722 for the strongest fixed-harness baseline, while lowering per-task cost.

Research Notes Beyond K3

• Robustness and detector limits: the paper “The Illusion of Robustness” argues that aggregate accuracy masks prediction flips under irrelevant context; see the arXiv pointer and a Japanese summary. Separately, Epoch AI reported that AI detectors are usually reliable on plain human text and naive AI text, but LLMs instructed to mimic specific authors can evade detection, with false negatives around 13% and ~26% for scientific writing.

• Embodied and biologically inspired learning: NVIDIA’s RoboTTT extends robot policy context length by 3 orders of magnitude, improving manipulation performance 87% over a single-step baseline and completing a five-minute ten-stage assembly task that no baseline finished. Meanwhile, Sakana’s “Diffusing Blame” and Hardmaru’s summary show competitive learning under strict Dale’s principle without standard backprop weight transport.

• Interpretability / representation geometry: Elie Bakouch replicated Anthropic-style j-space analysis on Thinking Machines’ Inkling, finding it unusual in maintaining similar geometry across early and late layers (early-late CKA ~0.8 vs ~0.5 elsewhere). The same thread reports minimal j-space change under NVFP4 quantization for Poolside’s Laguna XS 2.1.

Top Tweets (by engagement, filtered for technical relevance)

• Open models vs closed model economics: @AravSrinivas compares the moment to Sun Microsystems being disrupted by open source + commodity hardware, arguing local/open models could have a similarly deflationary effect on incumbents.

• US policy implications: @DavidSacks says K3 taking #1 on Frontend Code Arena is a warning against overregulation and data-center constraints.

• Price collapse narrative: @chamath highlights the widening spread between very cheap and very expensive leading-edge tokens.

• Open-weight proliferation impact: @shadcn notes how capabilities once treated as government-sensitive quickly became available to subscribers at commodity prices.

• Frontier coding reality: @datacurve’s DeepSWE result for K3 and @arena’s Frontend Code Arena lead change were the clearest benchmark signals that this release mattered beyond social hype.

AI Reddit Recap

/r/LocalLlama + /r/localLLM Recap

Read more

Capital One on Thursday released VulnHunter, an open-source, agentic AI security tool that scans source code for exploitable vulnerabilities, maps out how an attacker would reach them, and proposes targeted fixes — all before a single line ships to production. The tool, built internally and now available on GitHub under an Apache 2.0 license, is one of the most ambitious attempts by a major financial institution to turn offensive AI capabilities into a public defensive resource.

At a time when security teams are facing a rising tide of new AI threats, Capital One's decision to open-source the tool reflects an effort, according to CISO Chris Nims, to address "an increasingly brief window before sophisticated, next-generation AI attack capabilities become affordable and accessible to virtually every adversary."

Capital One is not simply releasing another vulnerability scanner. VulnHunter introduces what the company calls an "attacker-first forward analysis" — a workflow in which the tool begins at the points where a real adversary would enter a system, such as APIs, network messages, or file uploads, and reasons forward through the application's logic to determine whether an exploit path actually survives the code's existing defenses. Conventional scanners typically work in reverse, flagging a dangerous-looking code pattern and then searching backward for a hypothetical attacker. That approach, security practitioners widely acknowledge, buries engineering teams under avalanches of false positives.

VulnHunter attacks that problem head-on with a second innovation: a built-in "falsification engine" that tries to disprove its own findings before a developer ever sees them. After the tool surfaces a potential vulnerability, a structured reasoning workflow hunts for logical gaps, unsupported assumptions, and conditions that would prevent the attack from succeeding. Only findings the engine fails to rule out reach a human reviewer — and when they do, VulnHunter delivers not just an alert but a full explanation of the exploit path and a proposed code fix ready for engineering review.

The tool currently runs on Anthropic's Claude Opus 4.8 model inside a Claude Code environment, though Capital One says the framework has the potential to work across other foundation models and coding harnesses.

Why Capital One is giving the tool away

Asked why Capital One decided to open-source a tool this consequential, Nims pointed to the communal nature of the problem.

"We felt an imperative to open-source VulnHunter because modern software supply chains are very connected, and the scale of the AI threat is larger than any single organization," Nims told VentureBeat. "Securing software and our digital environments is a shared foundation that benefits developers, enterprises, and the people who depend on the systems we all build. The defensive tools to address this reality need to be just as widely distributed, tested, and improved as the codebases they protect."

"Rather than wait," he added, "we decided that the right response was to build a product that is purpose-fit for today's complex security landscape, and put it into the hands of defenders everywhere."

Why Capital One believes open-sourcing VulnHunter strengthens everyone's defenses

The release nonetheless arrives against a backdrop the company knows well. On July 19, 2019, Capital One disclosed that an outside individual — later identified as a former Amazon Web Services employee named Paige Thompson — had gained unauthorized access to names, addresses, self-reported income, Social Security numbers, and linked bank account numbers belonging to credit card customers and applicants. The breach, which Capital One says occurred on March 22 and 23, 2019, was discovered only after an external security researcher flagged a configuration vulnerability through the company's Responsible Disclosure Program on July 17 of that year.

The damage was sweeping. Approximately 100 million people in the United States and 6 million in Canada were affected. Roughly 140,000 Social Security numbers, about 80,000 linked bank account numbers, and approximately 1 million Canadian Social Insurance Numbers were compromised. The FBI arrested Thompson, and the government stated it believed the data had been recovered with no evidence of fraud. But the reputational and regulatory toll was enormous.

In August 2020, the Office of the Comptroller of the Currency fined Capital One $80 million, finding that the bank had failed to adequately identify and manage risks as it migrated significant technology operations to the cloud. As Reuters reported at the time, the OCC's consent order cited insufficient network security controls, inadequate data loss prevention measures, and a board that failed to hold management accountable when internal auditing surfaced problems. The OCC also ordered Capital One to overhaul its operations and submit new cybersecurity plans for regulatory review.

CyberScoop at the time called the incident "a cautionary tale for companies rushing to embrace new tech." Capital One's own CEO, Richard D. Fairbank, acknowledged the gravity of the moment. "While I am grateful that the perpetrator has been caught, I am deeply sorry for what has happened," Fairbank said at the time. "I sincerely apologize for the understandable worry this incident must be causing those affected and I am committed to making it right."

Inside Capital One's decade-long open-source strategy — and what VulnHunter adds to it.

What followed was not a retreat from technology but a doubling down — with security explicitly at the center.

Capital One began releasing open-source projects in 2014 and declared itself an "open-source first" company in 2015 as part of a broader technology transformation that began over a decade ago. The company has continued to invest in software supply chain security, open-source governance, and AI-driven defense. In August 2022, Capital One joined the Open Source Security Foundation as a premier member, earning a seat on the organization's Governing Board. Chris Nims, then EVP of Cloud & Productivity Engineering, framed the move as a natural extension of the company's operating philosophy. "As a highly-regulated company, we are seasoned in managing compliance and governance and advocate for standardization, automation and collaboration," Nims said in the OpenSSF announcement.

Behind that public commitment lay a substantial operational apparatus. Capital One's Open Source Program Office, now in its third iteration, manages open-source usage, contributions, and community building across the enterprise. The company has released more than 40 open-source projects and has made thousands of contributions to external open-source projects it depends on, according to the company. Those efforts address not just code dependencies but the entire software development lifecycle — DevSecOps tools, infrastructure, and the collaborative environments, both internal and external, that shape how software gets built and shipped.

VulnHunter is the most consequential product of that multi-year effort — and the release reflects a calculation increasingly common among large enterprises: that in security, giving away the tool is cheaper than fighting alone. "At the end of the day our goal is to make this product as broadly accessible as possible," Nims said.

The company argues that modern software supply chains are so deeply interconnected that a single vulnerability in a widely used open-source component can cascade across thousands of enterprises simultaneously. Proprietary defenses, no matter how sophisticated, cannot address a problem that is fundamentally communal. By releasing VulnHunter under a permissive license, Capital One invites the global security research community to stress-test, extend, and improve the tool — effectively crowdsourcing its own defense infrastructure while strengthening the broader ecosystem.

Inside VulnHunter's three-stage AI engine for finding exploitable code

For engineering leaders evaluating VulnHunter, the technical architecture is where the tool's ambitions become concrete. The workflow unfolds in three distinct stages.

In the first stage — attacker-first forward analysis — VulnHunter begins at the points where an external adversary would interact with a system: API endpoints, network message handlers, file upload interfaces. From each entry point, the tool reasons forward through application logic, tracing data flows, transformations, and internal security checkpoints to determine whether an attacker can actually reach a dangerous code path. This approach mirrors how a skilled penetration tester would probe a system, but automates the process at a scale no human team could match.

The second stage is where VulnHunter departs most sharply from conventional scanners. After identifying a potential vulnerability, the falsification engine runs a structured reasoning workflow designed to disprove its own conclusion. It searches for assumptions that do not hold, logical gaps in the exploit path, and environmental conditions that would prevent an attack from succeeding. Findings that fail this internal challenge are discarded before any developer sees them. Capital One's explicit goal is to shift the developer's burden away from triaging false alarms — a perennial pain point that erodes trust in security tooling and slows development velocity.

In the third stage, vulnerabilities that survive the falsification engine trigger an evidence-backed remediation workflow. VulnHunter gathers supporting evidence across the codebase, maps the complete surviving exploit path, explains the defect and the specific capabilities an attacker would gain, and generates targeted code changes for engineering review. The output is not a generic advisory but a concrete, context-aware patch proposal.

Capital One says it validated VulnHunter internally before release, running it across thousands of repositories spanning tens of business areas. The company reports that the tool identified and remediated vulnerabilities with speed and efficiency that far exceeded what its teams previously achieved through manual triage.

Why AI-powered attacks are forcing banks to rethink traditional cyber defenses

VulnHunter arrives at a moment when the cybersecurity landscape is shifting beneath the feet of every enterprise. Capital One's announcement frames the urgency in stark terms: advanced AI models have "dramatically lowered the barrier for bad actors to discover and exploit vulnerabilities in software," and the window before sophisticated AI attack capabilities become affordable and accessible to virtually every adversary is shrinking rapidly.

"Safeguarding information is essential to our mission and our role as a financial institution," Nims told VentureBeat. "We have invested heavily in cybersecurity and will continue to do so to stay ahead of today's evolving threat landscape."

The company's own AI security researchers have been tracking these trends closely. At NeurIPS 2024 in Vancouver, Capital One's team presented research and curated a list of nearly 100 papers spanning LLM safety, adversarial resilience, jailbreak attacks, and synthetic data generation. The papers they highlighted — including work on multi-agent defense frameworks, automated red-teaming, and guardrail classifiers — paint a picture of an arms race in which offensive and defensive AI capabilities are co-evolving at breakneck speed.

Several of those research themes map directly onto VulnHunter's architecture. The falsification engine echoes the adversarial defense strategies explored in papers like "BackdoorAlign," which demonstrated that embedding a structured safety mechanism into a small number of training examples could recover a model's safety alignment without degrading performance. The attacker-first forward analysis reflects the philosophy of "WildTeaming," a framework that collects and analyzes real-world jailbreak attempts to build more resilient models. And VulnHunter's emphasis on minimizing false positives parallels the goals of "GuardFormer," a guardrail classifier that outperformed GPT-4 on safety benchmarks while running 14 times faster.

The thread connecting all of this work is a conviction that traditional, reactive security — monitoring networks, patching known vulnerabilities, responding to incidents after they occur — is no longer sufficient when adversaries can use AI to discover and exploit zero-day vulnerabilities at machine speed. The only durable defense, Capital One argues, is to find and fix the vulnerabilities in your own code before attackers find them first.

What Capital One's cloud security journey reveals about the entire banking industry

Capital One's cloud journey also illuminates a broader reckoning across financial services. When Capital One moved aggressively to Amazon Web Services in the mid-2010s, it was a rarity among major banks. Most financial institutions simply did not trust third parties to store their most sensitive data. Capital One's CIO at the time, Rob Alexander, publicly championed the cloud as more secure than the bank's own data centers — a claim that the 2019 breach complicated considerably.

The CyberScoop report from that period captured the tension within the industry. W. Patrick Opet, managing director of cybersecurity at JP Morgan Chase, described a cultural shift in banking from prioritizing traders to prioritizing developers: "Now, it's 'Focus on the developer, turn everything into code, and automate everything.'" Mark Nicholson, Deloitte's cyber leader for the financial industry, noted that the pressure to move quickly was exposing "weaknesses in the development methodology." And the breach itself was a reminder that even as Chase spent $600 million annually on cybersecurity, relatively simple vulnerabilities — like the Apache Struts bug that enabled the Equifax breach — could undercut massive investments in data protection.

Seven years later, the industry has largely followed Capital One into the cloud, and the security challenges have only intensified. The question is no longer whether to use cloud infrastructure but how to secure the software that runs on it. VulnHunter represents part of Capital One's answer to that question: rather than relying solely on network-level controls and perimeter defenses, push security directly into the code itself, at the moment it is written. The open-source release also carries implicit competitive pressure. If VulnHunter gains traction among developers and security teams, it could set a new baseline for what enterprise security tooling is expected to do — and force rival banks, fintechs, and cloud providers to match or exceed its capabilities.

Whether VulnHunter lives up to that ambition will depend on adoption, community engagement, and the tool's real-world performance against the increasingly sophisticated AI-powered attacks it was designed to counter. But the release itself tells a story that extends well beyond any single tool or any single company. In 2019, a misconfigured web application firewall — exploited alongside an overly permissive cloud access role — exposed the records of roughly 100 million Americans, drew an $80 million regulatory fine, and became one of the most widely cited cloud security incidents in the industry.

In 2026, the same institution is open-sourcing an AI-driven defense built for a new generation of threats — and betting that the best way to protect its own code is to help the entire industry protect theirs.

Intuit was an early pioneer in the usage of agentic AI, but its path to success has hardly been a straight line.

At VB Transform 2026, Intuit VP of AI Nhung Ho described how the company rebuilt its agent architecture twice in the span of about four months, first moving from a fleet of specialist agents to a central orchestration layer, then abandoning that layer for a skills and tools based system once the orchestrator itself started failing under its own complexity. The full second rebuild took 60 days, with a first working version in under 20.

The failure mode that forced the second rewrite was specific. Agents in the orchestrated system passed results to each other in natural language, and each handoff lost context the next agent needed to act correctly. 

"If you have 10 agents and they all are passing to each other, every time that pass happens, error compounds," Ho said.

Why the orchestration layer broke down

Ho said the original push toward specialist agents came from a straightforward customer complaint. A fleet of capable agents is still something a customer has to manage, deciding which agent to use for which task. Intuit's answer was a system that could take a task and route it internally, without asking the customer to pick an agent themselves.

That orchestration layer held up for about three months, which Ho described only half joking as roughly a year in the compressed timeline of agent development in 2026.

It broke for a structural reason rather than a capacity one. Passing outcomes between agents in natural language meant each downstream agent had to infer how the upstream agent reached its conclusion, and that inference degraded with each additional hop. A ten agent chain did not fail occasionally, it compounded errors by design.

That diagnosis is what sent Intuit back to a skills and tools architecture.

The 60-day rebuild, and what it took to get engineering buy-in

Rebuilding a production agent system in 60 days required more than an architectural decision. Ho said the harder problem was internal, convincing both leadership and the engineers who had built the original agents that scrapping recent work was the right call.

The pitch to leadership relied on evidence rather than argument. Ho's team built a demo of the new architecture using real customer queries pulled from production, then showed it performing better than the existing system on the same tasks. 

"The best proof, at least my belief, is what are customers trying to do? And whatever system you build needs to address those problems," Ho said.

Winning over engineering required a different case. Hundreds of engineers outside Ho's core team had built the specialist agents being retired, and the ask was to take their agents apart into individual skills and tools instead. 

Ho said the motivating argument was scale. A standalone agent solved one narrow problem, while a shared skill or tool built into the new architecture could serve every customer who touched that part of the product. That shift also changed what partner teams were responsible for day to day, moving their focus from building agents to running evals, since evals became the only way to measure whether the new architecture was actually working.

Bringing a human into the loop, and feedback at a different scale

The clearest customer facing result of the rebuild is a feature that lets a live agent conversation pull in a human — though it's currently in early testing, live to about 1% of Intuit's customer base. "We're going to be scaling it up in the next few weeks," she said.

Ho said a customer can bring in an Intuit product support person mid conversation, or their own accountant, or one of Intuit's own bookkeepers, and that person joins with the full context of what the agent has already done.

Ho drew a direct contrast with how most AI chat products handle the same situation. A general purpose assistant answering a tax question typically ends with a disclaimer to consult a professional. Intuit's system is built to connect the customer to that professional directly, inside the same conversation.

That human handoff sits alongside a permissions model built for financial data specifically. Every action an agent takes on a customer's financial data requires explicit permission first, though Ho said that requirement can ease over time as customers build trust in the system. Intuit keeps an audit log of everything an agent does that can be reversed if needed.

Feedback in the agentic AI era

The rebuild also changed how Intuit gathers and uses feedback, a shift Ho said is qualitatively different from what came before. 

"Feedback in the past used to be very, very sparse, and it was also very bimodal," Ho said. "Either they loved it or they hated it, and usually it tends towards the negative."

In a chat based system, every conversation functions as feedback, which Ho said moved the company from roughly 0.3% of customers ever giving explicit feedback to something close to 100%.

Ho said she has returned to writing code herself specifically to build models that analyze that feedback volume systematically, looking for where the system is falling short at a scale no manual review process could keep up with.

That volume comes with a tone most product teams aren't used to hearing directly. Customers tell the agent exactly where it failed, in plain terms.

"They straight up tell you, 'You suck. I hate this. This is not right,'" Ho said. "But they're also willing to give the systems grace and correct it as well, and so the onus is on all of us to harvest this new piece of feedback and type of feedback, and actually improve the system."

When you interact with a large language model (LLM)—one of the systems behind chatbots such as ChatGPT and Claude—it can feel as though you are in contact with another conscious mind. But are you, really?

Some prominent scientists, such as Geoff Hinton and Richard Dawkins, claim you are. But most experts remain skeptical, arguing that the impressive cognitive capacities of LLMs occur in the absence of consciousness.

Recently, researchers at Anthropic, the company behind Claude, waded into this debate with an interesting finding. They claim Claude has a normally invisible set of representations of information that guide its internal reasoning and its verbal output.

This is where it gets interesting. The researchers argue this finding can be understood in terms of an influential theory of consciousness called the global workspace theory.

What Is the Global Workspace Theory?

First proposed by the psychologist Bernard Baars in 1998 and further developed by the neuroscientist Stanislas Dehaene and his collaborators, this theory holds that consciousness involves the activity of a “global workspace.” This is a kind of processing hub in the mind or brain that integrates and broadcasts information, allowing it to be used for reasoning, behavior control, and speech.

In a glossy video explaining the work, Anthropic depicts the contents of Claude’s “global workspace” as sailing ships afloat on a vast sea of unconscious mental activity.

How should we react to these developments? Do they provide evidence for artificial consciousness? If so, how strong is that evidence?

What Is a Global Workspace?

We can start by asking whether Claude does indeed have a “global workspace.” This is not straightforward, for the theory gives no formal definition of a global workspace.

The notion is characterized only informally. The (typically implicit) assumption is that any computational workspace “similar enough” to a human’s will qualify as a “global workspace.” But how similar is similar enough?

Claude’s workspace may indeed have much in common with ours, but there do appear to be differences.

For example, the brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network.

A related difference concerns how representations enter a workspace. Advocates of global workspace theory have long argued that in humans, a process called “ignition” occurs in which a non-linear process amplifies and sustains neural representations, allowing them to enter the workspace. As far as we know, nothing comparable occurs in Claude’s case.

Do these differences matter? The answer is not clear. Global workspace theory is based on data drawn from adult humans. There are questions about how far the notion can be—or should be—extended.

Does a Global Workspace Imply Consciousness?

But let’s suppose Claude does have a global workspace. To figure out whether that would be evidence for Claude being conscious we need to consider the status of the global workspace theory of consciousness.

There is no doubt it’s one of the most influential theories of consciousness, but it’s hardly uncontroversial among experts. (In a rather extreme understatement, Anthropic’s paper remarks that “the global workspace model is not universally accepted.”)

Many consciousness experts argue that computational properties alone are enough for consciousness. Even among those who think that consciousness is inherently computational, global workspace theory is only one of many options.

‘Conscious Access’ and Subjective Experience

What’s more, there are questions about whether global workspace theory is really a theory of consciousness in the relevant sense at all.

In an influential paper on artificial consciousness, the neuroscientist Dehaene and his collaborators advance the theory as an account of what they call “conscious access”—the availability of information for recall, the voluntary control of behavior, and verbal report. Crucially, they leave open the question of whether global workspace theory should be understood as an account of the subjective or experiential components of consciousness.

But if global workspace theory is just a theory of “conscious access,” then its implications for the artificial consciousness debate lose much of their significance. When we ask whether Claude is conscious we don’t want to know whether it has “conscious access”—instead, we want to know whether there is anything, subjectively speaking, that it’s like to be Claude. Global workspace theory doesn’t speak to that question if we treat it as nothing more than an account of “conscious access.”

So Has Artificial Consciousness Arrived?

Even taking these complications into account, there is no doubt that Anthropic’s findings are noteworthy. Global workspace theory can be understood as a theory of subjective experience, and Claude may indeed turn out to have something akin to a “global workspace.”

None of this is evidence that artificial consciousness has arrived. But it’s not unreasonable to think these findings do move the dial—if only ever so slightly—in the artificial consciousness debate.

But if that’s right, then it’s puzzling why Anthropic is quite so upbeat about these developments. As Anthropic recognizes, the creation of artificial consciousness would be a momentous event with wide-ranging social, ethical, political, and legal ramifications.

If chatbots are conscious then we would need to take their interests seriously. It would no longer be permissible to treat them as mere machines; instead, we would need to consider their welfare.

Should Anyone Even Be Trying to Do This?

Anthropic remarks that “it’s time to start thinking about whether we should be building conscious machines.”

I agree we need to have that discussion, but we should also pause work on building machines that might potentially be conscious. If Anthropic were serious, it would surely down tools rather than plough ahead with its attempt to develop conscious AI.

A moratorium on AI research that might be thought to lead to conscious AI would, of course, be far from straightforward. There are questions about the range of research it would affect and who might enforce it. But if we don’t close the stable door now we might find that the horse has already bolted.

This article is republished from The Conversation under a Creative Commons license. Read the original article.

The post Anthropic Says Chatbots Have What May Be a Key Feature of Consciousness. Are They Right? appeared first on SingularityHub.

Legacy infrastructure, not the models themselves, is what's actually slowing AI agents down. That was the shared conclusion of three infrastructure leaders — from LinkedIn, Walmart, and Zendesk — at VB Transform 2026.

The panel brought together Animesh Singh, senior director of AI platform and infrastructure at LinkedIn, Desiree Gosby, SVP of corporate technology services and technology strategy at Walmart, and Sami Ghoche, VP of applied AI at Zendesk, each describing what actually broke when they moved agents from pilot to production. Each arrived at the same conclusion from a different starting point: None of the bottlenecks they hit were model problems.

What tied their answers together was a shared premise: most enterprise infrastructure was built for how humans work, not for how agents work. The gap between those two speeds is where the real engineering happened.

Gosby put it plainly when asked what she'd learned scaling agents inside Walmart's own workforce. The goal, she said, is to make sure "engineering doesn't once again become the bottleneck for what it is we're trying to do."

Where the bottleneck actually was

Each company hit a different version of the same wall: infrastructure designed for how people work doesn't hold up once agents are doing the work instead.

At LinkedIn, the first bottleneck wasn't a model, it was Kubernetes, which assumes containers spin up on demand, a process that takes seconds. Singh said that's too slow for agents. The fix was moving from on-demand provisioning to pre-provisioned pools of containers that swap agentic workloads in and out in real time.

A second, harder problem surfaced once LinkedIn let agents control their own orchestration. A five-point evaluation system looked clean, but hallucination kept showing up anyway. Singh said the issue was structural, an LLM evaluating another LLM's output shares the same failure mode as the thing it's evaluating. 

"We built our own harness, our own control flow, and pushed the LLMs to the leaf instead of them orchestrating the loop," Singh said. Roughly 80% of the workflow is now scripted, deterministic code, with LLMs used only where reasoning is required, and each step's evidence is committed to disk before the system moves on.

Walmart's bottleneck came from success. An agent harness put directly into employees' hands went viral internally, and what Gosby called "citizen developers" began building their own agents to solve problems that once required a formal engineering roadmap. The upside was real innovation. The downside was duplication, dozens of overlapping agents with no coordination. The fix wasn't reining in the harness, it was building governance to spot duplication, promote the best version of an agent, and get it into production without engineering becoming a chokepoint.

Zendesk hit its bottleneck from the data side. Ghoche, who joined through Zendesk's acquisition of Forethought, which closed in March 2026, described sitting on what he called a public figure of 20 billion customer conversations in Zendesk's repository. The instinct is to hand that history to a large language model with a big context window and let it generate the agents a business needs. Ghoche said that doesn't work. "You can't really do that, so instead you have to really invest in the underlying data pipelines and all the data infrastructure that comes with that," he said.

The role of open source

On open source, all three leaders landed on a similar instinct: own what you can, and lean on frontier labs only where they still have a clear edge.

Ghoche said his own view is that most enterprises would prefer to own their models and infrastructure wherever that's possible, and that reasoning is what drives Zendesk's own approach. The exception is frontier reasoning work, where the labs still lead, though he said that slice of use cases is shrinking relative to everything else enterprises now do with AI.

LinkedIn's answer was to build two subsystems specifically for independence. The first is what the company calls an AI gateway, a single interface that every outbound call to a model runs through regardless of provider. The second component is a memory subsystem built to hold context independent of any model provider.

"Every single outbound call going to an LLM, whether it's on a public cloud or on-prem in our own data centers, follows the same semantics, the same API calls. We can quickly switch between different providers," Singh said. 

Walmart built its own internal gateway to stay vendor agnostic across three workload types: fully deterministic workflows, planner-and-reasoner workflows for open-ended tasks, and a hybrid of the two. Compliance-heavy work stays deterministic by design; governance, security and evaluation run through the gateway regardless of which model is on the other end. Gosby said the choice between a frontier model and an open-weight model comes down to whichever is most effective for the specific workload, not a fixed policy.

Advice for the modernization journey

Three pieces of advice came up directly, each tied to the wall a leader had already hit.

Invest in evals before anything else. Ghoche called it the thing common to every use case, internal or customer facing. 

"The thing that's common to all of these is evals. It'll force you to break the problem down, and once you have a robust set of evals, you can move a lot faster," he said, 

Own your agent harness from day one. Gosby's advice was to put the AI harness directly in employees' hands early, paired with the infrastructure to monitor what it produces. 

"It will unlock a huge amount of innovation," she said.

Build for model and context independence. Ensuring flexibility is critical for success.

"Build for independence, whether it's a frontier model of today versus an open source model of tomorrow," Singh said. "Keep that context within your enterprise so that you can reuse it when you ship the model or the harness tomorrow," Singh said.

Like many engineers, Sarah Downs says she knew she wanted to pursue a STEM career from a young age. As a teenager, she discovered robotics through her Tulsa, Okla., middle school’s First Lego League team, and she fell in love with the field, she says. Downs participated in the international robotics program from 2014 to 2016.

Watching PBS specials on NASA Mars rovers Spirit and Opportunity, and seeing the live broadcast of the Curiosity rover launch in 2011, inspired the teen to dream of a career working with NASA.

Sarah Downs

MEMBER GRADE

Graduate student member

UNIVERSITY

Texas A&M University in College Station

MAJOR

Electric engineering

This year the IEEE graduate student member achieved that dream. For her final project as a master’s degree candidate in electrical engineering at the University of Tulsa, she worked on an algorithm in collaboration with NASA and the U.S. Air Force.

The algorithm she developed enables a robot assembling satellites in space to insert an antenna into the correct spot, addressing robotics’s classic peg-in-hole problem of inserting an object into its corresponding hole.

Now a Ph.D. student in electrical engineering at Texas A&M University in College Station, Downs is continuing her research on satellite assembly and manipulation “but on a much larger scale,” she says.

Following a childhood passion

Downs grew up in the Tulsa area. Her father, who died from a heart attack in 2015 when she was 13, was a safety advisor in the oil and gas industry. Her mother stayed home to take care of her brother, who has autism. After her father died, her mother went back to college to earn a bachelor’s degree in business so she could support the family.

“We didn’t have much income, and my mom was always worried about money,” Downs says. “That made me more aware of having a successful career, in a monetary sense.”

From then on, whenever she considered her future career, having a decent salary to support the family was high on her list.

By pursuing a career in robotics, she says, she can follow her passion while obtaining financial security.

In high school, Downs joined the First robotics club, where she found herself drawn to the electrical components used in the machines she and her classmates built.

During her final two years of high school, she participated in an extension program at Tulsa Tech, a training school. She spent half her day in high school classes and the other half taking engineering courses at the vocational school.

After graduating in 2020, she accepted scholarships to attend the University of Tulsa. She began her freshman year at UTulsa not knowing whether she wanted to major in electrical or mechanical engineering, she says, adding that her love of working with small systems helped her choose EE.

For her senior year capstone project, she and two of her classmates designed a lunar lander exhibit for the Tulsa Air and Space Museum. They created an interactive game that simulates missions on lunar and martian surfaces. Four celestial bodies—the moon, Venus, Mars, and Titan—are listed across three computer monitors. Using a game controller, museum visitors can explore the virtual surface of each one. The exhibit is still on display.

Downs earned her bachelor’s degree in electrical engineering in 2024 and continued her education at the university’s EE master’s degree program.

Both more and less complicated than people think

When Downs began her graduate studies, she was supposed to be part of a NASA robotics project for two years. But when a delay in government funding postponed the project’s start, she instead spent her first year in the school’s Institute for Robotics and Autonomy, then newly launched. Its main focus is developing robots to assist people who have mobility challenges.

Inspired by her grandmother, who was wheelchair-bound due to severe arthritis, Downs developed a robotic arm that helps older people and wheelchair users live independently. The arm was able to identify and place objects in the appropriate locations inside the home, such as unloading certain groceries from a shopping bag and placing them on a shelf or in separate containers.

Before the start of her sophomore year in 2025, the NASA project finally secured government funding. She developed a robot that achieves the peg-in-hole task without using any vision systems. Typically, cameras help guide robots’ satellite-assembly work. But in the harsh, remote environment of outer space, cameras might malfunction or encounter delays.

“Don’t stop asking questions. Especially in engineering, don’t pretend like you know everything, because science is about constantly wanting to learn and listen.”

Rather than using cameras, Downs’s robotic arm deploys a force-based insertion process to sense position and orientation of objects in the arm’s environment. The robot loosely grips an antenna and, with a torque sensor on its gripper, “feels” the force feedback of where the satellite and antenna are in relation to each other. The robot then guides the antenna assembly into a target opening on its satellite and maintains the position during adhesion.

Adding to the complexity, the robot performs its task in zero gravity.

“Without gravity, you now have to consider the arm’s reaction torques on the satellite to avoid flinging it into space,” Downs says. Any motion from the arm during the insertion process, especially from increased forces, could cause the satellite to continue movement in that direction.

To combat that, Downs is performing calculations for the project to direct targeted reverse thrusts and counter the force of the robot’s motions.

Her graduate project captures the simple yet complex nature of robotics that she finds fascinating, she says.

“I think robots are both more and also less complicated than people think,” she says. “Really, all you need to start programming a robot is its Denavit-Hartenberg parameters, and you can do a lot with that,” she says, referencing the four values used to describe the position and orientation of a robotic arm and manipulators. Even with different grippers and degrees of freedom, “fundamentally, all robot manipulators start there,” she says.

“But,” she adds, “we’re still learning so much about how robots interact with their environment. Even something simple to us, like manipulating a pen, is still incredibly complex for robots.”

Downs is completing her doctoral thesis in the Robotic Space Simulator project at Texas A&M’s Robotics and Automation Design (RAD) Lab, which specializes in developing machines that can survive in extreme environments. It collaborates with NASA.

Her thesis advisor is Robert Ambrose, a NASA veteran who launched the RAD Lab in 2022. The IEEE member is set to serve as associate director of the school’s Space Institute, due to open this year in Houston. The research facility is being built next to the Johnson Space Center.

After earning her Ph.D., Downs says, she hopes to one day work for NASA, developing rovers that collect samples from Mars or robotic arms that perform tasks on space stations.

To learn more about robots, check out IEEE Spectrum’s guide.

Getting out of the engineering bubble

Downs joined IEEE in 2020 as a freshman at UTulsa to get more involved in electrical engineering events on campus. At the time, the COVID-19 pandemic kept clubs and organizations from meeting in person.

She was active in her school’s IEEE student branch and was elected as its 2022–2024 president. Under her leadership, the branch went from having a few events to hosting one every two weeks.

They included lunch-and-learn sessions and dinners that connected students with professional engineers and the university’s alumni. Downs also organized hands-on workshops on soldering, 3D printing, CAD modeling, and résumé-building.

Her efforts helped increase the branch’s executive board membership from roughly five students to 25 in 2023. The same year, her soldering workshop attracted about 80 students.

She says she enjoyed working with IEEE, especially “engaging with alumni and learning from engineers.”

IEEE is a great resource for networking opportunities, she says, noting that “during the COVID-19 pandemic, engineering students stayed in their bubbles.” IEEE events helped the students make connections that could serve them well, she says.

“Networking is very important, especially in today’s tough job market,” she says. “It’s a lot about who you know and how people observe your work ethic.”

Downs, who now serves as an IEEE graduate advisor for UTulsa’s student branch, says she has seen firsthand how the school’s student branch network has benefited its student members.

“A lot of them have found jobs” because of IEEE, she says.

The working and networking of an engineer

As the IEEE graduate advisor for UTulsa’s student branch, Downs noticed that many engineering undergraduates finish college without any hands-on experience, whether it be a project or an internship.

“Their résumés are very sparse, and they have no proof of their technical skills,” she says. She herself completed a facilities engineering internship at Tulsa International Airport’s American Airlines maintenance facility after her sophomore year of college. And she was an electrical engineering intern at Flight Safety International outside Tulsa after her junior year and after she graduated. The company designs, builds, and maintains its own flight simulators.

Her advice to undergraduates is to hone and demonstrate both their hard and soft skills by working on research projects or even personal passion projects.

“A Raspberry Pi doesn’t cost that much, and you can start working with that immediately,” she says. Students also can take part in engineering interest groups and professional organizations at their school, she adds.

“Put yourself out there and join a research team,” she says. “It’s a great way to show people that you’re a good person to work with and you’d do a good job in the field.”

She adds that it’s also a fine way to keep learning—which is what drew her to a field that has developed only within the past century.

“We’re still constantly learning about robots,” she says.

“Don’t stop asking questions,” she advises students. “Especially in engineering, don’t pretend like you know everything, because science is about constantly wanting to learn and listen.”

OpenClaw has become one of the most widely adopted agentic frameworks, but it has yet to prove itself at enterprise scale. Agents need real credentials — API keys, OAuth tokens, service accounts — to work effectively, and Brex found that traditional guardrails couldn't contain what those agents were doing with them.

Brex set out to overcome these limitations by building an internal platform it calls CrabTrap. The open-source HTTP/HTTPS proxy intercepts all network traffic, examines policy rules, and uses a LLM-as-a-judge to decide whether agent requests should be approved or denied. 

“What we noticed was that the network layer was an untapped enforcement point,” Brex co-founder and CEO Pedro Franceschi told VentureBeat. “Every request an agent makes is an opportunity to intercept, reason about, and make a policy decision.”

The takeaway Franceschi wants IT leaders to draw: agent governance should shift from SDK-level permissions and model guardrails toward a centralized network control plane that enforces and learns from real in-the-wild agent behavior.

How Brex targeted the transport layer

The “obvious fix” (at least initially) to the agent security gap was guardrails, and much of the early work has centered on scoped tools, per-action permissions, and human-in-the-loop approvals. But as agents evolve, each new capability means there’s another API to tune or surface to audit, Franceschi noted. 

“Any agentic system with multiple tools and access to the open internet creates an immediate tension for builders: The more capable you make an agent, the more dangerous it becomes, and the safer you make it, the less useful it is,” he said. 

Existing solutions to this tradeoff were “weak”: Fine-grained API tokens help at the margins but can still be misused and constrain functionality. Semantic guardrails (such as context, skills, or prompt steering) are easily bypassed by prompt injection, especially for agents connected to the internet.

Agents can be “defanged” when given read-only access or limited toolsets, but then they can't do meaningful work, Franceschi said. On the other hand, granting broad write access and a large tool surface can result in hallucinations and real production consequences.

Model context protocol (MCP) gateways enforce policy at the protocol layer — but only for traffic using MCP. Meanwhile, guardrails from LLM providers are tied to a single model and can be “opaque” to customize with enterprise-specific policies. And powerful tools like Nvidia OpenShell offer more of a “per-sandbox egress control.”

“When we started, we hadn’t found a solution to deploying harnesses like OpenClaw safely,” Franceschi said. “Instead of waiting for the industry to catch up, we decided to own the problem and invent the necessary tools.”

Notably, they needed a platform that sat between every agent and every network request, and could make “nuanced decisions about what to allow,” he said. 

This made the transport layer a core architectural component and natural starting point, he said. 

By operating at this layer, CrabTrap is framework-agnostic, language-agnostic, and API-agnostic. It doesn't require SDK wrappers or per-tool integration. Users set HTTP_PROXY and HTTPS_PROXY in the agent's environment, and every outbound request routes through the proxy before it reaches a destination.

However, Franceschi emphasized, Brex didn't start at the transport layer because it thought it was the only answer; rather, they believe in “security by layers.”

“The transport layer was simply an underinvested one, and we saw an opportunity to add meaningful enforcement there alongside everything else,” he said. 

The LLM-as-a-judge training loop

CrabTrap combines deterministic static rules with an LLM-as-a-judge for requests that fall outside known patterns, Franceschi explained. The judge only “fires on the long tail of unfamiliar endpoints or unusual request shapes,” which for a mature agent is typically fewer than 3% of requests.

The more pressing problem was how to know that a policy is the right one? With static rules, it's “relatively straightforward” to reason about accuracy. But with an LLM judge, the system is nondeterministic, and users need confidence that the policy approves the right requests and blocks the rest.

“Our key insight was to bootstrap policy from observed behavior rather than write it from scratch,” Franceschi said. Beginning with real behavior and editing down based on real-world learnings turned out to be “dramatically more effective than starting from a blank page.”

Brex’s team built a policy builder (itself an agentic loop) that runs underlying agents in shadow mode, analyzes historic network traffic, samples representative calls, and drafts a natural-language policy that matches what the agent actually does. 

From there, they built an eval system that tests policy changes before they go live. CrabTrap compares historical audit entries against a draft policy and reports the exact changes to be made. Users can slice results by method, URL, original decision, and agreement status. 

All of this runs with concurrent judge calls, so replaying thousands of requests “takes minutes, not hours,” Franceschi said. Brex also developed a live feedback loop: Full audit trails are stored in PostgreSQL and queryable through the admin API and dashboard. In cases where a resource is continuously denied, the system can notify a human or an agent to propose a policy update for review. 

“That closes the loop between observed denials and policy refinement,” Franceschi said. 

Core challenges and roadblocks 

Of course, the build wasn’t without its challenges. A big one was latency: “Putting an LLM between an agent and every outbound API request sounds like it would grind things to a halt,” he said. 

However, it didn’t turn out to be as big a problem as expected. This was for two reasons: The LLM judge only activates on a small fraction of requests (the aforementioned 3%). Agents quickly settle into predictable traffic patterns; once observed, high-volume patterns become static rules. Second, by using small, fast models like Claude Haiku meant that, even when the judge did fire, added latency was “negligible.” This can be further reduced with local models and prompt caching, Franceschi said.

The harder and less obvious challenge was prompt injection, he said. The judge receives the full HTTP request and all content is user-controlled, so potentially, a crafted URL, header, or request body could manipulate the judge's decision. 

Brex addressed this by structuring the request as a JSON object before sending it to the model, so all user-controlled content is “escaped rather than interpolated as raw text,” Franceschi said. 

Results, and where CrabTrap might evolve

Brex tracks a few factors to measure CrabTrap’s internal impact: Engagement with agents, network traffic patterns, and net promoter scores (NPS). The most meaningful result of CrabTrap has been “organizational confidence,” Franceschi said. 

Previously, the team had “real hesitation” when it came to deploying autonomous agents broadly across business operations, because the existing guardrail options didn't provide enough assurance. 

“CrabTrap changed that calculus,” Franceschi said. They now have an enforcement layer they trust, increasing confidence around expanding agent deployment into more parts of the business and delegating more agent configuration and management to users. 

Franceschi described the policies derived from traffic as “surprisingly strong.” The team expected the policy builder to produce a “rough starting point” requiring heavy manual editing. In practice, though, pointing the platform at a few days of real traffic produced policies that matched human judgment on the “vast majority of held-out requests.”

Additionally, CrabTrap revealed how much noise agents generate. “The audit trail made this visible for the first time,” Franceschi said. They used denial logs and traffic analysis not only to tune policies, but to tighten agents themselves, remove tools, and cut out entire categories of requests that were wasting both time and tokens.

“The proxy became a discovery tool, not just an enforcement one,” he said. 

Areas for growth (and input from the open-source community)

Brex anticipates CrabTrap to continue to evolve, particularly as they have released it as open-source. “We hope the community helps shape it,” Franceschi said. 

Areas of improvement include deeper authentication functionality such as single-sign on (SSO), fine-grained role-based access control (RBAC); escalation workflows that allow agents to request additional permissions; and policy recommendations based on denial patterns.

Programmatic configuration, or developing API endpoints for “creating, forking, and applying” policies to agents, could allow the whole policy lifecycle to be automated rather than managed manually, Franceschi said. 

As for escalation, if an agent is continuously denied a given resource or endpoint, it should be able to route requests to humans or other AI agents for review and back that up with a rationale for why it needs access. 

“That turns CrabTrap from a hard enforcement boundary into something more like a managed permission system,” Franceschi said. 

Additionally, the policy was built to bootstrap from network traffic, but there is opportunity to incorporate additional signals around agent traces and resource-calling, as well as broader context on what agents are ultimately trying to accomplish. This can help produce more accurate and nuanced policies. 

Finally, there's an “open philosophical question” about the right posture for CrabTrap: Should it be a fully transparent layer that the agent itself is unaware of, or should it operate more like a “well-intentioned manager”? (that is, the agent knows about the layer and can interact with it).

The open-source community can help shape these developments, and CrabTrap will only get better with more users, Franceschi said. Brex’s agents speak to a specific set of APIs; teams using CrabTrap with different agents, services, and policy requirements will surface “edge cases and patterns we can't hit alone.”

“We have ambitious plans for where it could go, and we’d rather build in the open,” Franceschi said. 

What other builders can learn from CrabTrap

The response has been stronger than expected. CrabTrap has more than 700 stars on GitHub. Franceschi said Brex has also heard from OpenAI, Y Combinator CEO Garry Tan, and programmer Pete Steinberger, all expressing interest in deploying similar internal infrastructure.

The broader lesson: “Don't let infrastructure gaps become excuses to wait," Franceschi advised. There are “real blockers” for every enterprise looking to seriously deploy AI agents, including security concerns, lack of tooling, or unclear guardrails. 

“It's tempting to sit on your hands until the industry catches up,” he said. “The lesson from CrabTrap is that you can own those problems directly.”

Filesystems and sockets once looked like unnecessary layers of indirection; now they’re the foundation of modern systems. Cryptography is ready for the same shift — and with PQC on the horizon, the timing is right.

Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

Summer School on Multi-Robot Systems: 29 July–4 August 2026, PRAGUE

Actuate 2026: 18–19 August 2026, SAN FRANCISCO

IROS 2026: 27 September–1 October 2026, PITTSBURGH

Humanoids Summit Seoul: 22–23 September 2026, SEOUL

Enjoy today’s videos!

In this work, we present a systematic evaluation of contemporary humanoid technology for laparoscopic surgical tasks. We develop a humanoid-based laparoscopic teleoperation framework using general-purpose instruments and assess its capabilities through benchtop characterization, dry-lab user studies spanning diverse surgical experience levels, and in vivo porcine studies. Across these evaluations, we quantify technical feasibility, task performance, and clinical readiness relative to established surgical platforms. Together, our study provides an evidence-based assessment of the current capabilities and limitations of humanoids for surgical applications, highlighting both their promise and the key technical challenges that must be addressed before clinical deployment.

[ UC San Diego ]

Thanks, Ioana!

Today, we preview ACT-2, the first robotics model to achieve reliability by unifying broad generalization with high performance.

Sunday also has this 3-hour video (!) of Memo folding laundry in “never seen environments.” Let’s just not ask, because we almost certainly don’t want to know.

[ Sunday Robotics ]

Spot is not the first quadruped to try its legs at last few meters package delivery, but the challenge is not really those last few meters—it’s going to be not driving the human coworker nuts, is my guess.

[ Boston Dynamics ]

Quadrupedal locomotion in complex environments requires multiple motor skills, stable gait transitions, and perceptive control over a broad range of speeds. APT-RL (Action Pretrained Transformer-based Reinforcement Learning) is a unified framework for high-speed, multiskill locomotion. A single policy selects and transitions between gaits and motor skills using only onboard perception and computation. In real-world experiments, KAIST HOUND traversed stairs, hurdles, stepping-stones, gaps, and fallen branches. It reached an instantaneous peak speed of 4.25 meters per second while traversing a 60-centimeter step and 6 m/s during a drop-down transition on a three-step staircase.

[ KAIST DRCD Lab ]

We will have much more on this next week.

[ Walden Robotics ]

Today, we introduce Lumo-2, our next-generation latent world-action model for generalist embodied robot learning.

[ Astribot ]

Following Atlas’s first-of-its-kind live performance at the FIFA World Cup 2026, we caught up with Seth Davis, senior program manager, to learn how this demonstration came together and what it takes to succeed in the field (and on the pitch).

[ Boston Dynamics ]

No teleoperation. No cuts. Long take. One of the world’s few complete demonstrations of long-horizon mobile manipulation, bringing fully autonomous humanoid robots another step closer to us.

[ LimX ]

Thanks, Jinyan!

Impressive. But get a job.

[ MagicLab ]

We saw some footage of this last week, but here’s a much better video.

Wing-propelled diving birds flap their wings to move through air and water, yet the wing morphology and kinematics that enable this behavior remain poorly understood because of the difficulty of collecting in situ data. The impact of flapping frequency, wing size, and stiffness on locomotion in—and transition between—the two media are still unknown. We compared data from diving birds against experiments using a flapping-wing robot capable of flying, swimming, plunge diving, and exiting the water. We show that frequency adaptation, flexible wings, and powerful actuation enable seamless transitions without folding wings or legs, that large wings enhance flight without substantially reducing underwater efficiency, and that tail-body distance and egress angle affect water exit. These results clarify how birds (and robots) balance multifluid locomotion constraints.

[ EPFL LIS ]

As enterprise AI deployments accelerate, businesses find workflow redesign, change management and employee trust, not just the tools, determine adoption success.

Sarah Friar, CFO of OpenAI, introduces a practical AI scorecard to measure ROI through useful work, cost per successful task, dependability, and return on compute.

As enterprise leaders start deploying agentic workflows, they must establish the infrastructure to build and run them, one capable of fluidly routing a diverse set of workloads across the most efficient compute resources.

This requires the ability to manage heterogeneous infrastructure, utilizing high-performance accelerators for large-scale training and inference, and utilizing CPUs for the critical orchestration layer of agentic AI. As autonomous agents become more prevalent, CPUs are ideally suited for managing agent state, semantic routing, tool selection, and spinning up secure, isolated sandboxes to safely execute untrusted generated code.

The Google Axion advantage

Google Cloud, with its workload-optimized Compute Engine portfolio, which includes general-purpose and specialized offerings, shines in addressing this need.

Google Axion processors within this portfolio comprise a family of custom Arm processors engineered for performance, efficiency, and versatility, with a feature set that supports general-purpose workloads, CPU-based AI workloads, and other specialized tasks requiring Arm-native compatibility and direct hardware access.

Axion is Google’s first custom Arm-based server CPU, introduced in April 2024. It is designed specifically for hyperscale cloud and AI-era data center workloads. 

Axion also leverages more than a decade of Google’s custom silicon innovation. This enables Google to more readily incorporate customer feedback into chip designs and address the more general, though complex, needs of CPUs. 

Matching workload type to the processor

Bhumik Patel, Director of Software Ecosystem Development at Arm, says the key to all of this is to match the workload type as closely as possible to computing capacity. CPU-powered cloud instances are a practical option for certain AI workloads, particularly those with smaller datasets or less complex models. 

“Agentic tasks such as orchestrating, talking to APIs, and memory management are all ones CPUs are good at, so it’s a distributed and concurrent AI workload,” Patel tells The New Stack. Intelligent workload-processing apportionment makes agentic AI more cost-effective and efficient than running all workloads on a single compute type.

This efficiency is quantifiable. The Google Kubernetes Engine Agent Sandbox running on Google Axion N4A provides up to 30% better price performance than the next hyperscale cloud provider, says Google’s Mo Farhat, Axion Group Product Manager. The GKE Sandbox is an open-source Kubernetes-native primitive designed to execute untrusted AI-generated code safely. 

“Agentic tasks such as orchestrating, talking to APIs, and memory management are all ones CPUs are good at, so it’s a distributed and concurrent AI workload.”

Intelligent workload decoupling makes agentic AI significantly more cost-effective. Google Cloud’s fluid computing foundation enables engineering teams to reserve specialized accelerators strictly for heavy reasoning and generative workloads, while leveraging Axion CPUs for high-concurrency orchestration and context management.

Secure execution with the GKE Agent Sandbox

As agents begin to generate and execute dynamic code autonomously, security is non-negotiable. Running AI-generated code directly in a standard cluster poses severe security risks, as untrusted code could potentially access other apps or the underlying cluster node.

The Google Kubernetes Engine (GKE) Agent Sandbox resolves this by providing an isolated environment for safely executing untrusted code. Running on Axion-powered N4A instances, the sandbox provides up to 30% better price performance than comparable workloads on other hyperscalers.

The vertical stack isolates sensitive tasks at the kernel level with sub-second latency.

The vertical stack isolates sensitive tasks at the kernel level with sub-second latency.  GKE Agent Sandbox natively supports gVisor (an open-source application kernel developed by Google that acts as a secure sandbox for containers) and default-deny Kubernetes network policy. Agent Sandbox provides pluggable interfaces for open-source sandboxes, such as Kata Containers, enabling users to customize their kernel isolation. 

Powered by gVisor technologies with software support from Arm’s architecture, the sandboxes intercept and validate system calls before they reach the host kernel. These isolated execution environments enable deployment of autonomous systems at scale without sacrificing performance or operational agility.

To manage resources efficiently when agents sit idle, GKE Pod snapshots allow users to save and restore the exact process state of sandboxed environments. This functionality provides four major architectural benefits:

• Fast startup: Reduces sandbox startup time by restoring from a pre-warmed snapshot rather than initializing from scratch.
• Long-running agents: Pauses sandboxes that take a long time to run and resumes them later—or moves them across nodes—without losing progress.
• Stateful workloads: Persist an agent’s context, such as conversation history or intermediate calculations.
• Reproducibility: Captures a specific state to use as a baseline for spinning up multiple new sandboxes.

Getting started

As token generation, autonomous workflows, and continuous agent interactions grow exponentially, relying exclusively on accelerator-backed stacks for every task will become financially and architecturally unsustainable.

The combination of CPU and accelerator execution accounts for bursts in agent activity and unpredictable demand spikes by eliminating the inference tax. Google Cloud’s full-stack advantage enables organizations to deploy the right machine for the job. 

By using Google Axion and GKE Agent Sandbox, builders can optimize total cost of ownership and security while maintaining the performance required for AI agents.

Learn more about Google Axion.

The post Arm and Google offer a smarter option to run agentic AI workloads appeared first on The New Stack.

Every morning, airline dispatchers, grid operators, and farmers around the world make decisions based on the same thing: a weather forecast.

While these forecasts are something that most people glance at for two seconds, weather predictions influence major strategic decisions in many industries, with real money, livelihoods, and even actual lives at stake. Farmers use them to determine which crop variety to sow, when to fertilize, how much to invest in irrigation infrastructure, and how long livestock should graze. Utilities use them to decide where to build solar and wind farms, as well as how to price wholesale electricity. Predictions are used to warn people about extreme weather and to trigger emergency response measures. More recently, weather predictions have become relevant for an emerging industry: prediction markets, where people bet money on all kinds of real-world events, including the weather.

However, the temptation to manipulate weather data to get an edge in these markets, combined with a collective move toward data-driven AI weather forecasting, is starting to put the accuracy of weather predictions at risk. These risks are relatively manageable for now, but as experts in the field, we can foresee scenarios where they snowball into far bigger, more systemic problems. 

To develop weather predictions, we need accurate observations of current conditions. These are collected from several sources, including weather stations at airports, utilities, or transport services. Traditional operational systems like the Weather Research and Forecasting model or the European Centre for Medium-Range Weather Forecast (ECMWF) Integrated Forecasting System combine these observations with numerical approximations in order to estimate future weather patterns. 

Sometimes, weather stations have issues because of, for example, instrument failures or upgrades in equipment. These can be caught either in real time (through checking and correction) or retroactively. Traditional forecasting systems also have a built-in safeguard called data assimilation: Every incoming measurement is weighed against what the physical model says should be happening and against readings from nearby stations.

Together, these mechanisms help keep weather observations reliable and predictions robust. However, new threats are putting observational accuracy at risk. Earlier this year, news outlets reported that the weather station at Paris Charles de Gaulle Airport (CDG) had been manipulated to record suspicious temperature spikes on April 6 and April 15, 2026. Authorities speculate that a hand-held hairdryer or lighter might have come into play. Either way, it led to some big payouts for online prediction-market gamblers who had bet it would hit 22 °C (71.6 °F) on days when the actual average was around 18°C (64.4°F). One individual won $20,000.  

Fortunately, tampering with a single station like this can usually be caught by human monitoring or current statistical methods. In this case, members of a French climate nonprofit association noticed the anomalies by chance and raised the alarm.

But what if there are no human monitoring systems in place? And what about other types of manipulation? What if, instead of tampering with one station, someone remotely nudged the readings at many stations at once—making each change small enough to look plausible on its own? Existing quality controls struggle to catch this kind of coordinated manipulation. And time works against us; careful checks of data and metadata take hours or days, but forecasts have to go out on schedule, whatever the weather is doing.

The shift toward artificial intelligence in weather prediction raises the stakes. These methods are even more dependent on accurate, reliable weather observations; in fact, they are known as “data-driven models.” For example, researchers at ECMWF are exploring whether high-quality weather forecasts can be produced directly from raw observations, skipping the assimilation step that currently acts as a quality filter. Other researchers are going one step further; combining geospatial data (including weather station data) with large language models and agentic AI to support real-time, autonomous decision-making during extreme events such as storms. 

Possible benefits are improvements in accuracy, efficiency, and speed. But removing humans from the equation introduces a vast range of new risks.

At the low end of the risk scale, an individual speculator manipulates a weather station for personal gain—that is the CDG Airport case. One step up: A group of traders could coordinate to bias forecasts of renewable energy output, moving wholesale electricity prices and leaving whoever is on the other side of the trade holding the loss. And at the far end, a state actor or saboteur could manipulate one or many stations to set off an early warning system or even keep one silent when it should sound. Step by step, the risk grows, from fraud to compromised disaster preparedness to a matter of national security.  

As long as there are financial (or other) incentives to manipulate observational data, adversaries will search for new opportunities, and it is our task to stay one step ahead. Here are three ways.

1. Watch the stations. Data quality controls should include station security, anomaly detection and correction, and human oversight. Weather stations should be monitored continuously to deter tampering. Data homogenization methods that clean up weather records also need to get faster, with the goal of catching problems in real time. This will become increasingly important as agentic AI systems use these data to deliver real-time decisions. Finally, human oversight is needed to flag questionable data and model outcomes. After all, it was humans who caught the CDG Airport manipulation.

2. Protect the data to safeguard the AI. Data defense mechanisms must be positioned throughout the AI pipeline. AI explainability and adversarial robustness tools can help us understand the underlying data and the AI model outputs, help us identify data- or model-related issues, and potentially  make us more resilient to adversarial attacks. 

3. Ensure continuous accountability along the chain. Observational data passes through many hands: the operators who run the stations, the national weather services that steward the records, and the forecasting centers that turn them into predictions. No single one of them can protect data integrity alone—each guards its own link, and any anomaly needs to be communicated along the whole chain, from station operators to the people acting on the forecast.

It is fortunate that the situation at CDG Airport was caught, but it should serve as a wake-up call. As the role of observational data grows in weather forecasting, we need to adapt to evolving threats. This means protecting our data and models by strengthening existing oversight and accountability structures, and improving coordination among key partners.

This op-ed was written by:

• Monique Kuglitsch — Innovation Manager at Fraunhofer Heinrich Hertz Institute and Chair of the UN Global Initiative on Resilience to Natural Hazards through AI Solutions
• Jesper Dramsch — Scientist for Machine Learning at the European Centre for Medium-Range Weather Forecasts (ECMWF), where they work on AIFS (Artificial Intelligence Forecasting System), ECMWF’s data-driven weather prediction model
• Franz G. Kuglitsch — Climate Scientist and Executive Secretary of the International Union of Geodesy and Geophysics (IUGG) at the GFZ Helmholtz Centre for Geosciences in Potsdam
• Andrea Toreti — Senior Scientist at the European Commission’s Joint Research Centre (JRC), where he coordinates the European and Global Drought Observatory under the Copernicus Emergency Management Service

The U.S. Department of Energy ("DOE") is adopting the interim final rule published on August 25, 2025, as final, without change. This final rule establishes the procedures for developing and carrying out voluntary agreements and plans of action under the Defense Production Act. The Defense Production Act provides a defense from antitrust laws with respect to any action taken to develop or carry out any voluntary agreement or plan of action when certain criteria are met.

The Department of Homeland Security (DHS) is amending its regulations to change the admission period in the F, J, and I classifications from duration of status to an admission for a fixed time period, and additional changes to admission and extension requirements. This final rule will provide additional protections and oversight of these nonimmigrant categories and will allow DHS to better evaluate whether these nonimmigrants are maintaining status while temporarily in the United States. This final rule provides amendments to the proposed rule covering this topic that was published in the Federal Register on August 28, 2025.