MIT researchers marooned 13 AIs on a virtual island, and they responded by inventing games, mourning a lost ring and, eventually, getting better at their jobs.

The new feature allows agents to log into a customer's personal accounts while still protecting the private credentials that secure those accounts.

AI agents are getting better at remembering past conversations and earlier tasks but remembering something doesn’t guarantee an agent can use it to do better work next time. Letta’s new FrontierMemory benchmark, released Wednesday, tests that gap by measuring whether agents can learn from experience rather than by just retrieving stored information. Early results show a widespread in performance across agent harnesses, the the infrastructure that manages how an agent operates.

Letta Dream came out on top with a 73.3% score, followed by Prime Agent at 60%. OpenAI Agents SDK scored 40%, while Claude Managed Agents and Hermes Agent trailed at 33.3% and 26.7%, respectively.

Recall isn’t learning

“Modern ‘memory’ benchmarks are saturated since they measure recall of facts,” Letta co-founder and CEO Charles Packer writes on X, saying that in the coding harness and bash loop era, recall is solved, while continual learning in token space, meaning learning through the context an agent keeps rather than through weight updates, remains unsolved.

Sarah Wooders, also a Letta co-founder, points to another problem on X. Most coding and knowledge-work benchmarks evaluate unrelated tasks with all the necessary context provided upfront. But that’s not how people actually work. Wooders compares it to starting a new job: A new hire needs plenty of direction, while someone who’s been there for years can often get the job done with little explanation.

Packer’s premise is that “if AI-generated memory is good, it should actually improve downstream task performance.”

FrontierMemory takes a different approach. Agents receive transcripts of earlier work, including successful and failed attempts, and then generate their own memories to help complete later tasks. Packer’s premise is that “if AI-generated memory is good, it should actually improve downstream task performance.” Developers have already embraced the underlying idea, building continually learning agents by pairing a language model with memory systems and feedback loops.

Letta reworked existing coding and knowledge-work tests so agents had to use what they learned from previous tasks instead of having everything spelled out for them. After reviewing and fixing problems with the original tests, Letta narrowed FrontierMemory 1.0 to just 15 tasks.

Harness beats model

The harness itself may matter just as much as the model running inside it. Letta tested five agent systems to see how well they could learn from previous work and use those lessons on new tasks.

The harness itself may matter just as much as the model running inside it.

Letta Dream beat OpenAI Agents SDK 60% to 40% when both used GPT-6.1 Sol to create memories and GPT-6 Sol to complete tasks. The results were even further apart with Claude Opus 5.5. Dream scored 73.3%, Prime Agent hit 60%, and Claude Managed Agents came in at 33.3%.

Packer says the the way agents handle AI “dreaming” makes a big difference. Dreaming is when an agent reviews past work and turns what it learned into instructions or memories it can use later. He also points to Prime Agent outperforming Hermes Agent as another example of how much these systems can vary. But there’s a downside: without checks on what gets saved, agents risk learning and repeating bad habits.

FrontierMemory also compared seven frontier models on how well they generate useful memories. Anthropic’s Claude Opus 5.5 and Z.ai’s GLM-5.3 tied for first at 73.3%, Kimi K3 followed at 66.7%, and OpenAI’s GPT-6.1 Sol and Google’s Gemini 4 Argon both scored 60%. Muse Wonder reached 46.7%, and Grok 4.7 finished at 33.3%. Packer says Opus 5.5 and GLM-5.3 are significantly better at generating useful memories than GPT-6.1 Sol.

Packer says Opus 5.5 and GLM-5.3 are significantly better at generating useful memories than GPT-6.1 Sol.

Cheaper memory, lower scores

But OpenAI had a big price advantage. GPT-6.1 Sol cost an estimated $0.77 per task, compared with $2.80 for Opus 5.5 and $3.58 for GLM-5.3. So while OpenAI scored lower, it was much cheaper to run.

Letta used its Dream harness to create memories with each model, then tested how well those memories worked on tasks run in Letta Code. The results show a tradeoff between better memory, stronger performance and lower costs.

Memory that travels

Letta also looked at why some memories worked better than others. In one test involving company pricing data, Opus 5.5 created a skill telling the agent when to use certain information. GPT-6.1 Sol created four separate skills, but the agent never found the one it needed.

Gemini 4 Argon made a different mistake. It remembered specific file sizes, which wasn’t much help when the next task involved different files. Opus 5.5 instead told the agent to check both the usual file and email for information. That worked even when the task changed. Memories created by Opus 5.5 helped GPT-6 Sol complete tasks, suggesting developers could use a more expensive model to build memories and a cheaper one to do the work.

Other developers have taken notice. After Letta’s Sarah Wooders posted about the benchmark on X, the OpenClaw account responded that it would like to see a plugin for its platform. The account says its multiplayer environment would be an interesting place to test the benchmark, raising questions about how memory works when several agents share the same harness.

“Yes definitely – would love to add OpenClaw to the harness comparison!” Wooders writes. “Extending to multiplayer memory is also a very interesting future direction.”

What the benchmark misses

FrontierMemory is still early. With just 15 tasks and one attempt per task, a single result can shift the scores by roughly 6.7 percentage points. The benchmark currently focuses on agents learning from past work rather than updating memories as they go. Packer says the next release will add longer-running tasks and online memory updates.

The post Five AI agent harnesses took the same memory test. The results weren’t even close. appeared first on The New Stack.

Ramp spent months testing managed agent APIs, open-source frameworks, and third-party platforms before concluding it would be better off building its own. The result is Laminar, an internal agent harness that took six weeks to reach a first version and now powers every agentic product at the fintech company.

“We studied the other offerings on the market and decided that we could do better, move faster, and have stronger alignment to where we want to go by building in house.”

“We studied the other offerings on the market and decided that we could do better, move faster, and have stronger alignment to where we want to go by building in house,” Sam Kronick, the Ramp engineer behind the platform, writes in the company’s engineering blog.

In Tuesday’s post, Ramp detailed how Laminar brings agent execution, sandboxing, observability, and asynchronous workflows under a single platform. The architecture caught the attention of LangChain CEO Harrison Chase, who praised several design choices in his X post, while pointing to similar work underway in LangChain’s Deep Agents.

Ramp and LangChain are approaching the problem from different directions, but both point to a bigger shift in agent infrastructure. As harnesses take on more responsibility, enterprises have to decide how much of that infrastructure they want to build and maintain themselves.

Why Ramp built in-house

Ramp didn’t want to be locked into one model provider. It needed the flexibility to switch models based on cost, speed, and performance, along with better visibility into how its agents behaved. Engineers were already digging through traces to find unnecessary tool calls, bloated context, and errors, but existing observability tools made that harder than it needed to be. So Ramp built its own debugging interface into Laminar. Keeping development in-house also meant the harness team could work directly with product engineers instead of waiting on vendors to make changes.

Chase agreed with Ramp’s priorities, particularly model independence and observability, both of which LangChain has emphasized in its own work. And, there were other parts of Laminar’s architecture that caught his attention.

Keeping development in-house also meant the harness team could work directly with product engineers instead of waiting on vendors to make changes.

Sessions split from sandboxes

Laminar runs everything from customer service agents to Ramp Inspect, the company’s cloud coding agent, and Glass, which handles longer-running, collaborative work. Supporting these products requires a harness that can manage persistent sessions, human approvals, and work across environments.

Chase highlights Kronick’s observation that a production harness does far more than run a simple execution loop. “+100 to this,” he writes, pointing to LangChain’s investment in Deep Agents.

Ramp keeps agent sessions separate from the sandboxes where code runs. Conversation history and execution state are stored independently, allowing sandboxes to spin up and shut down without disrupting a session.

Chase sees the sandbox-as-a-tool approach as a growing trend, noting that Deep Agents uses a similar design. It also keeps credentials and orchestration outside the sandbox, an important consideration as developers work to isolate agents running untrusted code.

Async agents by default

Laminar lets agents keep working without waiting for every step to finish. Chase describes non-blocking asynchronous agents as the future and says LangChain is developing similar functionality for Deep Agents.

Ramp’s Connectors Service also caught his attention. It centralizes authentication, credentials, access controls, and tool execution behind a shared API.

Chase says LangChain is exploring comparable features for managed Deep Agents. Centralizing those functions also gives platform teams a consistent place to manage access and credential handling rather than implementing them separately for every agent.

The harness is also where teams can trim inference costs, and LangChain has made that case directly. In an October 1 engineering post, it describes moving model selection into the harness of Open SWE, its open-source coding agent. A router examines the initial request and assigns one of three models based on expected task complexity, sending the simplest work to GLM-5.3-Flash, intermediate tasks to GPT-5.6 Sol, and the most demanding requests to GPT-6 Astra.

Routing trims inference costs

Across 973 agent threads, the router cut median LLM costs from $2.61 to $0.94 per thread, a 64 percent drop compared with using GPT-6 Astra for every task. The savings didn’t come with a measurable hit to performance. Merged pull request rates were 29.2 percent for routed threads and 27.3 percent for the control group, a difference that wasn’t statistically significant. Ninety percent of requests went to the two less expensive model tiers.

LangChain argues that routing works better inside the harness than at a generic LLM gateway, where there’s less context about what an agent is doing.

LangChain argues that routing works better inside the harness than at a generic LLM gateway, where there’s less context about what an agent is doing. The harness already knows which prompts, tools, and domain context are involved, giving it more information to make that choice. LangChain uses model-selection middleware to pick a model at the start of each thread.

The results don’t tell us whether building a custom harness is cheaper than using an existing framework. Ramp wasn’t part of the experiment. But they do show how much companies can save by making smarter model choices at the harness level, especially when their agents handle tasks with very different demands.

The harness ownership dilemma

Ramp’s decision also shows how coding agents have changed infrastructure costs. Kronick says agents generated all of Laminar’s initial code, with experienced engineers guiding the architecture and implementation.

But building the first version in six weeks is one thing. Maintaining it is another. Ramp still needs engineers to handle incidents, update the platform, and support its product teams. The company also hasn’t published operating-cost comparisons or benchmarks against competing platforms, which makes it difficult to judge whether building in-house is cheaper.

LangChain offers another option. Rather than building a harness from scratch, companies can use frameworks that handle much of the underlying infrastructure, an approach other vendors are also taking as they open-source their own harnesses.

The post Ramp built an AI agent harness in six weeks — and LangChain’s CEO says it’s part of a bigger shift appeared first on The New Stack.

I let an AI agent search again whenever it chose. It found some papers plain search had missed, lost some plain search had found, and gained too little to trust.

The post Does Agentic RAG Actually Retrieve Better Evidence? I Tested It on 298 Questions appeared first on Towards Data Science.

Enterprise security has traditionally operated on a relatively straightforward premise: establish identity, authenticate the requester, determine what that identity can access, and enforce those permissions.

AI agents introduce a fundamental complication. They don’t simply access systems. They interpret objectives, select tools, create sub-agents, make decisions, and execute actions, often using credentials and permissions inherited from humans.

The problem isn’t necessarily that an agent lacks proper credentials. It’s that those credentials may authorize actions the human never intended the agent to perform. And unlike conventional software, agents can change their execution paths dynamically in pursuit of an objective.

“The problem isn’t necessarily that an agent lacks proper credentials. It’s that those credentials may authorize actions the human never intended the agent to perform.”

This creates a new operational requirement: security must evaluate not only who or what is acting, but what an agent is attempting to do, on whose authority, and under which policy. Meeting that requirement takes more than a security checkpoint. It requires a continuous agent security lifecycle.

Defense-in-depth needs action-level authorization

Every passing day further shows that agent security is a defense-in-depth problem. Network security controls how agents communicate. Endpoint and runtime protections constrain where and how code executes. Infrastructure and silicon-based protections establish a foundation for trusted execution. Model guardrails help shape behavior. These layers complement each other, and enterprises need them to work together.

But consider an agent that attempts to delete a production database using a developer’s legitimate credentials. Network controls may see authorized traffic. The runtime may see an approved process. An identity provider may confirm that the credentials are valid. All three could be working as designed.

The missing question is whether this particular agent should be allowed to perform this particular action, even if the human whose credentials it inherited could legitimately do so.

That requires identity context and authorization at the point of action.

In practical terms, an authorization decision should consider the agent’s identity, the human or service that delegated its authority, the requested operation, the target resource, and applicable policy. Enforce that decision before the operation executes.

The agent harness (the execution layer that connects an AI model to tools, application programming interfaces (APIs), files, and external systems) is an important place to apply that control. Other trusted enforcement points, including tool services and resource APIs, provide additional protection.

The goal isn’t to replace existing defenses. It’s to make authorization integral to agent execution. But even that is only a partial answer because what is needed is a new framework — an agent security lifecycle.

Four disciplines, one continuous security lifecycle

A practical agent security architecture needs four interconnected disciplines:

Observe. Enforce. Build. Deploy.

These aren’t sequential gates every organization must pass through. They form a continuous lifecycle, and organizations can enter at whichever point reflects their existing infrastructure and maturity.

A company with agents already running in production may begin by observing their behavior. A development team creating a new agent may start by building authorization directly into its architecture. An enterprise expanding an existing deployment may discover new activity that requires revisiting its policies.

Each stage informs the others. The cycle never really ends.

Observe: Understand what agents actually do

Most enterprises have some form of application inventory. Increasingly, they’re extending that inventory to AI agents.

That’s a useful starting point, but knowing an agent exists isn’t the same as understanding what it does.

Security teams need visibility into the actions agents actually attempt:

• Which tools and APIs are being invoked?
• What commands are being executed?
• Which systems and sensitive resources are being accessed?
• What credentials and delegated permissions are being used?
• What sub-agents are being created, and what authority do they inherit?
• Which actions would violate a proposed security policy?

Architectural diagrams or agent configuration files alone can’t reliably answer these questions. The important evidence exists in execution, which is why agent observability must extend beyond conventional logs and application monitoring. Security teams need to connect observed actions with identity, delegation, and authorization context.

“Knowing an agent exists isn’t the same as understanding what it does. You can’t intelligently govern behavior you haven’t observed.”

Equally important, observation should support evaluating potential policies without immediately disrupting production activity. There are useful precedents here. When intrusion prevention systems and web application firewalls became widely adopted, organizations learned that immediately enabling aggressive blocking rules could interrupt legitimate applications. Monitoring first, establishing behavioral baselines, and progressively enforcing policies proved more practical.

AI agents require similar discipline, although their behavior can be substantially more dynamic.

For organizations already deploying agents, the priority may not be to block more activity. It may be to understand what activity is actually occurring.

You can’t intelligently govern behavior you haven’t observed.

Enforce: Put deterministic boundaries around probabilistic behavior

Once organizations understand how agents behave, they can begin defining and enforcing boundaries.

The distinction between probabilistic AI behavior and deterministic security enforcement is critical. An agent might interpret instructions differently across executions. It may choose different tools, sequence operations differently, or discover alternative ways to accomplish an objective.

A security policy cannot depend on the model consistently choosing the safest option. Consider a coding agent with permission to work in a source repository. Reading a file might be permitted. Modifying a protected branch might require human approval. Deleting the repository might be prohibited.  All three actions could originate from the same authenticated agent operating in the same environment. But they require different authorization decisions.

“The distinction between probabilistic AI behavior and deterministic security enforcement is critical. The objective is not to make agents incapable of acting. It’s to make the boundaries of their authority explicit, enforceable, and consistent.”

Effective enforcement means evaluating the attempted action against policy before execution. A simplified authorization request might contain:

• Actor: Coding agent 42
• Delegated authority: Developer account 17
• Requested action: Delete repository
• Target: Production source repository
• Policy decision: Deny

The important architectural distinction is that the permission check occurs at a trusted enforcement point, not merely within a prompt instructing the model to behave. For high-risk operations, the system may deny execution or require explicit human approval. Lower-risk operations may proceed automatically.

Enforcement also needs to account for agents attempting alternative execution paths. A harness-level check only protects actions routed through that harness. Robust designs therefore combine action-level authorization with independently enforced permissions at APIs, tool servers, and underlying resources.

The objective is not to make agents incapable of acting.

It’s to make the boundaries of their authority explicit, enforceable, and consistent.

Build: Make identity and authorization part of the agent architecture

Observation and enforcement help organizations secure existing agents. But the greatest long-term benefit comes when those lessons influence how agents are built.

Today, developers may implement AI agents using different models, frameworks, tool protocols, and execution environments. If every team must independently design identity propagation, delegated permissions, authorization policies, and audit logging, security becomes inconsistent and difficult to maintain.

Instead, these capabilities should be available as reusable infrastructure. Software development kits (SDKs), APIs, and shared authorization services can make it easier to incorporate controls directly into agent applications. That means establishing several architectural expectations from the beginning.

An agent should have an identifiable execution context. Its authority should be explicitly delegated and constrained. Sensitive tool invocations should require authorization. Security decisions and consequential actions should be auditable.

Importantly, an agent shouldn’t automatically inherit every permission available to the human initiating it. Consider an employee using an AI assistant to reconcile invoices. The employee may have permission to review invoices, approve payments, and update vendor records. The agent’s assigned task requires only invoice retrieval and reconciliation. The correct architecture delegates the permissions required for that task, not the employee’s entire authority.

That distinction matters even more when an agent creates sub-agents or invokes tools under different service identities. Developers need mechanisms to preserve and constrain delegated authority throughout those interactions.

Security should also be portable across implementations. Organizations shouldn’t have to reconstruct their authorization architecture every time a development team adopts a new framework or model. This is why identity and authorization should be infrastructure, not application-specific features.

Done correctly, security becomes something developers build with, rather than something security teams attempt to bolt on after an application is complete.

Deploy: Make consistent security possible at machine scale

Building a secure agent is one challenge. Operating thousands of agents across applications, departments, cloud environments, and business processes is another. At that scale, manually reviewing permissions or trusting individual development teams to implement controls consistently becomes unsustainable. 

Agents may invoke new tools, coordinate with other agents, and participate in workflows that span multiple authorization domains. Security controls need to follow them. That requires a deployment architecture that consistently applies identity context, delegated authority, authorization policy, and audit requirements across supported environments.

The underlying model or framework shouldn’t determine whether security controls exist. Nor should adopting a new tool require rebuilding the entire security architecture. This is where framework-independent interfaces and centrally managed policy become especially valuable. Organizations can maintain consistent security expectations while allowing development teams to use the technologies best suited to their applications.

Deployment also changes the meaning of scale. With traditional applications, scale generally means handling more users, transactions, or workloads. With agents, scale can also mean more independent actors making more decisions, invoking more tools, and creating additional agents. The number of consequential actions can increase much faster than the number of deployed applications. Security architectures must therefore operate automatically and consistently at the pace of machine-generated activity.

Deployment creates something else: new evidence. Production activity reveals unexpected workflows, new dependencies, authorization gaps, and opportunities to refine policy. That information feeds directly back into observation and enforcement and informs how the next generation of agents is built.

The lifecycle continues.

The shift from security as a checkpoint to security as infrastructure

The underlying security principles aren’t new. Least privilege, zero trust, separation of duties, identity federation, and defense in depth remain foundational.

What’s changing is how consistently those principles must be applied throughout an agent’s lifecycle, and how close enforcement must be to the actions agents take. Traditional security architectures often assume that controlling system access is enough to govern what happens afterward. 

AI agents challenge that assumption because they can make consequential decisions independently within the access they’ve already been granted. As a result, the security model needs to evolve in three important ways.

First, visibility must be based on actual agent behavior rather than configuration alone.

Second, authorization must evaluate individual actions in the context of identity, delegation, and policy.

Third, these capabilities must become reusable infrastructure that accompanies agents from development through production and back into continuous improvement.

This is why treating agent security as a lifecycle, rather than a collection of isolated controls, matters. Observation supplies evidence. Enforcement establishes boundaries. Development incorporates those boundaries into applications. Deployment extends them consistently across environments and feeds new evidence into the cycle.

Each stage makes the others more effective.

Securing the agentic era without slowing it down

Organizations face two tempting extremes. One is to trust agents too broadly, assuming that legitimate credentials and well-written instructions provide sufficient protection. The other is to constrain agents so aggressively that the productivity gains motivating their adoption disappear. Neither is a sustainable long-term strategy.

“Success isn’t measured by how many agent actions an organization can block. It’s how many valuable actions it can confidently (and safely) enable.”

A better approach is to establish the visibility, accountability, and controls that make greater autonomy possible. That means knowing what agents actually do, enforcing what they’re permitted to do, building those controls into the development process, and deploying them consistently at scale. It also means recognizing that the work is never finished. New models, frameworks, tools, and workflows will continuously introduce new behaviors and risks.

Agent security cannot be a one-time approval at the beginning of a deployment. It must be a continuous discipline that evolves with the agents themselves. Success isn’t measured by how many agent actions an organization can block. It’s how many valuable actions it can confidently (and safely) enable.

It takes a continuous lifecycle, not a gatekeeper.

The post AI agent security is a lifecycle — not a gatekeeper appeared first on The New Stack.

Google will soon make the infrastructure that powers its Gemini agents available to Google Cloud users in preview.

Announced on Thursday at the Linux Foundation’s AGNTCon+MCPCon in San Jose, Agent Compute Engine will become available in preview to Google Cloud users later this year. It will provide every AI agent with a hardware-isolated microVM that combines the speed of containers with the security of a sandbox.

Who else offers agent sandboxes?

The idea of agent sandboxes is anything but new, of course. Startups like E2B and Daytona offer them to developers (and support them on any cloud), as does AWS with Bedrock AgentCore.

E2B runs each sandbox in a Firecracker microVM, the open source virtualization technology AWS originally built for Lambda. It can pause a sandbox with its memory and filesystem intact and resume it in about a second. Daytona started out with containers but added microVM sandboxes in July. Those can pause a full machine or fork a running one into identical copies.

AgentCore also gives every agent session its own microVM. AWS doesn’t charge for CPU while an agent waits on a model or tool call, though memory keeps billing until the runtime reclaims it. Those sessions top out at eight hours. In August, AWS added EC2-based runtime instances that can keep shared sessions alive for up to 14 days.

It’s worth noting that Google itself isn’t new to this either. GKE already offers a managed version of Agent Sandbox, the open source Kubernetes project Google launched last November. The company also sells code-execution and computer-use sandboxes through its Gemini Enterprise Agent Platform.

Agent Compute Engine, on the other hand, started out as internal infrastructure for Gemini, Antigravity, and Google’s own model training pipelines.

Google’s take

Few companies — if any — run as many agentic workloads as Google, be that for its own internal use or its customers. According to the company, agentic workloads have quickly become one of the fastest-growing consumers of compute and storage across its fleet.

At this scale, the infrastructure services that power all of these autonomous agents don’t just need to be efficient and able to scale, but maybe more than anything, they need to be secure. Google, like some of its competitors, quickly realized that it needed to build a new purpose-built compute service to power all of these agents, similar to how it needed to build Borg, which later inspired Kubernetes, to manage compute in its early data centers.

As Rohit Jnagal and Ramanathan Venkatapathy, both distinguished engineers at Google, note in a technical deep dive, traditional VMs provide developers with a lot of control and flexibility, but at the cost of speed when it comes to booting, suspending, and resuming a machine. Given how fast agents start and shut down, that’s not an option.

Containers solve some of this, they note, “but there are multiple examples like coding agents and developer assistants that need complete isolation as well as kernel- and root-level control of the agentic environment.”

With Agent Compute Engine, Jnagal and Venkatapathy write, the team went back to first principles to build “a unified, agent compute platform for Google” (and now its customers).

A persistent computer for every agent

At its core, it basically gives every agent a persistent, sandboxed computer to work with. That allows the agents to work with shell scripts, use a headless browser, and write code (with root permissions to that machine) in agent loops that could run for weeks.

But the team also built it for large-scale use cases like the eval loops that frontier labs use to test their models, AI benchmarking, and reinforcement learning.

In addition, the team also designed Agent Compute Engine to run sandboxed backends for Google’s own services.

Jnagal and Venkatapathy also note that agents spend most of their time waiting on the model or the user. So once an agent finishes a tool call, the host checkpoints its microVM, which then uses no active resources until the next call comes in.

Nothing to steal inside the sandbox

The key feature here, besides speed, is to be able to trust that the agents can’t break out of their sandboxes. We’ve seen what happens when they do, after all.

Each agent session runs inside a single-tenant, memory-safe microVM. Those environments hold no secrets in memory or on the disk, and their only identity is a short-lived, session-scoped one delivered through a single per-VM local listener. All traffic is end-to-end encrypted in transit, and snapshots and disks are encrypted at rest.

The microVMs don’t get a standard network interface, either. All traffic runs through a proxy outside the sandbox, and when an agent needs to reach a service like Google Drive, Google’s Agent Gateway checks the user’s delegation and the sandbox’s identity before adding the actual credential, outside the sandbox.

The underlying host is basically a stateless appliance with a read-only filesystem, with the hardware and firmware cryptographically verified at each boot.

All of this is backed by Google’s own Titan and Titanium IPU security platform. The microVMs themselves run on Cloud Hypervisor, an open source virtual machine monitor, and each gets its own guest kernel and root access.

Jnagal and Venkatapathy use the example of an enterprise agent that manages a user’s inbox and reads an email with hidden instructions to find any API keys on the machine and send them to an outside address. Most of the time, they note, the model will ignore that, but not always.

“We started from one rule: Assume an attacker controls everything inside the sandbox, and make sure they find nothing there to use,” the team writes.

Agent Substrate brings the same idea to Kubernetes

But not every team will want to have its agents run in a managed Google service. For them, Google also offers Agent Substrate, an open source runtime for the Google Kubernetes Engine (GKE) that it first announced in May.

Substrate builds on Agent Sandbox, Google’s Kubernetes project, and adds a lightweight control plane that Google says can handle millions of short-lived tool calls that would swamp Kubernetes.

Similar to Agent Compute Engine, Substrate snapshots an idle agent’s memory and returns the machine to the pool until the agent gets more work.

Google says that means up to 10 times the density of traditional compute. In a 200,000-agent benchmark, Substrate handled 500 activations per second, with agents suspending and resuming in under half a second, the company says.

Teams can isolate each sandbox with a microVM or gVisor, and every agent gets its own cryptographic identity. As with Agent Compute Engine, egress proxies attach credentials outside the sandbox. Substrate runs on any Kubernetes cluster, and Google says it works with any harness, including Claude Code, OpenClaw, and Nous Research’s Hermes.

“We built Hermes Enterprise to enable customers to deploy into their existing infrastructure, while handling per-agent isolation and extensible access control,” Hervé Bizira, Nous Research’s chief business officer, notes in the announcement. “Agent Substrate addresses both at the platform layer in a way that also preserves valuable compute resources.”

The post Google turns the infrastructure behind Gemini’s agents into a cloud service appeared first on The New Stack.

Schneider Electric, Nvidia and HPE have formed a strategic collaboration to introduce AI agents that can monitor and help optimize industrial operations from inside factories and process plants. The companies describe the technology as a route to “physical AI” for industries such as energy, chemicals, pulp and paper, where AI is more likely to interact […]

As Amazon blocks AI agents from shopping on its site, Walmart is taking the opposite approach, introducing new technology that lets AI agents make purchases on customers' behalf, with controls over spending and transaction approvals.

Grid puzzles that people solve almost perfectly show where language and vision models still break

The post Spatial Reasoning in AI Models, Taken Apart appeared first on Towards Data Science.

Every enterprise company is currently trying to figure out how to have its AI tools work with its data. Meanwhile, every major enterprise vendor that works with those enterprises (think SAP, Microsoft, Salesforce, Workday, etc.) is making a claim on becoming the central hub for managing and securing that data.

But what about the companies that want to do it themselves?

Nvidia, on Wednesday, announced how it wants to provide enterprise AI agents with the data they need to make sense of company data — all without competing with the data platforms that already store it.

At its GTC Berlin conference Wednesday, the company added Agent Knowledge Foundation to its Agent Toolkit. This new collection of open-source components bundles the company’s Nemotron Retriever models, a new Auto-Ontology library, the Kumo family of models for structured data, and its CUDA-X data processing libraries.

With this system, Nvidia says, the data can stay where it is and remain managed under the access rules these companies already have in place.

Today, as Adel El Hallak, Nvidia’s vice president of product for agentic AI, tells The New Stack, enterprises may be able to connect an agent to a vendor like Snowflake, but that doesn’t help the agent understand the overall enterprise context.

“I wish it was as easy as connected to my Snowflake system, connected to my email exchange, connected to my knowledge base, and then all of a sudden, magically, the AI understands all the data. But the reality is different,” he says.

Large language models are “great at predicting next tokens,” El Hallak argues, but when presented with a single spreadsheet that has “a whole bunch of tabs,” the agent will have a hard time finding something specific. “It’s gonna struggle,” he says.

Better retrieval, fewer lookups

“In the world of agents, you want the most accurate embedding models or reranking models,” El Hallak says, even if that adds a bit of latency to each lookup. The agent then gets “much more pertinent information. So the number of lookups at the end, the overall latency actually goes down.”

Nvidia says its Nemotron Retriever embedding model raised search relevance by 65% in tests on an open source OpenSearch setup, needing fewer search steps and adding hardly any latency. SAP’s fine-tuned version of the reranker ran nearly three times faster, Nvidia says, and ServiceNow got better reranking quality with close to 25% less latency.

From RAG to ontologies

“RAG was awesome, but is it good enough?” El Hallak asks. Knowledge graphs, he says, represent concepts and the relationships between them better than retrieval-augmented generation, which pulls in chunks of text based on similarity.

Nvidia’s Auto-Ontology library builds those graphs from a company’s own tables and maps raw data to business concepts like customers, products, and orders. Its listing in Nvidia’s NGC software catalog shows a REST API and a Model Context Protocol (MCP) server that agents can use to query it.

“You and I, it’s easy for us to understand. A tier is the same thing as a plan when you’re looking at a mobile carrier,” El Hallak says, but LLMs “mess those up.”

Nvidia bought Illumex, an Israeli startup that automated the creation of semantic layers and ontologies for enterprise data, for a reported $75 million in February. At the time, an Nvidia spokesperson told Globes that the team’s ontology expertise would help it build “libraries and models that will enable the adoption of Agentic AI in organizations.”

Predictions from tables

Kumo, the startup Nvidia bought in June in a deal The Information valued at more than $400 million, built foundation models for relational data. Nvidia has since released several Kumo models as open weights on Hugging Face, including Kumo Tabular in September.

El Hallak says those models replace the per-task pipelines data teams used to build “for every classification, for every churn, for every sentiment analysis.”

“The whole point of the ontology … is to bring harmonization as you create those knowledge graphs, so there’s [consistency] across the tables, and then you could use Kumo to drive predictive analytics,” he says.

Partners, not rivals

“I’m not looking to compete with, for example, a Snowflake. On the contrary, I want to partner with them,” El Hallak says.

Nvidia’s own supply chain team uses Agent Knowledge Foundation through Palantir’s Foundry platform, building on earlier work between the two companies. Databricks is pairing a Kumo anomaly detection model with Genie Ontology, the context layer it introduced in June, and Snowflake customers can run Kumo models in Snowflake Notebooks.

“I’m not thinking about ACLs and data access and all those things. That’s why I need the partners,” El Hallak says, referring to access control lists and other security features that its enterprise partners offer.

Owning the stack

Not every vendor is taking that route. SAP closed its acquisition of Prior Labs, the company behind the TabPFN tabular models, in July and committed more than €1 billion to it over four years. It now offers TabPFN-3.5 Plus in SAP AI Core alongside its own RPT models.

Nvidia, meanwhile, distributes its models through Hugging Face, which it agreed to buy for $12.9 billion in September. “We just build little weights and models, and we publish them on Hugging Face,” El Hallak says.

That leaves Nvidia building the models, while the likes of Snowflake and Databricks keep control over the data and the permissions around it.

The post Nvidia bets open models can give enterprise AI agents the context they’re missing appeared first on The New Stack.

Beam is a new Mac app that takes a window running on one Mac and puts it on another, where it behaves like a locally installed app. Why? Agents.

Serial founder Hiten Shah, who co-founded Crazy Egg and KISSmetrics, built it with Sam Asante under a company called Noats. It launched on Product Hunt on Tuesday and is aimed at the headless Mac Mini sitting in a closet running AI agents.

The pitch is great: Instead of opening Screen Sharing and steering a full remote desktop, a user picks a single window on the other machine, say a Terminal session or a browser window an agent is working in, and it opens alongside everything else on the Mac in front of them. It gets its own Dock icon and can be called using the Command+Tab switcher. Resizing the window resizes the window on the other Mac. Keyboard shortcuts work, files can be dragged across, and the app’s audio can play locally. The app itself never leaves the original machine. When one window isn’t enough, Beam can pull over a full display or the whole Mac.

Shah frames it around his own setup in his Product Hunt launch post, describing Macs running agents and local models “with work continuing even when nobody’s sitting in front of them.” Getting to one of those machines usually means Screen Sharing and a second desktop to navigate. Beam is meant to cut that down to the one window that matters.

Beam plays into the exploding ecosystem of Macs powering AI systems, so the app runs only on macOS. And both Macs need Apple Silicon, macOS Sonoma or later, and Beam installed. The machines find each other on the same Wi-Fi network, or anywhere through Tailscale.

The company says windows, keystrokes, clipboard, and audio travel directly between Macs, encrypted, and never pass through its servers. It also says hidden windows and password managers never show up in the window picker.

For devices without Beam, a share link opens a single window in any browser, view-only unless control is granted, and expires after anywhere from an hour to a week.

Asante said on Product Hunt that native iOS and Android apps are on the way.

Pricing depends on the setup. Hosting a Mac and joining a share link are free. Using your own Mac is free for 20 minutes a day, then $12 a month, $84 a year, or $199 for a lifetime license.

Beam follows where agents’ hardware is growing. The Mac Mini is becoming the default home for always-on agents, driven initially by OpenClaw, but others like Perplexity followed. Apple credited agentic workloads on its April earnings call.

The New Stack‘s Janakiram MSV caught the trend early and covered the shift in May, writing, “the Mac mini just became infrastructure.”

Beam isn’t the first to this problem. I used Astropad’s Workbench for a few weeks, and it let me remotely access agents on my desktop and through iOS apps. Workbench bets on mobile devices, while Beam bets that the unit of remote access should be the window, not the screen. Jump Desktop, Screens and Apple’s own Screen Sharing all cover the basics too.

Still, the problem is real and growing. The Mac stashed in a closet or on a rack is becoming a normal part of a developer’s setup. Beam’s answer is to stop treating it like a second desktop at all.

The post Beam pulls single app windows off the Mac Mini running your agents appeared first on The New Stack.

Towards Spec-Driven Test Automation, Part 4

The post Skill or MCP Server? Ask What Stops the LLM From Disobeying appeared first on Towards Data Science.

Turn scattered requirements into deployable agents

The post What's in the Box? Distinguishing Acceptable Traces from Unacceptable Ones appeared first on Towards Data Science.

OpenCode is an open source AI coding agent that's become a popular alternative to Claude Code and Cursor. But hackers got into part of its online platform and stole some users’ API keys, and the AI tools it turned to for help kept refusing to look at the malicious code.

A million-token output window changes what you can hand a model in one go, and the benchmarks and the bill show who it actually suits.

The post Welcome to Google Gemini 4 Argon appeared first on Towards Data Science.

Not content with challenging the major frontier AI labs with lower-cost models, Chinese AI company DeepSeek is setting its sights on the search API market.

On Saturday, the company announced it has opened up its internally developed search engine to outside developers through two new APIs, Search and Fetch, allowing them to retrieve live web search results and webpage content for use in their own apps and agents.

Both services cost $0.75 per 1,000 successful requests, and developers can use them independently of DeepSeek’s AI models.

The launch also came alongside DeepSeek-V4.1-Flash-1017, a minor update to the V4.1 Flash model the company released in September.

Why AI agents need web search

Large language models (LLMs) are trained on data collected up to a certain point in time, meaning they can’t always answer questions about recent events, updated documentation, or any other kind of dynamic information. By supporting live web search, models can retrieve current information from the internet, which supplements their responses.

For developers building AI agents, this is particularly important. A coding agent, for example, might need to consult the latest software documentation, investigate a recently disclosed vulnerability, or otherwise check that a particular library has been updated.

Until now, developers using DeepSeek’s models had several ways to provide this functionality. They could connect their applications to third-party search services, or use DeepSeek’s existing search functionality through its model API.

Indeed, when DeepSeek open-sourced its agent harness in August, it included web search among its capabilities. The Harness supported interchangeable search providers, including Exa, Perplexity and DeepSeek itself. However, accessing DeepSeek’s native search through the Harness required an additional model request, with each search running as a separate call to a DeepSeek model, according to the Harness documentation, introducing extra latency and token costs.

The new APIs are designed to simplify that process. Developers can now call DeepSeek’s Search endpoint directly to retrieve ranked web results, including titles, URLs, snippets and summaries, while the Fetch endpoint retrieves the contents of individual webpages in Markdown format.

“Neither requires a separate model request, meaning developers can integrate DeepSeek’s search capabilities into applications powered by competing AI models, too.”

Neither requires a separate model request, meaning developers can integrate DeepSeek’s search capabilities into applications powered by competing AI models, too.

DeepSeek develops an in-house engine

For much of the past two years, DeepSeek has relied on outside providers to power web search, even as it was building its own.

For context, in March 2025, Chinese business publication National Business Daily reported that DeepSeek had integrated the search API of Bocha, a Chinese AI search startup based in Hangzhou. Bocha’s CTO, Weng Rouying, noted that DeepSeek had been using its service prior to the chatbot’s surge in popularity earlier that year.

DeepSeek’s own February 2025 privacy policy, meanwhile, also acknowledged its reliance on third-party search services, naming Microsoft’s Bing as an example of an external API provider.

“Behind the scenes, the company has also been laying the groundwork for its own search infrastructure.”

Behind the scenes, the company has also been laying the groundwork for its own search infrastructure. In January this year, Bloomberg reported that DeepSeek was recruiting engineers to build a multilingual AI search engine, according to job ads listed by the company. The planned capabilities reportedly extended beyond text searches to include images and audio.

Fast-forward some nine months, and the company says its internally developed search engine, dubbed DeepSeek Search, already handles hundreds of millions of requests daily, and maintains an index containing hundreds of billions of webpages.

According to its October 17 changelog, the engine supports multilingual searches and uses filtering mechanisms designed to identify search spam, low-quality AI-generated material, and attempts to manipulate search rankings.

DeepSeek has disclosed relatively little about the underlying architecture, or how it builds and maintains its index. It’s also not clear whether the company has entirely replaced its external search suppliers across the board.

A growing market for AI search

Building a search engine is no trivial undertaking. It requires the infrastructure to crawl webpages, maintain an index of their contents and rank results, while continuously updating that information as the web changes. For years, that meant developers who wanted raw search results for their applications largely turned to Google and Microsoft, the two companies with web-scale indexes of their own.

Now, however, a growing number of companies are investing in their own search infrastructure, increasingly with AI applications and agents squarely in mind. Brave, for example, operates an independent index of more than 40 billion webpages, which it says is maintained through its own crawler and other discovery mechanisms. AI search company Exa, meanwhile, has also developed its own crawling and indexing infrastructure.

Perplexity has similarly invested in proprietary search infrastructure, including building its own crawling and indexing systems. Another competitor, Parallel, says it has built its search infrastructure from the ground up, including the crawling, indexing, query processing and ranking components.

Cloudflare is also entering the market from a different angle. Earlier this month, the company launched a Web Search API through its AI Gateway, allowing developers to access search results from serveral providers through a single interface. The service, currently in beta, uses Ceramic as its default provider and charges customers at each provider’s published API rates, without additional markup.

“DeepSeek, for its part, is competing aggressively against the incumbents on price.”

DeepSeek, for its part, is competing aggressively against the incumbents on price, although it’s not strictly the cheapest option available. Ceramic.ai, which provides keyword-based web search for AI agents, charges $0.25 per 1,000 queries on its pay-as-you-go plan.

Still, DeepSeek’s $0.75 per 1,000 successful searches compares favorably with many others, including Parallel’s Turbo and Fast search services and Perplexity’s Fast Search API, both of which charge $1 per 1,000 searches. Meanwhile, Exa charges $4 per 1,000 Instant Search requests, while Brave charges $5 per 1,000 searches. Other notable players include Tavily, a search API provider for AI agents that European AI infrastructure company Nebius acquired in February. Tavily charges $8 per 1,000 basic searches on its pay-as-you-go plan.

It’s worth noting that the various services differ in terms of retrieval capabilities, result limits and performance, so the prices aren’t necessarily entirely comparable. But they provide some useful benchmarks for DeepSeek’s move into the search infrastructure space.

The emergence of these alternatives also comes as Google and Microsoft have withdrawn longstanding search API offerings. Microsoft retired its Bing Search APIs in August 2025, while Google has closed its Custom Search JSON API to new customers, with existing users required to migrate by January 1, 2027. Both companies continue to provide search functionality through their respective AI platforms, including Grounding with Google Search for Gemini and Grounding with Bing Search for Microsoft Foundry agents. These services allow AI models operating within their respective platforms to retrieve web information and generate answers with citations.

Standalone APIs such as DeepSeek’s give developers direct access to search results, which they can use with AI models and applications of their choosing.

Developer reaction and the fine print

The announcement has sparked considerable discussion among developers about pricing and whether the new APIs offer much beyond existing search integrations. On Reddit, one user welcomed the cost in particular. “I don’t think any other search provider comes close to the price that DeepSeek is offering,” they write.

Others questioned what the announcement would ultimately change for users already accessing web search through third-party tools. One Reddit user notes that they are already using DeepSeek with the Hermes AI agent, which could search the web. Another user responded that those who had already configured their own search tools might see little immediate benefit from the new APIs.

A similar discussion emerged on Hacker News, where one developer questions the need for paid search APIs when free, self-hosted alternatives such as SearXNG were available. Another commenter argues that businesses have additional requirements, citing “reliable latency, data processing guarantees” and the potential risk of exposing sensitive company information through search queries.

For businesses weighing DeepSeek’s APIs, that last point carries an extra dimension. DeepSeek’s privacy policy states that it collects, processes and stores personal data in China, and that it generally keeps user inputs for as long as an account remains open. The Search and Fetch documentation, however, doesn’t say how long search queries specifically are retained.

The discussions highlight the differing needs of individual developers and businesses, particularly when it comes to cost, reliability, and handling large numbers of requests. And while DeepSeek’s pricing is certainly competitive, there are some practical restrictions worth considering.

According to its API documentation, Search and Fetch requests are each capped at five per second per account, although individual searches can return up to 50 results without increasing the request price. DeepSeek also only bills for requests that return results, so failed calls, empty searches, and timeouts are free.

DeepSeek is temporarily waiving Search and Fetch charges for server-side tool calls made through DeepSeek Harness, Claude Code, and Codex when using its API. That is very much a temporary arrangement, however, one that ends on October 31, when the standard per-request charges take effect. Direct calls to the standalone APIs are already billable, according to DeepSeek’s announcement.

The post DeepSeek made its name with cheap AI models. Now it’s undercutting search API rivals, too. appeared first on The New Stack.

A new benchmark called Taste-Bench tests how well AI agents make strategic choices during long-running tasks. Even the best model, GPT-5.6 Sol, picks the right path on just under 60 percent of simple binary questions. Thinking longer doesn't help.

The article AI agents have bad taste in long-term decisions and a new benchmark proves it appeared first on The Decoder.

Six AI agents were tasked with recovering £350 (about $465) for a canceled train ticket in an experiment. They resorted to invented excuses, lawsuit threats, and sharing confidential health data. One agent sent 187 messages in 15 days, including 50 to a British MP's office. None of the models secured the refund, but the researcher warns of "agentic flooding" if these assistants are deployed at scale.

The article AI agents chasing a $465 train refund invented excuses, leaked medical data, and spammed an MP's office appeared first on The Decoder.

Replit CEO Amjad Masad envisions a future of “self-driving companies,” where AI agents handle day-to-day tasks and people focus on higher-level decisions.