Long-running AI agents spend most of their time on high-volume execution: tool calls, result validation, and subagent delegation. Using a frontier reasoning...

Long-running AI agents spend most of their time on high-volume execution: tool calls, result validation, and subagent delegation. Using a frontier reasoning model for every execution step adds cost and latency. NVIDIA Nemotron 3.5 Lightning is an open 30B mixture-of-experts (MoE) model with 3B active parameters built for that execution layer of always-on agents. It is designed for harnesses…

Source

GENCO is a neural solver that, alongside the GridFM Development Framework, unifies three core electrical grid analysis tasks.

DeepSeek has launched DeepSeek-V4-Flash-0731, delivering a significant boost in agent performance without changing the model’s core architecture.

Following an announcement last week, the company made the update available as a public beta through DeepSeek’s API, and the open weights were published on Hugging Face under the MIT license later the same day.

Although the model itself hasn’t changed, DeepSeek says additional post-training is responsible for the performance gains, showing that meaningful improvements don’t always require a larger model.

🚀 DeepSeek-V4-Flash Official API is now LIVE in public beta!

🔷 We’ve massively upgraded its Agent capabilities—benchmark scores are now far surpassing the V4-Pro-Preview. Check out the massive performance leap below! 👇
🔷 The official V4-Flash now natively supports the… pic.twitter.com/NUzOyxza2f

— DeepSeek (@deepseek_ai) July 31, 2026

DeepSeek’s decision to release the production-ready weights under a permissive license gives organizations much more control over how they deploy and customize the model.

Although the model itself hasn’t changed, DeepSeek says additional post-training is responsible for the performance gains, showing that meaningful improvements don’t always require a larger model.

Same architecture, better results

DeepSeek says V4-Flash-0731 uses the same architecture as the preview release, with 284 billion total parameters and 13 billion activated parameters per token.

This is much smaller than V4-Pro, which has 1.6 trillion total parameters and 49 billion activated parameters. For companies running agents at scale, the activated-parameter gap translates directly into inference cost — though model price alone doesn’t tell the full story.

Even though it is still the smaller model, DeepSeek says the updated Flash version now beats the earlier V4-Pro preview on several agent-focused benchmarks.

Even though it is still the smaller model, DeepSeek says the updated Flash version now beats the earlier V4-Pro preview on several agent-focused benchmarks.

The company reported 82.7 on Terminal-Bench 2.1, 54.4 on DeepSWE, and 70.3 on Toolathlon-Verified.

Benchmark claims under scrutiny

Early independent testing by Artificial Analysis found a lower Terminal-Bench 2.1 score of 79%, which suggests that DeepSeek’s reported numbers may not always match independent results.

DeepSeek also shared results from several internal tests, though they have not yet been independently verified. If those results hold up, they add to growing evidence that companies can get more performance out of existing models through post-training instead of simply making them larger.

Open weights, full control

The MIT license means organizations aren’t limited to using DeepSeek through its hosted API. The release adds to a trend towards open-weight models closing the gap with proprietary alternatives. That flexibility is paired with support for tools many developers already use. V4-Flash now supports the Responses API for building AI agents and multi-step workflows, and DeepSeek has published instructions for integrating the model into Codex-based development workflows.

Familiar APIs, lower switching costs

For teams already using OpenAI-style APIs, that lowers the barrier to trying another model because they can evaluate it without making major changes to their existing setup.

The V4 technical report also covers inference improvements, like speculative decoding with DeepSeek’s DSpark framework, which are designed to make serving more efficient. When combined with self-hosted deployments, these features give infrastructure teams more ways to adjust performance for their own production needs.

This release reflects that companies are now finding new ways to improve model effectiveness without making models larger, and they’re competing on how those models are delivered. While many AI vendors focus on hosted APIs, DeepSeek continues to publish downloadable weights that organizations can run on their own infrastructure. Support for familiar API formats also makes it easier for teams to test open-weight models without revamping present workflows.

Companies are finding new ways to improve model effectiveness without making models larger, and they’re increasingly competing on how those models are delivered.

The post DeepSeek’s smaller model just outperformed its own flagship appeared first on The New Stack.

How one open-source ecosystem made state-of-the-art AI accessible

The post The Python Ecosystem That Changed AI Development appeared first on Towards Data Science.

Moonshot AI has released the open weights for Kimi K3 on Hugging Face, giving developers access to one of the largest open-weight language models yet. The Monday release follows a wave of overwhelming demand that forced Moonshot to temporarily pause new API subscriptions. Now, organizations with the necessary hardware can deploy K3 themselves.

In its documentation, Moonshot describes the model as being built for “long-horizon coding and end-to-end knowledge work.” Another notable detail is that Kimi K3 uses an OpenAI-compatible API. Because teams can try the model without rebuilding their existing integrations, switching to K3 could be as simple as changing the endpoint and model name. 

For engineers who have already built around OpenAI-compatible SDKs, that makes it much easier to evaluate K3 alongside existing commercial models. Taken together with the one-million-token context window, it’s clear the company is targeting engineering teams that already build around models like Claude Fable 5 and OpenAI’s GPT-5.6 Sol. While K3 is openly available, running it is another matter.

While K3 is openly available, running it is another matter.

Kimi K3: Its massive size and requirements mean few will be able to run it

The model uses a 2.8-trillion-parameter Mixture-of-Experts (MoE) architecture and ships in the hardware-friendly MXFP4 format. The weights alone occupy roughly 1.4 TB of storage, and practical self-hosted deployments require a distributed GPU environment — realistically eight or more servers equipped with eight NVIDIA H100 or B200 accelerators each.

That changes the conversation around open-weight AI. As The New Stack recently noted, the case for ownable models has grown stronger after Anthropic’s Fable 5 was pulled offline by a Commerce Department directive, a warning that access is not ownership.

Releasing the model weights and technical report of Kimi K3.

Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.

New model architecture: 2.5x the intelligence per unit of compute, not just more params.

Alongside… pic.twitter.com/Yz5uWeMbIm

— Kimi.ai (@Kimi_Moonshot) July 27, 2026

Ownership versus API economics

Instead of paying recurring API costs to OpenAI or Anthropic, organizations trade those operating expenses for significant investments in GPUs, networking, storage, power and operational expertise. That benefit is control. 

For organizations operating under strict regulatory requirements, the trade-off may justify the infrastructure investment. For many others, managed APIs will potentially remain the more economical option. The economics of open-weight models at enterprise scale remain an active area of debate across the industry.

Moonshot positions K3 as a frontier-class model capable of competing with OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5 on a variety of public benchmarks.

Benchmarks versus real workloads

The developer community is already taking notice of these coding capabilities. As MindStudio recently noted, “If you want to understand why developers are paying attention to Kimi K3, the benchmark to look at is SWE-bench Verified… For most of its history, SWE-bench has been dominated by proprietary models.”

The company’s own documentation is notably candid about its ongoing limitations. K3 always runs with reasoning enabled and defaults to its highest reasoning-effort setting, though Moonshot has since added lower-effort tiers. It may also behave too proactively when prompts are ambiguous. Moonshot also cautions that switching models within an ongoing conversation can reduce response quality.

That type of transparency is refreshing, but it additionally reinforces that benchmark scores shouldn’t drive deployment decisions. Early hands-on comparisons, such as The New Stack‘s Fable 5 vs. K3 coding match-up, suggest K3 can match Fable 5 on programming tasks at roughly a third of the cost, but runs about four times slower.

Organizations evaluating K3 still need to test it against their own workloads. But early community sentiment shows promise; open-source developers are already successfully utilizing K3 for complex, system-level tasks like porting the Godot game engine to WebGPU.

Morningstar senior equity analyst Malik Ahmed Khan echoed that overall wariness about benchmarks. “While K3 constitutes progress, we’d hesitate to ascribe it near-parity with American frontier models, such as Fable 5, in actual tasks,” Khan writes in a research note published before the release of the model weights on Monday.

Geopolitical risks loom large

K3 also arrives under growing geopolitical scrutiny. Anthropic and U.S. officials have accused Moonshot AI of distilling outputs from American frontier models during training. Anthropic’s Head of Public Policy Sarah Heck characterized the practice as intellectual property theft, while White House Office of Science and Technology Policy Director Michael Kratsios publicly alleged Moonshot relied on Anthropic’s models during development.

Moonshot has denied the allegations. Huang Zhenxin, Moonshot’s head of enterprise business, told Chinese state media that K3’s performance gains stem from architectural enhancements — specifically Kimi Delta Attention and Attention Residuals — not distillation. Some industry analysts have also questioned whether the timeline supports large-scale distillation, noting that Fable 5 had only been publicly available since July 1 before K3 appeared on July 16.

Whether those claims are ultimately substantiated or not, they bring another consideration for enterprise buyers. Beyond performance and infrastructure costs, organizations evaluating K3 may also have to consider future compliance, procurement, and regulatory risks.

K3 matters because of where it’s aimed. Moonshot isn’t building another consumer chatbot; its documentation makes clear this model was built for enterprise coding agents, heavy knowledge work, and production systems. The fact that demand blew past Moonshot’s GPU capacity within 48 hours says it all: at this scale, infrastructure pressure is guaranteed, whether you’re making API calls or hosting the weights yourself. 

The post Moonshot opens Kimi K3 weights — but few can run it appeared first on The New Stack.

The current maelstrom of discussion surrounding the use of open-source software and open-weight AI models appears to be splitting opinion on what constitutes legitimate openness versus actions that might constitute theft and create new cybersecurity vulnerabilities.

In a direct move to address these industry-wide concerns, 37 partners announced on Monday the formation of the new Open Secure AI Alliance and how the newly created body will develop techniques and tools to safeguard software by rapidly identifying and patching vulnerabilities.

Open Secure AI Alliance inaugural partners 

The inaugural partners of the Open Secure AI Alliance are Adobe, Cadence, Capital One, Cisco, Cloudera, Cloudflare, Cognition, CrowdStrike, Databricks, Dell Technologies, DoorDash, Elastic, HPE, Hugging Face, IBM, LangChain, the Linux Foundation, Microsoft, Naver, NetApp, Nvidia, Nous, OpenClaw, Palantir, Palo Alto Networks, Red Hat, Reflection, Salesforce, SAP, ServiceNow, Siemens, SK Telecom, Snowflake, SpaceXAI, Synopsys, Thinking Machines, and TrendAI.

It’s a grouping of some of the most influential names in technology, but also includes two notable exceptions: OpenAI and Anthropic, two closed, proprietary AI labs. Their absence is understandable, as they operate closed labs and open-weight AI models are effectively the competition.

Nvidia VP of enterprise platforms, Justin Boitano, has explained that open-weight models are foundational to American AI leadership and cybersecurity. 

“To maintain U.S. leadership in the AI industrial revolution, the infrastructure that runs our economy needs safe, secure access to both closed and open models,” Boitano said. “For cybersecurity, open models and open harnesses are essential because they broaden defensive capability, increase transparency for defenders, and complement frontier closed models with customizable, localized controls.”

As regulators grapple with AI safety, Boitano predicts it will be important to “recognize open models and open tooling as defensive assets” — thus enabling transparency, independent evaluation and shared remediation.

Nobody can subpoena a downloaded weights file

Mark Vigoroso, founder & CEO of technology consultancy firm The Enterprise Edge, tells The New Stack that AI regulators have traditionally “built their entire AI safety apparatus” around auditing a handful of closed labs. And now, that approach is out of date.

“Open weight models blew past that closed model approach months ago,” Vigoroso says. “This alliance is an admission that the actual safety work now has to happen in the infrastructure layer: patch cycles, provenance, identity around who’s deploying what, because nobody can subpoena a downloaded weights file.”

“This alliance is an admission that the actual safety work now has to happen in the infrastructure layer: patch cycles, provenance, identity around who’s deploying what, because nobody can subpoena a downloaded weights file.”

Vigoroso argues that the AI safety debate is “stuck on model-level controls”, while the real regulatory gap is provenance of infrastructure and identity, i.e., knowing where a model came from, who deployed a model and what it touched, not just whether the model itself is safe.

“Groups like the EU AI Office, NIST’s Center for AI Standards and Innovation (CAISI), and the UK’s AI Security Institute (AISI) focus almost entirely on frontier closed models. Open weight models (Mistral, DeepSeek, and others) fall into a regulatory blind spot: Once weights are released, there’s no way to enforce downstream safety obligations.

Current regulatory frameworks for AI models assume a single accountable deployer; open source has none. That’s the real story: Regulators are writing rules for a centralized world while the ecosystem is decentralizing, Vigoroso says.

While detailed operational information explaining the intended actions of this alliance is currently scant, Nvidia has highlighted that it is contributing solid research to the Open Secure AI Alliance to speed the development of new cybersecurity tools and techniques. 

Harnesses integrate with models, making agents easier to test

The open source Nvidia Labs Object-Oriented Agent (NOOA) project is now available on GitHub to make advanced AI safety capabilities more accessible for agent harnesses. This research framework enables harnesses to integrate with models to make agent behavior easier to test, trace, audit, and govern.

Aparna Rayasam, CEO of verified identity and end-to-end encryption company Atsign, tells The New Stack that the “AI blitzkrieg conversation has reached a critical inflection point.” This moment is one where we cannot build the next era of open cognitive innovation on top of what Rayasam calls “legacy, Swiss-cheese infrastructure.”

…the “AI blitzkrieg conversation has reached a critical inflection point.”

“The formation of the Open Secure AI Alliance proves that AI safety isn’t just an algorithmic math problem — it is a foundational networking problem,” Rayasam says. “The massive, distributed pipelines required to train and run modern AI demand an entirely new paradigm of trust. True safety means ensuring that the data pipelines feeding these models are inherently invisible, un-attackable, and completely stripped of open network perimeters.”

The key notion here is that we are moving from a world of protecting data at rest to a world where the connective tissue of AI must be secure by design. 

One AI vendor to secure them all? No thanks.

Founder and CPO of agentic identity and permissions security company Reco, Gal Nakash, tells The New Stack that the launch of the Open Secure AI Alliance is an “important signal” which underlines why AI security can’t be solved by one vendor or one closed framework. 

“Jensen Huang’s point that every SaaS company will become a GaaS company captures why this matters now: Software is shifting from passive tools people log into, to AI agents that access data, take actions, and execute workflows,” says Nakash. “Open source tools and shared standards can help the industry move faster, but they need to be grounded in real enterprise context across identity, permissions, data access and behavior.”

Chris Boehm, Field CTO at automated, identity-driven microsegmentation company Zero Networks, tells The New Stack that news of the Open Secure AI Alliance makes him feel like he’s seen this before somewhere.

“This looks like the Trusted Platform Module (TPM) at Microsoft story all over again,” Boehm says. “It’s a case of an industry group defining what trusted hardware means, the platform vendors adopt it, and within a few years it’s a procurement requirement rather than a suggestion.”

He explains that “Windows 11 did exactly that with TPM 2.0 and Secure Boot”, and both Linux and Apple adapted. “I’d expect the same for AI infrastructure, where attested silicon becomes the floor for regulated workloads, and the vendor list narrows to whoever can meet it,” predicts Boehm.

“This looks like a case of an industry group defining what trusted hardware means, the platform vendors adopt it, and within a few years it’s a procurement requirement rather than a suggestion.”

A more global and geographically-inclusive approach is needed 

Amanda Brock, CEO of open technology body OpenUK, tells The New Stack that the Open AI Alliance is undoubtedly a great starting point, particularly with OpenAI’s security woes it divulged last week. 

“But, like the open letter on US Leadership in open weights, this is a US response to a US challenge,” Brock says. “Rumors of a forthcoming Presidential Executive Order to close down open models have been circulating for weeks — and worsened by the administration being thrown into turmoil over China’s Kimi K3.” 

For this alliance to succeed, Brock insists that it will need to take a more “global and geographically-inclusive approach”, beyond the US-centric founding members.

“It must also engage the open source ecosystem of individuals and innovators who are building the infrastructure, agentic harness functions and developer tools for AI. It’s important to realize that open AI infrastructure development shifts the innovation into the hands of the many, in direct opposition to the small number of corporate creators of frontier models,” Brock adds.

Nvidia’s Boitano echoes Brock’s view. In a blog post reviewed in draft by The New Stack, he writes that “open models turn more AI users into AI builders,” expanding opportunity, accelerating innovation, and keeping progress from being concentrated in only a few organizations or regions.

Boitano concludes by saying that open models also enable independent scientific research into how AI systems behave, allowing researchers to understand, evaluate, and improve them. It’s all about what he has called making broad, continuous defense possible.

Looking ahead, it feels like the next wave of AI safety bodies, movements, or alliances won’t just be model auditors—they’ll be trust-infrastructure standards bodies (encompassing identity verification, content provenance, credence, etc.) using borrowed approaches to governance and compliance. Ultimately, this may be the only enforcement layer that survives open weight proliferation.

The post Nvidia, Palantir, Hugging Face join 34 others in race to defend open-weight AI from cyber threats appeared first on The New Stack.

Building a great AI agent isn’t just about choosing the right models. The harness is the architecture surrounding the model. How it renders context, executes...

Building a great AI agent isn’t just about choosing the right models. The harness is the architecture surrounding the model. How it renders context, executes actions, manages state, and decides when a task is done shapes outcomes just as much as the model itself. Harness design alone can account for double-digit swings in benchmark results and significant differences in token cost…

Source

When you run Kubernetes at the scale we do on Amazon EKS, nodes break constantly. GPUs fall off the PCIe bus. Container runtimes wedge. Network interfaces disappear. Across tens of thousands of clusters, “rare” hardware failures happen multiple times a day, somewhere in the fleet.

For years, everyone responded the same way: an operator wakes up, reads a dashboard, SSHes into the node, cordons it, drains it, terminates the instance, and waits for a replacement. Every step is human-paced. Every step is toil. And if the failure lands at 3 a.m. on a weekend, the workload sits degraded for hours before anyone looks.

We built the EKS Node Monitoring Agent to help close that gap, which we open-sourced in April earlier this year. It detects node failures and writes Kubernetes NodeConditions that signal the problem to Karpenter, which then automatically replaces the node if required. The agent is one piece of a larger system. To understand where it fits, you need to understand what manages the nodes it monitors.

“Across tens of thousands of clusters, ‘rare’ hardware failures happen multiple times a day, somewhere in the fleet.”

AWS launched Amazon EKS Auto Mode that fully automates Kubernetes cluster infrastructure: compute provisioning, scaling, networking, storage, OS patching, and security hardening, so teams focus on applications, not cluster operations. It dynamically selects optimal EC2 instances (including GPU instances like P5, P6, and G6 families), scales based on workload demand, consolidates underutilized nodes, and keeps the operating system patched. EKS Auto Mode ships with automatic node repair as the default behavior: detection, severity classification, and Karpenter-driven node replacement all run out of the box with no add-on to install, no controller to configure, and no repair policy to write.

This is the story of how we built automatic node repair, the design decisions that shaped the system, and the hard lessons that came from operating it at GPU scale.

Six lessons from building self-healing Kubernetes nodes at scale

After operating this across thousands of clusters, the lessons compress into a short list. These are not unique to our system. The same patterns show up in NPD, NVSentinel, AKS Periscope, GKE’s auto-repair, and anyone building a custom node controller. They are folklore that should be a checklist. The open-source repo reflects each of these lessons in code, from the reason-code stability guarantees in the API to the jitter implementation that solved GPU workload interference.

“Your reason codes are an API contract. Additions are features. Renames are breaking changes.”

1. Your reason codes are an API contract. Every downstream consumer (repair controllers, dashboards, customer automation) keys on them by literal string match. Additions are features. Renames are breaking changes. Severity changes are breaking changes. Plan for them the way you plan for API versioning.
2. Absent and Unknown are not the same thing. “We are not watching” and “we are watching but cannot tell” require different responses from downstream automation. If your disabled monitor writes Unknown, some controller somewhere will eventually act on it. Emit nothing when you are not watching.
3. Don’t cross ownership boundaries. The kubelet owns workload-driven conditions. Your node-health agent owns hardware and infrastructure failures. Crossing that boundary means your repair system is fighting the kubelet’s eviction system, and one of them will make the wrong call.
4. Measure latency from the source. The detection SLO includes every hop in the signal chain: hardware event to driver log, driver log to journald, journald to agent poll, agent poll to NodeCondition write. The longest hop dominates. For kernel-level signals, journald flush cadence is the bottleneck. For GPU telemetry through DCGM, push-based policy violations (DBE, XID, NVLink) are near-instant, but polled field watches (NVSwitch fabric health, clock throttle) have a 5-minute floor. Know which path each detection uses.
5. Detection and diagnosis are separate systems with separate consumers. Detection feeds automation (fast, continuous, minimal data). Diagnosis feeds humans (on-demand, detailed, heavyweight). Conflating them degrades both.
6. Test telemetry interpretation against the spec, not empirical values. Hardware telemetry interfaces are not boolean. We read a DCGM bitfield for GPU fabric health and treated non-zero as failure. When a driver update changed the healthy return value from zero to a spec-defined non-zero mask, every GPU node was flagged unhealthy at once. The safety breaker held (by design), giving us time to ship the fix. The lesson: if you’re parsing packed enums or bitfields from GPU firmware, your test fixtures must come from the vendor documentation, not from what the field happened to return on previous hardware.

Node health detection in Kubernetes: Traps no one warns you about

Every node health agent in the Kubernetes ecosystem performs the same translation. Node Problem Detector (NPD), NVSentinel, GKE’s auto-repair, AKS’s Linux Extension, and the EKS Node Monitoring Agent all take noisy, low-level signals from a machine and translate them into a set of Kubernetes primitives: NodeCondition, Event, sometimes a CRD. The translation looks simple. It isn’t.

The output is a NodeCondition, which is just a type, a status (True/False/Unknown), a reason code, and a message. Four fields. But that surface area hides decisions that determine whether a repair action helps or hurts.

Reason codes are a public API. We learned this the hard way. In version 1.6.2, we changed NvidiaDeviceCountMismatch from Warning severity to Fatal. The technical reasoning was sound: once a GPU drops off the PCIe bus, it doesn’t come back without a node reboot or replacement. Leaving it as Warning meant GPU workloads kept getting scheduled onto degraded nodes, wasting expensive accelerator capacity. So we shipped the fix. Downstream automation broke. Customers had repair configurations keyed on the old severity. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. From that point, we treat every reason code addition as feature work and every rename or severity change as a breaking change.

“Absent” must not equal “healthy.” When we shipped per-monitor configurability in v1.6.0, we had to make a choice. A disabled monitor needs to produce some output (or no output). The three options: write True (your auto-repair now thinks the node is healthy because you’re not watching), write Unknown (ambiguous, might trigger repair depending on downstream logic), or omit the condition entirely. Only the third is safe.

This seems obvious in retrospect, but consider that NPD achieves the same result through a completely different mechanism: compile-time disable via build tags. NVSentinel delegates it to operator-authored CEL rules. The upstream Kubernetes spec defines what Unknown means, but if your repair automation treats Unknown as actionable, you will lose nodes for no reason. We chose to emit nothing when a monitor is off, and documented it as a hard contract.

Detection latency is bounded by the source, not by the agent. We originally told customers, “We detect kernel panics within 30 seconds.” This was wrong. Our agent’s detection time was under 30 seconds. But the kernel panic shows up in journald, and journald’s flush cadence is the actual bottleneck. If journald takes 45 seconds to write the line, our 30-second claim was incomplete.

For GPU faults, the picture is more nuanced because we use two detection paths with very different latency characteristics. The critical faults (double-bit ECC errors, XID errors, NVLink failures, page retirements, thermal and power violations) go through DCGM’s push-based policy violation channel. DCGM notifies our agent the moment it detects the violation; there is no polling interval. Detection of these faults is near-instant (sub-second in practice). A separate path uses a 5-minute field-value window to monitor NVSwitch fabric health, Fabric Manager status, and clock-throttle reasons. That window is the floor for those specific detections, but it does not apply to the critical GPU faults that trigger automatic repair. The lesson: the customer-facing SLO must include source-of-truth latency, and different signal paths within the same subsystem can have radically different floors.

Two severities, one switch: How auto-repair decides which nodes to replace

The kubelet already reports DiskPressure, MemoryPressure, and PIDPressure. NMA complements those with five additional conditions covering domains the kubelet does not monitor: kernel health, container runtime, networking, storage, and accelerated hardware. Every detection carries one of two severities, and severity is the switch that decides whether the repair cycle fires.

Condition severity is a terminal fault. It flips the matching condition to False and makes the node eligible for automatic repair. GPU device-count mismatches, critical XID and double-bit ECC errors, NVLink and NVSwitch fabric failures, a missing Fabric Manager, and Neuron DMA and HBM uncorrectable errors. On the networking and runtime side: VPC CNI process down, IPAMD unable to reach the API server, fork failures due to PID exhaustion, and pods wedged, terminating behind a broken container runtime. These are faults that won’t recover on their own. On GPU nodes, a single degraded accelerator can corrupt training checkpoints or waste thousands of dollars in compute per hour.

Event severity is informational. It posts a Kubernetes event, the NodeCondition remains True, and operators get visibility without disruption. Bandwidth ceilings, connection-tracking limits, Amazon Elastic Block Store (Amazon EBS) IOPS throttling, I/O delays, filesystem fragmentation, clock drift, liveness and readiness probe failures, kube-proxy anomalies, GPU thermal and power warnings, PCIe link degradation, and page-retirement thresholds. These signal trouble building before it turns terminal.

Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes and needlessly displace workloads. Too conservative, and degraded nodes serve traffic for hours while a GPU with a failing memory bank corrupts training checkpoints. The classification principle: if the failure is deterministic and infrastructure-owned (hardware broke, firmware crashed, a physical link went down), it triggers replacement. If the signal could be application-induced or transient, it stays informational. You never want to terminate a healthy node because a misbehaving pod saturated a resource.

“Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes. Too conservative, and degraded nodes serve traffic for hours.”

DiskPressure, MemoryPressure, and PIDPressure are the canonical examples. Every major auto-repair system (GKE, AKS, NPD) has independently converged on the same answer: don’t touch them. These are workload-driven conditions, not node-level faults. Replacing the node just moves the misbehaving workload to a fresh machine, where it will eat memory again. The correct response is kubelet-level pod eviction, not node replacement. If you’re building a node-health system, draw this boundary early and document it publicly.

The agent that hurt what it was protecting: GPU workload interference from health monitoring

The hardest lesson came from a customer running large-scale distributed GPU training. Their workload used NCCL collectives across hundreds of GPU nodes, where every node in a communication group must complete its step before any can proceed. One slow node makes every node wait.

They found that NMA itself was causing periodic slowdowns. The agent’s monitors all ran on independent goroutines, and when their polling intervals aligned, dozens of goroutines would wake simultaneously and burst onto many CPU cores at once. On a general-purpose web service, this would be invisible. In a distributed training job, microseconds of jitter on one node can cascade across the entire GPU cluster, causing measurable throughput loss.

The customer disabled NMA entirely and saw an immediate improvement. That was the worst possible outcome for us: a health agent that interferes with the workload it exists to protect is worse than no agent at all.

The fix was straightforward once we understood the problem. We added a startup jitter to every monitor’s polling interval. Each goroutine delays its first tick by a random offset (up to 20% of its base interval), staggering the wake times so they don’t align on boot. We cached system calls that hit /proc on every poll. We consolidated handlers that shared an interval into a single sequential work queue, reducing the goroutine count for monitors that didn’t need their own thread. The result was an agent whose CPU profile is flat and predictable rather than bursty.

The lesson generalized: if your health agent runs on the same host as the workload, its resource consumption pattern matters as much as its resource consumption total. A process that uses 0.5% CPU spread evenly is invisible. A process that uses 0.5% CPU in concentrated bursts can disrupt latency-sensitive distributed GPU workloads in ways that show up as lost training time rather than a CPU alarm.

This is why per-monitor configurability matters. Not every monitor is relevant to every workload. A dedicated GPU training cluster with one pod per node and no pod churn doesn’t need IPAMD monitoring or environment scanning. We shipped the ability to disable individual monitors so customers can keep the health coverage they need without paying the overhead of coverage they don’t.

How the repair cycle works

Karpenter is the compute controller that provisions and scales EKS Auto Mode nodes. It already owns the lifecycle of every node it launched, and consuming our NodeConditions for repair is a natural extension of that ownership. There’s no separate repair backend, no sidecar controller, no webhook chain. The same system that created the node is the one that replaces it.

Karpenter’s AWS cloud provider declares repair policies: each one pairs a condition type with a status that means “replace this node.” The policies include toleration windows that prevent reacting to transient blips:

• Accelerated hardware faults: 10 minutes. These are unambiguous (a GPU is either present or absent) and expensive to leave running (a training job on a degraded node wastes GPU-hours).
• Everything else (kernel, runtime, networking, storage, kubelet NotReady): 30 minutes. Enough time for a transient network blip or a temporary runtime hiccup to resolve on its own.

The flow:

1. The agent detects a terminal fault and flips the matching condition to False with a reason code.
2. Karpenter’s health controller sees the transition and starts a timer.
3. If the condition clears before the window expires, the timer resets silently. The node was never touched.
4. Past the toleration window, a safety gate checks fleet health. Karpenter will not repair more than 20% of nodes in a NodePool simultaneously. If a correlated event (a bad AMI rollout, a control-plane hiccup, a zonal impairment) trips conditions across many nodes at once, the system holds. Auto-repair also stands down while an Amazon Application Recovery Controller zonal shift is active, so deliberate traffic movement away from an impaired Availability Zone is not mistaken for a fleet of broken nodes.
5. Inside the safety threshold, Karpenter taints the node to block new scheduling, gracefully drains running pods (respecting PodDisruptionBudgets), terminates the instance, and provisions a replacement sized for the displaced workload.

The replacement node comes up with a fresh agent monitoring it from boot. No operator in the path. In our testing, the full cycle from fault injection to replacement node running workloads took under 12 minutes. Detection landed in under a second (critical GPU faults use DCGM’s push-based policy channel, not polling). Then 10 minutes of toleration, and roughly 90 seconds for the replacement to launch and register.

The part that surprised us: detection and diagnosis are not the same problem

Auto-repair handles the common case: broken node gets replaced, workload keeps running. But “why did that node fail?” is a different question, and one we initially tried to answer inside the detection path. That was a mistake.

Detection answers “is this node healthy?” It runs continuously with minimal overhead, and it needs to be fast: a condition flip that takes 5 minutes to produce is 5 minutes of degraded workload. Diagnosis answers “what went wrong?” It needs to collect detailed artifacts: full journald output, containerd state, network configuration, dmesg, GPU driver logs. In our testing, that collection completes in about 7 seconds and produces a compressed log bundle. Baking it into the detection hot path would have slowed down the thing customers care most about: how fast the system reacts.

We built them as separate concerns sharing an agent binary. The NodeDiagnostic CRD lets you request a full log bundle from any node through kubectl, without SSH. On EKS Auto Mode, where nodes are Amazon Elastic Compute Cloud (Amazon EC2) managed instances with no shell access by design, this is the only way to investigate after a GPU failure or any other node-level fault.

The experience is one command:

kubectl ekslogs <node-name>

The plugin creates a NodeDiagnostic resource. The agent on the target node detects it via a watch, collects system state into a compressed tarball, and stores it temporarily (available for 10 minutes). The plugin then downloads it through the kubelet’s Node Log Query API (KEP-2258, GA in Kubernetes 1.36). No SSH, no security groups, no key pairs.

This separation means detection doesn’t slow down to collect evidence, diagnosis doesn’t need to be always-on (saving node resources), and you can diagnose a node that auto-repair has already flagged but hasn’t yet terminated. The 10-minute window for accelerated hardware faults gives you exactly enough time to grab the logs before the node is gone. If you’re interested in further improvements, engage with us on EKS public roadmap.

What this means if you’re running EKS

On EKS Auto Mode, all of this is on by default. Auto Mode fully manages your cluster infrastructure (compute, networking, storage, patching, and security hardening) so you focus on applications, not cluster operations. The agent runs as a systemd service in the node image (not a DaemonSet you manage), Karpenter consumes its conditions as part of the compute lifecycle it already owns, and kubectl ekslogs gives you diagnostic access without SSH. There is nothing to install, configure, or operate. For GPU workloads, this means your expensive accelerator nodes are automatically monitored, classified, and replaced without any operator intervention.

On managed node groups or self-managed Karpenter, you can assemble the same loop: install the Node Monitoring Agent as an EKS add-on and opt each node group into auto-repair. The architecture is the same, just not pre-assembled.

The EKS Node Monitoring Agent is Apache 2.0 open source at github.com/aws/eks-node-monitoring-agent. 

The failure modes we hit when running it at scale, and the fixes that come out of them, flow back to anyone using it. If you’re building a node-health system or running ours and hitting an edge case, come build with us!

The post Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent appeared first on The New Stack.

Your organization constantly needs more information about system performance, usage, and data while in production — or better yet, before it heads to prod. The challenge of telemetry increases with the complexity of your stack and agentic sprawl. Because “it works in the testing environment” becomes moot in the face of non-deterministic agents.

After all, AI agents span multiple environments, and that leaves traditional log-metric-trace models insufficient to handle the volume of the agentic AI era. The situation can lead companies to think that the best option is to throw everything into the locked box of proprietary tooling, but that creates another problem: Information is siloed within each layer, fragmenting data and taking you further from realizing real AI ROI.

Unified context across fragmented workflows

The OpenTelemetry framework and the OpenSearch distributed search and analytics engine make for a powerful, open-source pairing that gives organizations of all sizes unified context across their fragmented workflows. In fact, OTel has crossed the 95% adoption threshold for new cloud-native instrumentation projects and has already become the default choice for Greenfield projects.

OpenSearch, sponsored by Amazon Web Services, is gaining traction with AI engineers, as it recognizes that observability and AI must be united. This year’s OpenSearch roadmap specifically focuses on making it the primary retrieval interface for AI agents and an essential piece of any retrieval-augmented generation and agentic AI stack. 

Join us on July 22

Just because open source doesn’t have a direct cost doesn’t mean it’s free. That’s why Dotan Horovits and Rekha Thottan of AWS are going to perform a live troubleshooting simulation using correlated logs, metrics, and traces, followed by a demo of how agentic traces flow through Otel pipelines. Also learn how the open-source evaluation framework Agent Health can provide a structured pre-production benchmark to flag unpredictable agentic behavior before release. 

Join us live on July 22 to learn along and ask questions to learn how your organization can adopt these open-source standards in the second half of this year — across agentic workloads and traditional infrastructure, at scale.

Register for the webinar here

The post Watch AWS engineers troubleshoot agentic AI with OpenTelemetry and OpenSearch appeared first on The New Stack.

In April, Andrej Karpathy published a GitHub gist file called “LLM Wiki,” a brief text document designed to help one build a personal knowledge base using LLMs. It’s based on the premise that an AI agent will keep what it knows as linked Markdown files it can read and rewrite, because a language model does not get bored maintaining cross-references and can touch fifteen files in a single pass. It was only a few thousand words with no product attached.

Two months later, Google turned that instinct into a published standard called the Open Knowledge Format. The OKF packages organizational knowledge, metrics, tables, and runbooks as plain Markdown that any agent can read without a proprietary account. Google is careful to call it v0.1 — a starting point rather than a finished standard.

Garry Tan, the Y Combinator president, got there first in a different lane. His gstack, an MIT-licensed Claude Code setup that crossed 66,000 GitHub stars within weeks, comprises 23 specialist roles, each a Markdown file. No runtime; no code; just prose that runs across ten different coding agents.

Markdown has become the substrate agents read and write

Three approaches, three different needs, one common solution. Karpathy sought agent memory, Google aimed for enterprise context in BigQuery agents, and Tan wanted a way to summon an engineering team from a terminal. All three turned to the same basic resource: a folder of Markdown files versioned in git.

Developers had already established this practice. CLAUDE.md and AGENTS.md are present in millions of repositories as the initial files an agent loads. OKF and gstack are the evolved forms of this convention – one focused on what the agent knows, the other on how it behaves.

This is the Git and JSON playbook tied to the agent’s knowledge. The formats that survived are the ones you could start using without changing anything. You can simply cat the file, clone the repo, and any tool you already use can parse it. MCP remains important as the interface an agent connects to. Markdown is becoming the format that carries the content.

The lock-in moved from the model to the files

The significant factor to observe here is the competitive advantage, not technical specifics. For two years, the belief was that owning the best model meant controlling the developer.

This perspective is now shifting. Replacing Claude with GLM or Codex, gstack continues to operate because the core intelligence evolved, but the documentation did not.

The moat is shifting from the model to the Markdown a team owns and accumulates over time.

The moat is shifting from the model to the Markdown a team owns and accumulates over time. A company’s OKF bundle, including its runbooks, metric definitions, and architecture decisions, is, by design, portable across clouds, models, and frameworks.

That kind of portability is the reason vendor-neutral formats exist and why Google’s OKF deserves a closer look.

If no one develops consumers for it, it remains just a good idea that Google released on a slow Friday.

The area where I am most likely mistaken is durability. Declaring Markdown standards is easy, but making them reliable is difficult. OKF is merely a 0.1 draft with a reference implementation, not a full ecosystem. If no one develops consumers for it, it remains just a good idea that Google released on a slow Friday.

The direction remains determined by three separate bets targeting the same file format within a single quarter. Your next agent is likely to interpret its context from a Markdown folder, and the creator of that folder now possesses an advantage that the model vendor cannot easily replicate.

The post Andrej Karpathy, Google and Garry Tan agree Markdown is the answer, but they’re not solving the same problem appeared first on The New Stack.

Researchers show that serving AI models with llm-d can boost inference speeds by up to 5 times and double throughput — all while using heterogeneous GPUs.

How much of Nvidia is reflected by their visionary CEO, Jenson Huang? With his praise and later support of OpenClaw, Huang took a big step beyond the corporate boundary to embrace the “bad boy” of the agent world. Where exactly does Nvidia fit in here?

The New Stack put that question to Nader Khalil, Director of Developer Technologies at Nvidia, as well as how Nvidia is working with developers on agentic AI projects.

Khalil, co-founder of Brev.dev, found himself acquired by Nvidia about two years ago. His company helped startups access Nvidia AI chipsets. He is still excited by the possibilities of AI, and his energy is proof that Nvidia is enjoying the moment. Khalil was expansive, showing a startup’s keenness for the pace of change around them.

Before anything else, Khalil defines what he believes an agent is. “I have some slides,” he threatens. But these are more to organize his thoughts on an oft-asked question- not an attempt to lecture.

“An agent is an LLM and a harness… Each loop should take us closer to our goal.”

“An agent is an LLM and a harness. And if you think about that, it involves two things. It involves the loop and the LLM. And obviously you don’t want each loop to do the same thing. You want to leverage the results from the LLM. That might include reasoning on new tools to use. Each loop should take us closer to our goal.”

Nader praises the early OpenAI initiative. “So ChatGPT innovated outside of the model. It was not just a great model they made; they also added prompts. There was a system prompt and then the user prompt; there was multimodal, and suddenly that felt really good as a way for me to use the LLM. Every user could benefit from a system prompt that OpenAI had written while you were using your individual prompt.” Khalil continues, “Then they added memory.”

“Suddenly my assistant became really useful because it remembers things about me. ChatGPT knows that I really like to barbecue. So when I ask a question, it remembers what my smoker is,” recalls Khalil. “The thing that I was missing was files.”

Of course, the story continues through Cursor to Claude. “But this is the harness. Everything here is the harness,” he says.

Khalil moves on to how Nvidia works today. “The way to get your product into this rapidly growing market is with skills. Hence the CUDA X library.”

These are the implementations of use cases that target GPU acceleration, usually for compute-intensive applications.

“And so we look at every product we build now, it needs to have a skill because you need to cater to this growing audience,” he says.

This is how Nvidia first works with in-house experts, and connects to their edge hardware.

Supporting OpenClaw

Khalil was happy with the wording that Nvidia are “supporting” OpenClaw. “We’re just squarely in the community”, agrees Khalil. “We do this by the way, through a lot of projects that are very important in the open source ecosystem.”

But OpenClaw is not just any project and could be considered quite a risk to associate with. “We have a couple of developers at the company that contribute to OpenClaw full time.” Pushed on the nature of the relationship a little more, Khalil offers, “I think we just try to contribute wherever we can. I think what’s very clear is that harnesses had a moment, right?”

“We have a couple of developers at the company that contribute to OpenClaw full time.”

It has been quite a moment. “There is a lot of change happening right now, and we’re really thankful to [Peter Steinberger], OpenClaw, and the community for creating this moment around agents and harnesses. We of course want to contribute.”

Related to this, the OpenClaw project currently has many unresolved pull requests (PRs). In fact, there were rumors that new PRs are no longer accepted at all.

“We saw Peter tweeting about some of the issues they had, and we just rolled up our sleeves and were eager to help. They bless us by allowing our contributions.”

“You know,” says Khalil, “We saw Peter tweeting about some of the issues they had, and we just rolled up our sleeves and were eager to help. They bless us by allowing our contributions.”

Khalil reflects on things a little more: “You know the cardinal rule of code. It is easier to write than it is to read.”

And at over 800,000 lines of code, this must be true. Khalil continues, “It is easier not to have to process this complicated codebase, but every successful project right now has the same issue. It is easier to enlist many agents to help write code and build these PRs. The bottleneck is in merging the PRs through.” As well as dealing with the fallacies.

“OpenClaw was a major change for the industry. It was a huge moment, and everyone’s eyes are on it. It got more stars than Linux in months. Developers care deeply about the project because it was influential, and so I think you’re gonna see a mountain of PRs, right?”

“It got more stars than Linux in months… so I think you’re gonna see a mountain of PRs, right?”

Their attitude to OpenClaw is clearly to accept its problems, like that raucous friend who seems to wind up in police custody after a wild party, but is good at heart.

Blueprints and microwaves

Hermes is one of the newer projects in the wake of OpenClaw (like NanoClaw) that wants to bottle the lightning but in a safer way. Nvidia is also embracing it, but Khalil backs up to explain how Nvidia looks at projects in general.

“So, NemoClaw is our blueprint. When we see amazing harnesses, we try to figure out how we can help enterprises adopt them. Consumers sometimes want the security to run any agent; then there’s the model and harness. Then there are the skills, right. You have to give it access to your terminal.’

The term “blueprint” takes on a bit more formality in Nvidia, meaning the structure for building AI agents and systems. And of course Khalil needs to show these working with the Nemotron model and other Nvidia solutions.

“There’s a blueprint for Hermes and a blueprint for OpenClaw”. It sets up the runtime, enables the policies if there’s a local GPU, and runs the model.

Working with agents in the enterprise is seen as a significant risk. “There are a bunch of camps,” says Khalil. “There are teams within enterprises who are more worried. We have a project called OpenShell that is our security runtime and we’ll work with.”

“Our goal is to create the tooling that’s needed in the ecosystem. Developers in industry and enterprises have actually been adopting agents. And we have been building for this audience. One way to do so is to build a specialized agent or a sub-agent. “

So Nvidia doesn’t offer a big takeover solution, but fits in with where teams already are.

Your microwave, your agent

“The way to think about it is like when you use a microwave that you haven’t used before, you have to press a lot of buttons or spend time figuring it out. But when it’s your microwave at home, you just go ‘Boop, boop. Done.’ Right?”

“So every industry in enterprise will be building these specialized agents, and many already have. Nvidia is already working with CrowdStrike and Cadence, Palantir, among many others.”

The future will be agents

Khalil believes a lot of the concern over long-running agents is slowly petering out. Which leads to the final question: is Nvidia looking to stay in the open sea where there may be dragons, or become a calm port for developers to work in?

“So our approach is: Who can we help and how?” Khalil shows no fear, or lack of sea legs. “The inflection point happened months ago, so we ask what can we do to usher in all of this technology.” Here, Khalil ties his — and, to a degree, Nvidia’s — future to green-field developers.

“There are gonna be some people quick to adapt. And some people that aren’t; and what we’re noticing, if you look at the adoption curve, many people have yet to experience this. So there’s much work in helping make sure that we deliver this safely.”

The post “An agent is an LLM and a harness”: What Nvidia really thinks about OpenClaw appeared first on The New Stack.

Developing autonomous vehicle (AV) policies requires bridging an important gap between training and deployment. Vision-language-action (VLA) models that can...

Developing autonomous vehicle (AV) policies requires bridging an important gap between training and deployment. Vision-language-action (VLA) models that can reason over more complex driving scenes and produce richer intermediate reasoning are predominantly trained in open-loop, where model outputs are directly compared to ground-truth behaviors without considering their effect on the environment.

Source

💾

AI is now essential infrastructure, powered by AI factories that generate intelligence in the form of tokens. As demand grows, these factories must scale...

AI is now essential infrastructure, powered by AI factories that generate intelligence in the form of tokens. As demand grows, these factories must scale faster, operate more efficiently, and lower the cost of intelligence across the five-layer stack: energy, chips, infrastructure, models, and applications. NVIDIA DSX platform provides the complete playbook for designing, simulating, building…

Source

High‑quality 3D medical imaging data is the foundation of modern radiology AI, but access to it is often constrained by data scarcity, privacy restrictions,...

High‑quality 3D medical imaging data is the foundation of modern radiology AI, but access to it is often constrained by data scarcity, privacy restrictions, and the high cost of expert annotation. As a result, training reliable 3D medical imaging models is frequently bottlenecked by small, narrow, and hard‑to‑share datasets, limiting model robustness and generalization. To help teams overcome…

Source

A so-called software supply chain attack, in which hackers corrupt a legitimate piece of software to hide their own malicious code, was once a relatively rare event but one that haunted the cybersecurity world with its insidious threat of turning any innocent application into a dangerous foothold in a victim’s network. Now one group of cybercriminals has turned that occasional nightmare into a near-weekly episode, corrupting hundreds of open source tools, extorting victims for profit, and sowing a new level of distrust in an entire ecosystem used to create the world’s software.

On Tuesday night, open source code platform GitHub announced that it had been breached by hackers in one such software supply chain attack: A GitHub developer had installed a “poisoned” extension for VSCode, a plug-in for a commonly used code editor that, like GitHub itself, is owned by Microsoft. As a result, the hackers behind the breach, an increasingly notorious group called TeamPCP, claim to have accessed around 4,000 of GitHub’s code repositories. GitHub’s statement confirmed that it had found at least 3,800 compromised repositories while noting that, based on its findings so far, they all contained GitHub’s own code, not that of customers.

“We are here today to advertise GitHub’s source code and internal orgs for sale,” TeamPCP wrote on BreachForums, a forum and marketplace for cybercriminals. “Everything for the main platform is there and I very am happy to send samples to interested buyers to verify absolute authenticity.”

Read full article

Comments

© NiseriN via Getty Images

Maximizing the value of AI infrastructure demands deep visibility into GPU utilization. Yet many platform teams running AI workloads on Kubernetes operate with...

Maximizing the value of AI infrastructure demands deep visibility into GPU utilization. Yet many platform teams running AI workloads on Kubernetes operate with limited visibility into how their GPUs are used. Most don’t know who’s consuming them, how much memory is in use, and whether Kubernetes pods are pending or silently idle. Without a signal, GPU fleets are routinely underutilized and slow to…

Source

Autonomous AI agents are becoming more capable. Open models, Model Context Protocol (MCP)-connected tools, and portable skills are also making agents easier to...

Autonomous AI agents are becoming more capable. Open models, Model Context Protocol (MCP)-connected tools, and portable skills are also making agents easier to extend.But scaling agent use with structural transparency and operational integrity requires more than runtime guardrails. Organizations and teams need to understand and trust the skills, or instructions, an agent is using.

Source

The compute capability of large GPU fleets presents unprecedented opportunities to innovate and provide value to customers in record time. Yet these...

The compute capability of large GPU fleets presents unprecedented opportunities to innovate and provide value to customers in record time. Yet these advancements come with a variety of challenges. At scale, teams are juggling heterogeneous hardware, fast‑moving software stacks, tight power envelopes, and spiky, multitenant workloads. A single hotspot, misconfigured driver, or subtle hardware fault…

Source

The release of MiniMax M2.7 adds enhancements to the popular MiniMax M2.5 model, built for agentic harnesses,...

The release of MiniMax M2.7 adds enhancements to the popular MiniMax M2.5 model, built for agentic harnesses, and other complex use cases in fields such as reasoning, ML research workflows, software, engineering, and office work. The open weights release of MiniMax M2.7 is now available through NVIDIA and across the open source inference ecosystem. The MiniMax M2 series is a sparse mixture-of…

Source