Most organizations that know they should sign their images still don’t. Not because they disagree, but because the path to doing it well has been too long. The result is a delivery pipeline built on trust that nobody can verify.

The problem space

Unsigned container images create an open door for attackers at every stage of the delivery pipeline. Malicious images masquerade as legitimate packages, waiting to be pulled by an unsuspecting team. Compromised CI/CD pipelines silently inject tampered artifacts into production builds with no cryptographic evidence of modification. Stolen credentials let a bad actor impersonate a trusted publisher. Even within a single organization, inconsistent practice means some teams sign while others skip the step entirely, leaving gaps in the chain of trust that nobody has mapped. Compounding all of it is base image inheritance. Every container image inherits the security posture of its parent, so one compromised base image can propagate across dozens of downstream services before anyone notices.

“Scanning is fundamentally reactive. One tells you what is inside. The other tells you whether you can trust it.”

Scanning is fundamentally reactive. It answers, “what vulnerabilities exist in this image?” It cannot answer the question that matters more as artifacts get harder to inspect: “who built this, and has it been modified since it left the build system?” That is the domain of cryptographic signing, which provides proactive provenance. The two are complementary, not interchangeable. One tells you what is inside. The other tells you whether you can trust it. 

Why the AI era makes this urgent

The workloads have changed faster than the tooling. Model weights, training datasets, inference runtimes, and agent tooling increasingly ship as OCI artifacts. A pickled PyTorch checkpoint itself has no CVE to match against. Safer serialization formats like .safetensors remove the code execution path, but they say nothing about who produced the weights or whether they’re the ones you meant to load. There is no vulnerability database for a set of trained weights, and the CVE and SCA based scanning that registries run has nothing to compare them to. 

This is not theoretical. In February 2024, JFrog researchers found a malicious PyTorch model on Hugging Face that opened a reverse shell the moment it loaded, abusing pickle’s __reduce__ hook to execute arbitrary code on torch.load(). Their analysis surfaced roughly 100 models on the hub carrying genuinely malicious payloads. No CVE fired, because there was nothing for a CVE to describe. The malice lived in the serialized weights. Model-specific scanning has since appeared to close that gap. Hugging Face runs ClamAV plus a pickle import scan on every file pushed to the Hub, statically disassembling the pickle’s opcode stream to flag dangerous imports. While they help, they are also already being evaded. In February 2025, ReversingLabs described nullifAI, two models that slipped past picklescan by compressing with 7z instead of ZIP and by corrupting the pickle stream immediately after the payload ran, so static analysis errored out on a file whose reverse shell would have already run. Hugging Face removed the models inside 24 hours and patched picklescan. That is the shape of the problem. Pattern matching scanners are a line that keeps moving, and each one answers whether a file resembles something known to be bad. None of them answers where the file came from.

“A tampered application image defaces a page. A tampered AI model artifact corrupts predictions at scale.”

AI is widening the attack surface in the same motion. Coding assistants suggest dependencies that never pass a human threat model, and that code gets containerized and shipped faster than review can keep up. The blast radius changed too. A tampered application image defaces a page. A tampered AI model artifact corrupts predictions at scale, poisons recommendations served to millions, or in the agentic case takes actions in production: API calls, tool invocations, spend. And when you consume a pre-trained model, you inherit every upstream decision about its training data and its security with zero visibility into any of them. Provenance stopped being a question about your application code. It became a question about the model, the agent, and the tooling that carries them.

But signing is not a checkbox. It is a chain. It only works if every link holds.

Why registry is the right layer

Operating the registry at the scale of Amazon ECR has taught us something that shaped how we think about supply chain security. Most teams don’t verify images. They verify addresses. An admission policy allows images from your registry account, push credentials belong to the pipeline rather than to people, and a scanner blocks critical CVEs. That stops a lot of attacks. What it can’t do is tell a good image from a bad one once it’s inside the boundary, because registry provenance is a claim about location, not origin. Anything that can write to the repository produces an image that looks legitimate: a leaked CI token, a misconfigured cross-account role, a compromised build step. Digest pinning tells you that you got the bytes you asked for, not that those were the right bytes to ask for. 

Every container image passes through a registry before it runs. It is the last system in the path that sees every artifact, knows who pushed it, and controls who can pull it. It already holds identity context, already enforces access policy, and already stores the metadata that describes what an image contains. The hard part of image signing is doing it consistently across every team and every pipeline without slowing anyone down. The registry is the only layer that can make it invisible.

“The hard part of image signing is doing it consistently across every team. The registry is the only layer that can make it invisible.”

Signing does not make forgery impossible. An attacker who fully compromises a trusted signing identity, stealing both the credential and the permission to sign, can produce a validly signed malicious image that passes verification. What signing does is shrink the attack surface. Without it, tampering anywhere in the path works, because nothing downstream checks. With signing and enforcement, none of it works unless the attacker compromises one narrowly scoped signer, and that rogue signature is an auditable event tied to an identity instead of an anonymous overwrite. Revoke the identity and the whole fleet stops trusting it in one change. Signing turns an invisible, unbounded problem into a scoped, attributable, revocable one.

The operational tax we set out to remove

Signing is a three-step process:

Sign: Generate a signature at build or push time, binding the image digest to a verifiable identity. The hard question is custody: who holds the private key, and how is it rotated and protected?

Verify: At pull time, and critically before the workload is admitted, check the signature against a trust policy which is a declared list of the identities you trust to have signed what you are about to run.

Enforce: A Kubernetes admission controller like Kyverno blocks any image not signed by a trusted identity from ever running. Signing without enforcement changes nothing.

Enabling signing comes with operational cost. Engineers had to install and configure client-side tooling like Notation CLI or Cosign, then own their signing keys, certificates, rotation schedules, and revocation lists, then build custom automation to wire signing into every pipeline. Across an enterprise with thousands of uniquely configured pipelines, that rollout took weeks to months. What we wanted to know was whether the registry itself could absorb the cost, so that signing could become a property of pushing an image rather than a project each team takes on. The answer to that question became Amazon ECR Managed Signing.

The mechanics are deliberately boring, which took some doing. You create a registry level signing configuration with up to ten rules, each pairing a signing profile with repository filters, and every matching push gets signed from then on.

Managed Signing answers the custody question by not giving you the keys. You configure a signing profile in AWS Signer, which pins the signing algorithm, a validity period, and the identity that appears in the signature. Signer keeps the certificate and the private key. This means no signing key ever sits in a repo, a runner, or a build log. Validity defaults to 135 months, so signatures won’t expire on you. Revocation is what you’ll actually use when you find out a build was compromised.

Then what gets signed, which is narrower than people assume. Signer signs a small Notary payload whose targetArtifact describes the image manifest: media type, digest, size. Not the image bytes directly. Because the signed material is content addressed, verification becomes a statement about exact bytes. The signature itself lands in the same repository as a detached OCI artifact, typed application/vnd.cncf.notary.signature, with a subject descriptor pointing at the image manifest digest. One image can carry signatures from several profiles as your trust requirements change. 

Signing happens asynchronously, which keeps Signer off the push path. A synchronous call would turn an availability dip or a throttle into a failed docker push for a developer, and it would put signing latency in front of every pipeline. The push commits first, and ECR calls SignPayload after. 

Verification and enforcement happen downstream, and the trust policy is where the whole design becomes legible. Your cluster operator writes it and imports it with notation policy import. It’s a short reviewable file:

{
"version": "1.0",
"trustPolicies": [
      {
        "name": "aws-signer-tp",
        "registryScopes": ["*"],
        "signatureVerification": { "level": "strict" },
        "trustStores": ["signingAuthority:aws-signer-ts"],
        "trustedIdentities": [
          "arn:aws:signer:us-east-1:111122223333:/signing-profiles/platform_images"
        ]
      }
  ]
}

That policy says a workload runs only if it carries a signature chaining to the AWS Signer root and produced by that specific profile. Admission does the work in order: resolve the reference to a digest, fetch the signature via OCI Referrers API, validate the envelope against its embedded certificate chain, walk that chain to the root in the trust store, check the signing identity against trustedIdentities, and check revocation. Revoking a profile makes verification fail wherever that profile is trusted. New admissions stop immediately and running pods pick it up when they’re next rescheduled. On EKS you get there with Gatekeeper and Ratify, or with Kyverno. Both paths use the AWS Signer plugin. Every link is checkable by the cluster itself, from the artifact plus a root certificate without asking the verifier to trust the registry it pulled from, or the pipeline that pushed.

Conclusion

Vulnerability scanning answers a question that mattered in the application era: what is broken inside this image? The AI era asks a harder one that scanning was never built to answer. Can you prove where this came from, and that no one touched it?

The cryptography was never the hard part. Making it the path of least resistance was. Sign, verify, and enforce, and let the registry carry the tax so your teams don’t have to.

To explore what’s referenced here, see Amazon ECR managed signing and signature verification on Amazon EKS.

The post Your container images are unsigned. In the AI era, that’s a ticking time bomb. appeared first on The New Stack.

When the image is the bottleneck: Machine learning changed what a container image looks like. A typical application ships in a few hundred MB and starts in seconds. A modern ML inference image carries a deep-learning framework, the CUDA stack, and sometimes the model weights. These images roughly reach 20 to 30 GB, with some even higher. On the GPU and accelerated instances these run on, pulling one of those images takes several minutes before the application can serve its first request: minutes during which provisioned accelerators are ready to process real work but waiting for images to be pulled.

We hit this bottleneck on a production ML platform running on Amazon EKS. The team needed pods ready within two minutes, but image pull alone consumed several minutes. Each pod pulled a roughly 30 GB container image on top of loading model data from a shared filesystem. The images were rebuilt on a regular cadence, so worker nodes faced cold pulls with no usable local cache. While the image was pulled, accelerators sat idle, autoscaling lagged demand, and request queues built up.

The natural first suspect for these large image pull times was the network or the registry. After all, 30 GB is a lot of data. However, profiling the image pull path showed neither was the bottleneck on accelerated instances with 100 to 400 Gbps of network bandwidth available. The real constraint was how the software used the hardware already available.

“The real constraint was how the software used the hardware already available.”

By rethinking the pull pipeline to leverage the network bandwidth, storage throughput, and compute these instances already had, we got those multi-minute pulls down to seconds. The improvements are available by default on EKS Auto Mode today, and we contributed the core changes upstream to containerd and the SOCI snapshotter. This is the story of how we dove into the internals of the image pull path, identified where time was being lost, and rebuilt those stages. It starts where we started: understanding what a large container image looks like and how it gets onto a node.

What a container image looks like at scale

A container image is not one file. It is a stack of layers plus a small JSON manifest listing them. Each layer is a tar archive of part of the filesystem (the base OS in one, CUDA libraries in another, your code in a third), gzip-compressed. For each layer, the manifest records a digest, a SHA-256 hash of the compressed bytes, so the node can prove it received exactly what was published. When the container runs, these layers are stacked and mounted together into a single unified filesystem.

Here is what real ML images look like when measured directly from their registry manifests:

Notice that layers within an image are not roughly equal in size. A single layer can account for more than half the total image, with individual layers often reaching 9 GB or larger, while the remaining layers are comparatively small. This size disparity has direct consequences for how long a pull takes, as the next sections explain.

Getting each of these layers onto a node involves six stages, and traditionally containerd, the industry-standard container runtime, performed most of these operations sequentially.

The stages of a pull

For each layer, containerd performs six operations in sequence. The first three are the download phase: fetch the compressed bytes from the registry over a single HTTP connection, verify the bytes by computing their SHA-256 and comparing it against the manifest digest, and write the compressed blob to local disk. 

The next three are the unpack phase: decompress the gzip archive, verify the decompressed content by computing a second SHA-256, and extract the files into the local snapshot directory managed by a containerd snapshotter, the component that stores and serves the on-disk representation of each layer. By default, containerd downloads up to three layers in parallel, but each layer uses a single connection, and unpacking remains strictly sequential across layers.

Two details are worth noting. Both SHA-256 checks mentioned above are mandated by the Open Container Initiative (OCI) specification, and on a multi-gigabyte layer, each of these hashes requires real work. The first is computed over the compressed bytes and proves the download was not corrupted or tampered with. 

The second is computed over the decompressed content. It gives the runtime a stable fingerprint of the layer’s actual filesystem data, which is what allows it to recognize shared layers across images and avoid redundant unpacking. Decompression is another deceptively expensive operation on large layers: gzip often triples a layer’s size on expansion, and because each block depends on the previous one, it runs on a single core. At the same time, the rest of the instance sits idle. 

Downloading layers can be parallelized, but the unpack sequence runs layer by layer. Layer two cannot begin until layer one finishes all six stages. The result is that at any given moment during a pull, the node is bottlenecked on only one resource: network bandwidth during download, CPU during decompression and hash verification, or disk throughput during extraction. The other resources sit idle, waiting their turn in the pipeline.

“Because layers are not equally sized, the single largest layer becomes the long pole in the pipeline.”

Because layers are not equally sized, the single largest layer becomes the long pole in the pipeline. A 10 GB layer that takes a minute to decompress on one core holds up the entire image, even if the other layers finish in seconds. Total pull time is effectively bounded by that one dominant layer moving through all six stages.

Existing approaches: working around the pull

There are well-known approaches to reducing image pull times. Each works around the bottleneck differently, whether by altering images, caching them, or relying on assistance from other parts of the stack.

Image size reduction: The most direct approach is to make images smaller through multi-stage builds, distroless base images, and stripping unused packages. For ML workloads, this hits hard limits. The GPU software stack alone (PyTorch, cuDNN, CUDA) imposes a compressed floor of roughly 3 to 4 GB that no build optimization can remove. Beyond that, ML images are assembled across organizational boundaries: a platform team provides the OS and drivers, a frameworks team adds deep-learning libraries, and researchers contribute application code and model weights. No single team controls the final artifact; model weights frequently end up baked in because serving frameworks expect local paths, and multi-stage builds yield only single-digit percentage savings on images whose bulk is irreducible.

Image caching (pre-pulling): Cache images on nodes by snapshotting image content into a volume and mounting it at provisioning time, so subsequent launches skip the pull entirely. This works for stable images but adds a dedicated pipeline stage for each image version. For ML workloads, the challenge goes beyond frequent rebuilds: a common pattern is a single pod consuming an entire node, so nodes scale in and out with each scheduling decision and every new node faces the full cold pull. This limits pre-caching to workloads with long-lived, static node pools.

“The GPU software stack alone (PyTorch, cuDNN, CUDA) imposes a compressed floor of roughly 3 to 4 GB that no build optimization can remove.”

Registry-side optimization: Some approaches serve image content remotely rather than pulling it to the node. Alibaba’s DADI, for example, presents container images as remote block devices that the node mounts on demand without a discrete pull step. This eliminates startup latency for workloads that access only a fraction of their image. Still, the container depends on the network throughout its lifetime and requires purpose-built serving infrastructure that may not be portable across providers.

Lazy loading: Instead of pulling the whole image up front, start the container immediately and fetch file content on demand. Projects in this space include eStargz and Nydus (which require converting the image to a new format) and AWS’s SOCI (Seekable OCI), which adds a seekable index alongside the unmodified image. These techniques work well when containers touch only a fraction of their data at startup. Still, ML images densely access the framework, CUDA libraries, and model weights before serving the first request, so nearly all the data ends up being fetched anyway. For these workloads, the pull pipeline itself needs to be faster.

Peer-to-peer distribution within the cluster: Tools like Dragonfly (a CNCF graduated project) and Spegel turn nodes that already have an image into seeders for nodes that need it, reducing registry egress and accelerating rolling deployments. For ML workloads, though, the first node still faces the full cold pull, and when images are rebuilt frequently, no node has the new version cached yet. P2P distribution complements rather than replaces improvements to the pull pipeline itself.

Fixing the image pull pipeline

Out of the six stages in the pull pipeline, we focused on two: downloading the layer blob from the registry, and unpacking the layers on the node. These are where the most time is spent and where the serialization cost is highest.

Since SOCI was already an open-source containerd snapshotter plugin with the plumbing to intercept and customize the pull path, it served as a staging ground where we could develop and validate these changes before contributing them upstream to containerd. 

Download: sharding a single layer into multiple requests

containerd traditionally uses a single HTTP connection to download each layer. We identified that splitting a layer into fixed-size chunks and fetching them concurrently over separate connections using HTTP range requests was significantly faster. At the same time, the containerd community also independently added parallel chunked download support in containerd 2.1, and we built on the same principle in the SOCI snapshotter.

However, there is one significant difference that allows us to keep the runtime’s memory footprint constant regardless of image size. The difference is where chunks live between arrival and final assembly. Our implementation writes each chunk directly to the local disk the moment it arrives, while containerd holds it in memory as the layer is assembled. This means the runtime’s memory stays flat whether you are pulling a 1 GB layer or a 15 GB layer, which matters on GPU nodes where system memory is shared with model weights and CUDA contexts.

Once download completes in seconds rather than minutes, the compressed layer hash that previously hid behind it becomes visible. Because the layer is now a complete file on disk rather than a stream consumed once, integrity verification and unpacking can proceed at the same time.

Unpack: All layers concurrently

After downloading, containerd decompresses and extracts each layer one at a time. This sequencing exists because in some filesystem backends, a later layer can overwrite files from an earlier one, so the order matters. The overlay snapshotter, which is the default on EKS and most Kubernetes clusters, sidesteps this constraint by keeping each layer in its own separate directory. The kernel mounts them together into one unified view only when the container starts. Because each layer extracts to an independent directory, unpacking one layer does not need to depend on another to finish.

With that established, we built an unpack path that decompresses and extracts all layers concurrently. The snapshotter detects whether the backend actually requires ordering and falls back to sequential if it does. For images with multiple large layers, total unpack time goes from the sum of all layers to roughly the time of the single largest one. We contributed this parallel unpack capability upstream to containerd v2.2, so the improvement is available to the broader community.

What this looks like in practice

With chunked parallel download, a large layer that previously took over a minute on a single connection finishes in single-digit seconds. When you then unpack all layers concurrently rather than sequentially, the full pipeline for a large ML image compresses from several minutes to well under a minute on instances with fast local NVMe storage. The machine spends its available compute and bandwidth actively pulling rather than waiting on serial stages. On larger images and instances with faster storage, the gains are more pronounced because the gap between available hardware capacity and what a single connection can use is wider.

What’s next

The two changes above address download and layer sequencing, but decompression of a single large layer remains serial. On a 64 vCPU machine, decompressing a layer that expands to 18 GB means a lot of compute sitting idle. Two opportunities stand out:

Parallel decompression within a single layer. Libraries like rapidgzip can locate block boundaries and inflate blocks across cores in parallel.

Parallelizable integrity verification. Today, the layer hash requires a single sequential read over the entire compressed blob after all bytes have landed. A tree-structured hash like BLAKE3 would allow computing the layer digest from independently hashed chunks so that verification could run in parallel with download rather than as a separate pass afterward.

Addressing these opportunities will squeeze out the last remaining serial stages in the pull path, bringing total pull time closer to what the raw hardware is capable of delivering.

Using parallel download and unpack with Amazon EKS

This image pull optimization is enabled by default on G/P/Trn instances with EKS Auto Mode. You can also benefit from this on other node types through one of two mechanisms:

• Native containerd 2.2: parallel download and unpack built into the runtime. Use when your node already runs containerd 2.2.
• SOCI snapshotter: parallel download and unpack with a memory-bounded download path. Use on older nodes without containerd 2.2 or memory-constrained instances.

EKS AL2023 and Bottlerocket AMIs that ship with containerd 2.2 do not enable this feature by default, but you can set the containerd config explicitly as shown below:

AL2023: add the containerd config through the nodeadm NodeConfig in user data:

apiVersion: node.eks.aws/v1alpha1
  kind: NodeConfig
  spec:
    containerd:
      config: |
        [plugins.'io.containerd.transfer.v1.local']
          max_concurrent_downloads = 20
          concurrent_layer_fetch_buffer = 16777216
          max_concurrent_unpacks = 5

Bottlerocket (K8s 1.36): Bottlerocket generates the same containerd config from its settings API, so set the equivalent keys in user data:

[settings.container-runtime]
  max-concurrent-downloads = 20
  concurrent-download-chunk-size = 16777216
  max-concurrent-unpacks = 5

SOCI snapshotter is bundled in the optimized EKS AMIs (AL2023 and Bottlerocket).

Bottlerocket: enable SOCI through EC2 user data:

[settings.container-runtime]
  snapshotter = "soci"
 [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]
  max-concurrent-downloads = 20
 concurrent-download-chunk-size = "16mb"
 max-concurrent-unpacks-per-image = 5

Tune chunk size and concurrency under [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]; the right values depend on your instance type and images.

AL2023: enable SOCI through the nodeadm FastImagePull feature gate, which switches image pulls to SOCI’s parallel-pull-unpack mode, and you can override the SOCI tuning parameters through user-data:

apiVersion: node.eks.aws/v1alpha1
  kind: NodeConfig
  spec:
    featureGates:
      FastImagePull: true

The image pull problem was never really about the registry, and it was never about needing faster hardware. The network, the storage, and the CPU were always there. When containerd’s pull pipeline took shape, images were measured in hundreds of megabytes, and the sequential approach served that world well. 

“The image pull problem was never really about the registry, and it was never about needing faster hardware.”

As AI and ML workloads pushed images past 20 GB, the gap between available hardware throughput and what the pull path was utilizing became impossible to ignore. For workloads running on the overlay snapshotter with high-bandwidth instances and fast local storage, parallelizing download and unpack closes most of that gap today. Other snapshotters and storage backends may have different constraints, and opportunities like parallel decompression and parallelizable integrity verification remain open.

We have been contributing these changes upstream because this is where they belong: in the runtime itself, available to everyone by default rather than locked behind additional software. Some of that work has already landed in containerd 2.2, and more is in progress. 

If you are working on container runtimes, image formats, or compression tooling and any of the open problems described here interest you, we would welcome collaboration. The faster we can collectively close the remaining serial stages, the sooner multi-gigabyte images stop being a deployment bottleneck for the entire ecosystem.

The post Pulling multi-gigabyte container images in seconds on Amazon EKS appeared first on The New Stack.

“AI kill switch” entered the public conversation because it gives people a simple way to talk about a complex fear. 

As AI systems become more autonomous and harder to evaluate with familiar operating assumptions, a clearly defined intervention capability sounds reassuring. If something starts behaving in a way that creates unacceptable risk, people want confidence that someone has both the authority and the mechanism to stop it. It’s the “kill switch.”

Recent reporting around OpenAI models escaping a sandboxed testing environment and reaching Hugging Face gave that concern a concrete example. CNBC reports that the incident helped trigger a bipartisan bill requiring certain AI companies to maintain the ability to shut down, throttle, or suspend their models, with the Department of Homeland Security given authority to order a slowdown or shutdown in cases involving potential catastrophic harm. 

The political reaction is understandable. When a new category of risk surfaces, especially one the public does not yet know how to evaluate, leaders look for a way to make an abstract concern into something actionable. In this case, that language has formed around shutdown authority.

People who operate large environments tend to hear a different question underneath the policy language. If a shutdown order arrives, what exactly gets shut down?

In a modern production environment, answering that question usually means tracing more than one system. An AI-enabled service may depend on endpoints, APIs, cloud resources, identity systems, package registries, data pipelines, workflow automation, logging tools, and downstream applications that act on model output. 

Some of those dependencies may belong to different teams. Others may sit outside the company entirely. A few may have started as experiments and later become part of a production path without receiving the same scrutiny as the original architecture. By the time the service is important enough to raise governance concerns, it may no longer resemble a bounded application with a single owner and a clean operating surface.

Writing shutdown authority into legislation is far simpler than carrying that decision through a production estate shaped by years of migrations, exceptions, integrations, acquisitions, temporary fixes, and team-level decisions. That implementation gap is where the issue becomes most relevant to infrastructure teams.

For the last several years, much of the AI safety conversation has focused on acceptable use, privacy, model behavior, and human-in-the-loop oversight. Those topics still deserve attention, especially as organizations formalize where AI may be used, which data can be shared, and how employees should evaluate generated output. 

“Writing shutdown authority into legislation is far simpler than carrying that decision through a production estate shaped by years of migrations, exceptions, integrations, acquisitions, temporary fixes, and team-level decisions.”

As AI moves deeper into production workflows, the discussion also needs to involve a more practical concern: when an AI-enabled system creates unacceptable risk, can the organization understand the affected environment well enough to constrain it quickly, consistently, and with evidence?

The phrase “kill switch” may drive the public discussion, but the practical answer lives in the systems surrounding the AI capability.

The limits of a single control

Emergency stops are the kind of control most people picture when they hear the phrase “kill switch.” They make sense in physical systems. Manufacturing equipment, industrial machinery, and certain safety-critical devices can be designed with direct shutdown mechanisms. Software estates already stretch that metaphor, and enterprise AI stretches it further.

The model may be the most visible part of the discussion, although it is rarely the full surface area. An AI assistant used in software delivery might have access to repositories, CI/CD tools, artifact stores, ticketing systems, secrets, test environments, and deployment workflows. An AI agent used in IT operations might read telemetry, recommend remediation, open change requests, call automation scripts, or modify infrastructure through approved orchestration paths. 

Stopping one part of that chain can leave other paths untouched. Turning off a service may not revoke the credentials it uses. Suspending inference may leave downstream systems acting on outdated outputs. Interrupting the wrong dependency can create a separate service incident while the original risk remains only partly contained. Anyone who has worked through a security incident, emergency patch cycle, or major outage knows how quickly a clean decision turns into a sequence of technical tradeoffs.

A credible response plan must account for the system as it exists now, not as it looked during an architecture review months earlier. Infrastructure teams bring useful skepticism to that exercise because they are used to tracing scope, access, ownership, dependencies, and verification paths under pressure. They also know that many environments contain a gap between documented intent and production behavior.

Once AI is embedded in business workflows, those operational details become part of the governance conversation. They expose the places where policy language has moved faster than the infrastructure knowledge needed to make policy executable.

Before containment comes discovery

Much of the public conversation assumes organizations know where AI is running, which is a generous assumption in many enterprise environments.

AI can enter an enterprise through obvious channels, such as internally approved model providers or purpose-built applications. It also arrives through less visible paths. A SaaS product adds an AI feature. A development team experiments with an open source model. A hosted API gets attached to an internal tool. A vendor introduces an AI capability inside software the company already approved. Over time, the line between an “AI system” and a system that happens to use AI becomes harder to define.

This is where the kill switch metaphor starts to show its limits. If the relevant systems are discovered during the response, the team is already behind. Dependency questions, business impact, access paths, and evidence collection all become harder when the basic inventory is still being assembled.

“If the relevant systems are discovered during the response, the team is already behind.”

Infrastructure teams have seen versions of this problem before. During incident response, a service thought to be isolated turns out to have undocumented consumers. During a cloud migration, a supposedly unused integration is suddenly linked to a business process. During an audit, ownership records, configuration data, and actual operating conditions refuse to line up cleanly. AI adds a new category of concern, but the underlying visibility problem is familiar.

The challenge quickly expands beyond the model itself. Teams need to understand which systems call external models, where generated content influences workflows, which accounts and automation paths sit between a recommendation and an action, and how third-party AI capabilities have found their way into the environment.

Most discussions start with how to stop risky AI behavior. In many environments, the more revealing question comes earlier: can the organization produce a reliable picture of where AI touches the estate, which systems depend on it, and which workflows would keep moving if access changed?

Containment depends on the state of the system

Most infrastructure teams know that containment is less a single action than a set of operating conditions. The difficult work happens before the incident, when teams decide what should change if risk reaches a level that requires intervention.

Under ordinary circumstances, a service operates with a defined set of permissions, connections, dependencies, and logging requirements. Under restricted operation, selected assumptions change while investigators preserve evidence and determine whether the risk has been contained. A team might turn off endpoints, suspend integrations, limit external network access, revoke or rotate credentials, increase logging, or isolate workloads while the situation is being investigated.

The details vary because environments vary. That is exactly why generic answers tend to fall apart. A useful containment plan must match the systems it is intended to govern, including the dependencies that surround them and the business processes that rely on them.

Inventory sounds mundane until a response effort depends on it. In my work with infrastructure and compliance teams, I’ve repeatedly seen organizations struggle to maintain an accurate picture of their environments as cloud resources, Kubernetes clusters, SaaS services, and AI projects multiply faster than governance processes can track them during normal operations, which creates audit and support headaches. During a containment event, those same blind spots slow investigations, complicate dependency analysis, and make evidence harder to produce.

Governance eventually reaches production

Governance efforts often begin with documentation. Committees are formed to define responsibilities, agree on escalation paths, and establish a common language for discussing risk before an incident forces the issue. 

The conversation shifts once someone asks whether the control can be demonstrated. A document can describe who has authority to suspend an AI-enabled service. Still, it cannot turn off an integration, revoke a credential, increase logging, or prove that a set of systems entered a restricted condition. A risk register can identify a containment scenario, although it cannot document which nodes changed, when they changed, and whether they remain aligned with the required configuration.

Security, compliance, and infrastructure teams know this gap well. It appears in patching programs, configuration baselines, incident response exercises, supply chain reviews, and disaster recovery planning. Written controls tend to be cleaner than the real-world environments they describe. Production systems reflect years of accumulated decisions, exceptions, migrations, temporary fixes, acquired assets, and workarounds that may outlive the original reasons they existed.

“Written controls tend to be cleaner than the real-world environments they describe.”

AI increases urgency because some systems are becoming more autonomous and more connected to business workflows. It also adds outside pressure. When an incident becomes visible enough to prompt legislative action, boards and customers start asking sharper questions. A company may be able to point to an AI policy, but boards, customers, and regulators eventually want to understand exactly how that policy translates into action.

If leadership declares an AI-enabled workflow should be restricted, the discussion moves quickly from oversight to execution. Teams need to know where to intervene, which systems are affected, who owns the required changes, how completion will be verified, and what evidence remains once the response is over.

A vague answer may pass during experimental stages but becomes much harder to defend once AI is embedded in production services, regulated workflows, customer-facing systems, or environments connected to critical business operations.

Why a mandate will not solve the estate problem

A federal shutdown authority, if enacted, would place legal pressure on a narrow class of powerful AI providers. It would not remove the implementation burden for organizations that adopt, integrate, fine-tune, host, or embed AI systems inside their own environments.

Even if a major AI provider can throttle or suspend a model, each enterprise still must understand its own exposure in the context of its applications, workflows, dependencies, and operating assumptions. Which applications depend on that model? Which workflows fail open or fail closed if access is restricted? Which internal systems contain cached outputs, calculated decisions, or agent-created changes? Which business processes need manual fallback when an AI service is unavailable?

Policy debates often treat AI control as if the decisive action happens at the model layer. Sometimes it will; however, in many business environments, the risk will live in the connections that surround the model. A hosted AI service may be suspended while local workflows, scripts, integrations, and access tokens continue following their last known configuration.

“A serious AI containment strategy has more in common with mature infrastructure management than with an emergency stop button.”

A serious AI containment strategy has more in common with mature infrastructure management than with an emergency stop button. It requires an up-to-date inventory of AI-adjacent systems, a map of dependencies and access paths, predefined restricted conditions for high-risk services, tested procedures for applying those conditions, and evidence that changes were enforced. Ownership also needs to be clear, since fast action becomes difficult when authority is scattered across teams.

The work is less exotic than the public conversation can make it sound. AI-enabled systems still need to be managed as production systems with real dependencies and business impact.

Infrastructure teams belong earlier in the conversation

Spend enough time running infrastructure, and you develop a complicated relationship with documentation. Most organizations have diagrams, inventories, and governance processes, and all of them serve a purpose. The challenge is that production systems keep evolving long after those artifacts are created. Acquisitions introduce systems that do not fit cleanly into existing models. Applications gain integrations nobody anticipated during the original design process. Temporary exceptions become permanent. Cloud resources intended to live for a week are still running after a year.

Most of this happens for defensible reasons, usually in support of uptime mandates, delivery pressure, customer needs, or business continuity SLAs. The result is that operational knowledge becomes dispersed across people (some of whom will inevitably have moved on), tickets, runbooks, monitoring systems, and memory rather than living neatly in one place.

Infrastructure teams spend their days tracing dependencies, untangling ownership questions, and figuring out how systems behave outside a design review. Bringing that perspective into AI governance conversations early can prevent containment plans from depending on assumptions that did not translate into production. It also helps organizations understand the difference between disabling a model, restricting access to a service, isolating a workload, and preserving evidence during an investigation.

Scale complicates things further. A manual action that works effectively for ten systems may fail across hundreds or thousands. A change one expert can perform during business hours may become fragile if that person is unavailable when an event occurs. A runbook that looks adequate in a tabletop exercise may not survive a live environment where dependencies have changed, and the current ownership is unclear.

The phrase “kill switch” will probably remain part of the public debate because it is simple, memorable, and familiar. Practitioners do not have to accept the metaphor literally to leverage the attention it creates. They can redirect the conversation toward more useful questions: what restricted operation would mean for a given service, which dependencies would have to change, which controls can be applied reliably, which steps remain manual, and how the organization would prove the response worked.

These questions are less dramatic than a big red button, but their answers are also most likely to improve readiness.

Control starts before the incident

The Hugging Face incident gave the industry a vivid story, and Washington responded with the language of shutdown authority. That reaction is understandable. Leaders want mechanisms that sound equal to the risk, especially when the public conversation moves faster than the technical details can be explained.

By the time an organization begins thinking about containment, much of the hard work should already be done. Teams should already understand what is running, who owns it, what depends on it, and how changes will ripple through the environment.

AI can reduce certain workflow bottlenecks, but it also exposes weak inventory, unclear ownership, and brittle operating assumptions faster than many teams are prepared to handle. A future incident will not pause while teams locate assets, clarify ownership, identify credentials, or discover that a service dependency was never documented.

The current debate may be framed around new kill switches. For most organizations, the more useful work starts with building and maintaining an accurate picture of the systems, dependencies, and workflows that already exist across the estate.

The post The “AI kill switch” assumes you know what you are trying to shut down appeared first on The New Stack.

Consider a platform team managing a shared GPU cluster with a mix of B200s, H100s, and recently added B300s. Every Monday morning, the on-call engineer finds a queue of pending jobs from the weekend. Training workloads are stuck because they landed on H100s and triggered Out-Of-Memory (OOM) errors. Inference jobs sit idle because the small MIG (Multi-Instance GPU) slices are exhausted, even though larger slices sit empty right next to them.

Their fix? A 200-line Bash script running every 30 minutes to reconfigure MIG profiles, reschedule stuck jobs, and send a Slack alert when it succeeds, or a PagerDuty alert when it fails.

The root of the problem

Here is what was actually broken: Kubernetes treated every GPU as an identical unit. The resource limit nvidia.com/gpu: 1 was the extent of its awareness. The scheduler had no idea if it was handing a pod a 192GB B200 or an 80GB H100. A training job requiring 150GB of VRAM would land on an H100 and immediately OOM, while B200 nodes sat completely idle nearby.

“Kubernetes treated every GPU as an identical unit.”

The industry’s accepted “fix” relied heavily on node labels, taints, tolerations, and separate node pools per GPU type. Every workload manifest hardcoded hardware assumptions. Adding a single new GPU generation meant updating 40 different Helm charts.

The MIG illusion

MIG made this worse. MIG slices a single GPU into smaller, isolated partitions, each with dedicated memory and compute. Instead of one inference job monopolizing a B200, you can run seven smaller jobs on the same card.

In theory, this sounds great. But when you enable MIG in Kubernetes, each profile becomes a separate, rigid resource type (e.g., nvidia.com/mig-1g.10gb, nvidia.com/mig-3g.40gb). There is no fallback logic. You cannot instruct a job to “try a small slice first, and use a large one if nothing else is free.” When small slices run out, jobs sit pending, while large slices go to waste.

“When small slices run out, jobs sit pending, while large slices go to waste.”

This inefficiency was accepted as the cost of running GPU workloads on Kubernetes. Then, Kubernetes 1.34 shipped.

Dynamic Resource Allocation (DRA)

Kubernetes 1.34 introduced Dynamic Resource Allocation (DRA), fundamentally changing the scheduling model. GPU drivers now publish structured data. Instead of requesting a generic nvidia.com/gpu: 1, workloads can express explicit intent using Common Expression Language (CEL):

1. “Give me an H100 or better with at least 40GB of memory.”
2. “Give me a MIG slice: small if available, medium if not, or a full GPU if necessary.”
3. “Give me four GPUs connected via NVLink.”

Example 1: Hardware and memory requirements

“Give me an H100 or better with at least 40GB memory.”

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: h100-or-better-40gb
spec:
  spec:
    devices:
      requests:
        - name: gpu
          deviceClassName: gpu.nvidia.com
          count: 1
          selectors:
            - cel:
                # Attribute names are illustrative.
                # Your NVIDIA DRA driver must expose these fields.
                expression: >
                  device.attributes["gpu.nvidia.com"].memory >= quantity("40Gi") &&
                  device.attributes["gpu.nvidia.com"].generation in ["H100", "B200", "B300"]
---
apiVersion: batch/v1
kind: Job
metadata:
  name: training-job-h100-or-better
spec:
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
        - name: gpu
          source:
            resourceClaimTemplateName: h100-or-better-40gb
      containers:
        - name: trainer
          image: nvcr.io/nvidia/pytorch:24.12-py3
          command: ["python", "train.py"]
          resources:
            claims:
              - name: gpu

Example 2: Flexible MIG fallback

“Give me a MIG slice: small if available, medium if not, or full GPU if needed.”

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-prefer-small-then-medium-then-full
spec:
  spec:
    devices:
      requests:
        - name: gpu
          deviceClassName: gpu.nvidia.com
          count: 1
          selectors:
            - cel:
                # Prefer any acceptable MIG profile or full GPU
                expression: >
                  device.attributes["gpu.nvidia.com"].profile in [
                    "mig-1g.10gb",
                    "mig-2g.20gb",
                    "mig-3g.40gb",
                    "full-gpu"
                  ]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-service-flexible-gpu
spec:
  replicas: 2
  selector:
    matchLabels:
      app: inference-service
  template:
    metadata:
      labels:
        app: inference-service
    spec:
      resourceClaims:
        - name: gpu
          source:
            resourceClaimTemplateName: mig-prefer-small-then-medium-then-full
      containers:
        - name: inference
          image: nvcr.io/nvidia/tritonserver:24.12-py3
          args: ["tritonserver", "--model-repository=/models"]
          resources:
            claims:
              - name: gpu

Example 3: Topology constraints

“Give me 4 GPUs that are NVLink-connected.”

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: four-nvlink-connected-gpus
spec:
  spec:
    devices:
      requests:
        - name: gpus
          deviceClassName: gpu.nvidia.com
          count: 4
          selectors:
            - cel:
                # Attribute names are illustrative.
                # Some NVIDIA DRA setups may model this through ComputeDomains.
                expression: >
                  device.attributes["gpu.nvidia.com"].fabric == "nvlink"
      constraints:
        - requests: ["gpus"]
          matchAttribute: "gpu.nvidia.com/nvlinkDomain"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: distributed-training-nvlink
spec:
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
        - name: gpus
          source:
            resourceClaimTemplateName: four-nvlink-connected-gpus
      containers:
        - name: trainer
          image: nvcr.io/nvidia/pytorch:24.12-py3
          command:
            - torchrun
            - --nproc_per_node=4
            - train.py
          resources:
            claims:
              - name: gpus

The engineering takeaway

This architecture requires one manifest. It eliminates fragile node selectors and the need to duplicate job definitions for every new hardware generation. As GPU clusters become increasingly heterogeneous, mixing H100s, B200s, B300s, and whatever silicon drops next, the old integer-based scheduling model breaks down. DRA represents Kubernetes finally maturing to support the nuanced realities of production AI workloads.

The post Say goodbye to K8s GPU pain: How DRA changes everything appeared first on The New Stack.

A developer supervising four coding agents has four changes in flight at once, each in its own git worktree. That isn’t an exotic setup anymore: Anthropic’s documentation now treats a worktree per session as the default way to run agents in parallel, and what was an expert workflow two years ago is the recommended starting point today.

The branches themselves aren’t new. Git made them cheap 20 years ago so developers could isolate changes and work on several things at once, but in practice a developer switched between branches and shipped one change at a time. That kept everything below the code layer singular: one continuous integration (CI) queue, one staging environment, one database everyone tested against. The number of changes contending for those shared resources was capped by headcount, and before agents, only larger teams ever hit the cap.

“Coding agents removed the cap. The branch can no longer stop at the code layer.”

Coding agents removed the cap. Those four branches are no longer something one developer rotates through. They are four active changes moving toward merge in parallel. The gap becomes unworkable: branching is free at the code layer and missing everywhere below it. Each change needs to exist all the way down the stack, not as a diff in a directory but as a running, testable version of the system. The branch can no longer stop at the code layer.

Parallel until the first shared resource

Code branches in milliseconds. A worktree gives each agent a private copy of the repository for the cost of a checkout, and 10 agents can work side by side without seeing each other’s edits.

The output shows up downstream. Telemetry from Faros AI across more than 10,000 developers found that teams with high AI adoption merge 98% more pull requests while review time grows 91%. Nothing downstream of code generation was sized for that arrival rate.

Then each change needs to run. There is one staging cluster, one seeded database, one message queue, one set of dependent services, and every branch that reaches this floor stops being parallel. Four agents produce four candidate changes in an afternoon, and all four line up behind the same shared environment to find out whether they work.

The queue is more expensive than it looks, because agents don’t wait well. An agent blocked on an environment either sits idle holding a stale view of the system or plows ahead validating against mocks, and the developer supervising it context-switches away. By the time the shared environment frees up, the cheap part of the work has to be partially redone.

The bottleneck isn’t code generation, and it isn’t review capacity alone. It’s the first shared resource a change touches, because a branch that can’t run is a branch that can’t be trusted.

“The bottleneck isn’t code generation, and it isn’t review capacity alone. It’s the first shared resource a change touches.”

A branch is a delta, not a copy

The way out is to stop treating branching as something git does and start treating it as something every layer does. Branch-based development names the pattern: each layer of the stack offers a cheap, instant, disposable branch primitive, so a change can exist end to end without duplicating anything it didn’t touch.

The mechanic is the one git established, and everyone has been living on for two decades: branches are cheap because they share everything unchanged and carry only the delta. The rest of the stack has been relearning that idea layer by layer ever since — share by default, isolate what changed.

Naming the pattern matters because each layer discovered it separately and called it something different. Worktrees, pipeline caching, preview deploys, database branching, and environment sandboxing sound like five unrelated features. They’re the same idea applied at five layers, and seeing that changes what you ask of the layers that lack it.

The upper layers learned this years ago

CI absorbed the lesson a decade ago. Every branch gets its own pipeline run on a shared runner pool, with build caches doing the copy-on-write work of reusing unchanged artifacts. Nobody provisions a build system per branch, and nobody queues behind a single global build anymore.

The front end followed. On Vercel, every push to a non-production branch gets its own preview deployment by default; Netlify works the same way, and the branch itself is one immutable build plus routing on shared hosting infrastructure. Reviewers stopped asking whether a change works on someone’s laptop, because the change is already running somewhere.

Both cases have the same shape: the expensive machinery is shared, the branch is thin, and creating one is cheap enough that nobody thinks about it. That’s what a layer feels like once it has a branch primitive.

Each of these primitives also changed behavior once it arrived. Per-branch CI made it normal to run the full test suite on every push instead of nightly. Preview deploys made it normal for a product manager to click through a change before merge. Cheap branches don’t just remove a queue; they raise the bar for what gets checked before merge.

The data layer was supposed to be the hard case

Databases carry state, so conventional wisdom said branching would never work there. Then Neon, PlanetScale and Xata shipped it anyway, and Neon’s documentation now makes the parallel explicit: branch your data the same way you branch your code.

A database branch is a copy-on-write view over shared storage pages, created in seconds regardless of how large the database is. Schema migrations and risky data changes get validated against production-shaped data instead of a stale seed script, and the branch disappears when the work merges.

“If the layer with the most state can hand out branches in seconds, statelessness was never the real requirement.”

The data layer matters to this story because it removed the best excuse. If the layer with the most state can hand out branches in seconds, statelessness was never the real requirement. Whatever is still unbranched is unbranched by choice.

The runtime is the last layer to learn the trick

The microservices runtime resisted longest because it looks nothing like a file tree. It has live traffic, a service graph and dozens of moving dependencies, and the naive branch, a full copy of the environment, is so expensive that most teams concluded branching did not apply here.

The copy-on-write move works anyway. Run one shared, stable version of the system that is continuously deployed from main. For each change, deploy only the services the change touches as a lightweight ephemeral environment, and route each test request through the changed services while everything else falls through to the shared stable versions. The environment branch costs roughly what the changed services cost, which is why one can exist for every change an agent produces.

Routing is the part that sounds exotic and isn’t. A request tagged with a label gets steered to the changed service versions at each hop, propagated through the call chain the same way trace context already flows through most instrumented systems. The shared stable environment plays the role of main, the changed services are the delta, and the label is the pointer that assembles a coherent view of the system per request.

This isn’t a hypothetical architecture. Uber built SLATE to give each developer an ephemeral environment routed against shared production-grade dependencies because contention over staging could not keep up with its developer count.

What an agent-native stack means

Put the layers together and a different development model appears. An agent picks up a task, and the change gets a worktree, a pipeline run, a preview, a data branch, and a running environment from the start. Validation stops being the scarce resource that serializes everything upstream of it.

Teams are already composing the lower layers. Bitso, a crypto exchange with 250-plus engineers, pairs an environment branch with a database branch for each change, so the runtime delta and the data delta travel together and shared staging stays out of the critical path.

That end-to-end branch is what the phrase agent-native software development lifecycle should mean. Not agents wired into yesterday’s pipeline, but a stack where any change, human or machine, can exist at every layer for as long as validation takes and disappear afterward.

The payoff compounds with agent count. When the branch primitive at every layer is a delta over something shared, validation concurrency scales with cluster capacity instead of with budget, and the number of changes a team can prove correct per day rises with the number it can generate. That is the ratio that decides whether agent adoption shows up as shipped software or as a longer queue.

The audit is cheap to run. Follow one change from worktree to validated and note the first layer where it waits on something shared. That’s where your stack stops branching. 

For most teams, the answer is the runtime, and if it’s yours, Signadot is a practical place to start.

The post Anthropic recommends a git worktree per agent. Your runtime infra makes that a problem. appeared first on The New Stack.

Every product team is chasing the same moment: The user opens a page and thinks, this understands me.

A shopper who loves floral prints should see more floral prints. A user who follows local politics should open their app to see news about local politics. A job candidate who keeps clicking remote roles should not keep getting shown in-office jobs.

That is not a niche feature anymore. It is the baseline expectation. Users decide quickly whether a product system understands them, and they rarely care whether the failure came from search, recommendations, merchandising rules, or stale data.

Here is the uncomfortable truth: Most teams do not have a personalization quality problem. They have a personalization architecture problem.

Personalization is not a widget bolted onto search. It is a ranking decision. The system has to decide, for this user and this request, what deserves the next slot. That means weighing the user, the item, the context, and the business goal at the same time. In many stacks, the ranking layer is the one place that cannot see all of those signals together.

The hard part is not collecting signals. The hard part is combining them while the user is still there.

Why personalization is hard in the first place

To put the right item in the right slot, a system has to understand several things at once:

• Intent: What is the user asking for right now?
• Item quality: What does each candidate actually contain or represent?
• User history: What has this person clicked, bought, read, watched, or ignored?
• Availability: Is the item in stock, fresh, nearby, legal to show, or ready to ship?
• Business priority: What should the business promote, protect, or de-emphasize?
  

Those signals often disagree. The most relevant item may not be the most profitable. The most profitable item may be out of stock. The user may say “running shoes,” but their behavior says “trail running, wide fit, under $120.”

They also move on different clocks. Product attributes change slowly. Inventory and price can move throughout the day. Preferences shift with every click. External context — weather, breaking news, a championship game, a cultural moment — can matter without warning.

Personalization means folding all of that into one ordered list, on every request, in milliseconds. The signals themselves are not the bottleneck. Query-time ranking is.

The usual stack makes the problem harder

Most personalization systems are assembled from tools that were each designed for one slice of relevance.

Keyword search engines are excellent at lexical matching. They are good when the query language and catalog language line up. But shoppers, readers, and job seekers rarely speak in neat index terms. You indexed “athletic performance running footwear”; they typed “running shoes.” Synonym rules can help, but they do not scale gracefully across long-tail language, changing catalogs, and new user behavior.

Vector databases start from the opposite side. They are good at semantic similarity: “Find me things like this.” That is powerful, but nearest-neighbor search is not the same thing as personalization. Real ranking has to blend semantic similarity with live behavior, stock, price, margin, freshness, eligibility, and business rules.

Re-rankers, recommendation services, feature stores, and rule engines are usually added to glue everything together. That is where fragmentation creeps in.

When retrieval and ranking live in separate systems, the ranker often works from a partial, stale, or precomputed view of the world. Click history, session context, and the user’s live preference vector arrive too late. Business rules become filters or overrides instead of ranking signals. Fresh inventory or price changes require coordination across multiple systems.

Every hand-off adds latency. Every boundary creates another place for signals to drift. Every “quick rule” becomes another hard constraint that can accidentally turn “show the closest match” into “show nothing.”

“Every hand-off adds latency. Every boundary creates another place for signals to drift.”

The deeper issue is a timing assumption. Many architectures were built around offline ranking: process the catalog, compute scores in a batch job, and serve those scores until the next rebuild. That works when preferences are stable. It breaks when the most valuable signal is the click that happened two seconds ago.

What changes when ranking happens in one real-time pipeline

A real-time personalization architecture treats retrieval, ranking, and inference as one serving problem.

That is the core idea behind Vespa’s approach: Text search, vector similarity, structured filtering, ranking expressions, tensor computation, and model inference can live inside one query pipeline. Instead of retrieving somewhere, enriching somewhere else, and ranking at the end, the system can rank with the relevant signals while it is still deciding what to return.

That architectural choice changes the shape of the problem.

1. Retrieval is hybrid from the start

Lexical search, semantic search, and structured filtering can run together instead of being reconciled after the fact. A product query can combine text, embeddings, filters, session behavior, and item attributes in one request.

That matters because personalization is rarely one signal. The user’s query still matters. So does semantic similarity. So do category, availability, price, and business constraints. Hybrid retrieval keeps those signals in play before ranking starts.

2. Ranking can express the actual objective

A personalization score should not be trapped inside one similarity function. It should be a formula that reflects the product’s goals.

That formula might combine BM25, vector similarity, user affinity, stock level, margin, popularity, discount depth, freshness, rating, distance, or a weather term. Some of those signals need normalization first. Some should matter only for certain categories or users. Some should be tested as weights.

The important part is that they are all terms in the same ranking expression, not scattered across services.

A simplified version might look like this:

final_score =
    0.30 * lexical_relevance +
    0.25 * semantic_similarity +
    0.25 * user_affinity +
    0.10 * availability +
    0.10 * business_priority

In production, the formula can be more nuanced. But the principle is simple: personalization, relevance, and business logic belong in the same scoring decision.

3. Model inference can run where the data lives

Some signals should come from learned models rather than hand-tuned rules: propensity to buy, churn risk, quality prediction, fraud risk, query classification, or a learned-to-rank model.

When inference runs in the serving path, those model outputs can become ranking features instead of delayed batch scores. That reduces the need to ship data to a separate inference service, wait for a response, and stitch the score back into ranking.

4. Updates become immediately useful

“Real time” should not mean “after the next index rebuild.” If inventory changes, stock should be rankable immediately. If a user clicks two yellow dresses, “yellow” should matter on the next request. If a merchandising team adjusts a ranking weight if the weight is exposed as a query-time input, the experiment should start producing useful feedback right away.

That is the difference between personalization as a nightly job and personalization as a live ranking decision.

Tensors make the personalization concrete

The most useful mental model is simple: represent the user and the item in the same feature space, then rank by how well they match.

In Vespa, tensors make that practical. A tensor can be a scalar, a dense vector, a sparse map of feature-weight pairs, a matrix, or a more complex structure. That means the same framework can represent semantic embeddings, product attributes, user preferences, business objectives, and model features.

For example, each item can carry a sparse feature tensor:

{
  "floral": 0.90,
  "yellow": 0.70,
  "short_sleeve": 0.80,
  "crew_neck": 0.65
}

Each user can carry a tensor with the same feature names:

{
  "floral": 1.00,
  "yellow": 0.37,
  "short_sleeve": 0.33,
  "crew_neck": 0.31
}

Because the two tensors share a shape, personalization becomes a dot product: multiply matching features, sum the result, and use that score inside ranking.

In a Vespa rank profile, the core expression is compact:

# schema: item attributes stored as a sparse tensor
field item_features type tensor<float>(feature{}) {
    indexing: attribute | summary
}
 
# rank profile: the user's live preferences arrive as a query tensor
rank-profile personalized {
    inputs {
        query(user_features) tensor<float>(feature{})
    }
    first-phase {
        expression: sum(query(user_features) * attribute(item_features))
    }
}

That one expression is not the whole ranking function. It is the personalization term. BM25, vector similarity, stock, margin, freshness, distance, or a model score can be added as other terms with their own weights.

The user tensor is where real-time behavior becomes powerful. Click a floral item, and the “floral” weight rises. Click two yellow items, and “yellow” rises; the application feeds click events into the user profile. The next query can use those updated preferences immediately, without waiting for a nightly profile build.

Business goals stop fighting personalization

In fragmented stacks, business rules often become blunt instruments: boost this category, hide that brand, force these items to the top, filter these out. That can satisfy a short-term merchandising goal while damaging relevance.

When business logic is part of the ranking expression, it can be more subtle. You can boost overstocked inventory without ignoring intent. Promote umbrellas when rain is forecast without turning every search into an umbrella search. Give new sellers a small exploration boost. Prioritize destocking before a new product line launches. Surface team merchandise during a championship run.

“When business logic is part of the ranking expression, the user still gets relevant results. The business still influences outcomes.”

The user still gets relevant results. The business still influences outcomes. The difference is that both are expressed as ranking signals instead of competing systems.

That also makes experimentation easier. A merchandising or growth team can test weights, traffic splits, and ranking profiles without asking engineering to rewrite the whole pipeline. Relevance becomes a controllable growth lever rather than a fragile side effect.

The same pattern applies beyond commerce

The examples above are easy to picture in apparel, but the architecture is not commerce-specific. Personalization is the same ranking problem in many products:

• Content feeds: Blend topic affinity, freshness, engagement, creator quality, and business rules.
• News: Rank by reading history, topic interest, locality, freshness, and source diversity.
• Jobs: Match candidate preferences such as remote work, seniority, compensation, location, and tech stack against role attributes.
• Geo search: Treat distance as one normalized ranking term alongside relevance, quality, and preference.
• Video and audio: Combine embeddings, viewing history, metadata, freshness, and learned ranking models.

Different domains need different features. The architecture pattern is the same: retrieve candidates, rank with the signals that matter, update those signals as behavior changes, and keep the decision close to the data.

Scale doesn’t have to be the trade-off

The natural concern is that a more expressive ranking system must be slower. In practice, that does not have to be true.

Vespa was built for large-scale serving from the beginning: billions of documents, high query volume, and low-latency ranking. The reason this works is multi-stage ranking. The system does not run the most expensive logic across every possible result. Instead, it uses a fast first phase to narrow the candidate set, then applies more precise ranking to the smaller group that remains.

For example, a cheap first phase narrows a huge candidate set. Then, once the candidate set is smaller, Vespa can apply full-precision scoring, richer tensor operations, business logic, and model inference where they matter most.

The result is a practical balance: speed across the full corpus, accuracy in the final ranking, and enough flexibility to personalize each query without turning the serving stack into a chain of fragile services.

What’s next

Personalization is not failing because teams lack data. Most teams already have plenty of signals: query intent, clicks, product attributes, inventory, margin, freshness, location, and business priorities. The harder problem is that those signals often live in different systems, move at different speeds, and arrive too late to influence the final ranking decision.

That is why personalization should be treated as a ranking problem. When retrieval, ranking, personalization, and business logic are split across separate systems, the ranker is forced to work with stale or incomplete context. The user moves faster than the architecture can respond. Every new signal becomes another integration project.

A unified real-time ranking pipeline changes that. User behavior, item attributes, semantic similarity, lexical relevance, inventory, and business goals can all become parts of the same scoring function. Tensors make those signals directly comparable and usable at query time. Instead of bolting personalization onto the end of the system, personalization becomes part of the decision the engine makes for every query.

The goal is simple: rank each result with the best context available, at the moment the user asks. That is when personalization stops feeling like a feature and starts feeling like relevance.

The post Personalization is a ranking problem — architecture makes it work appeared first on The New Stack.

The AI ecosystem is drowning in tutorials on how to build a retrieval-augmented generation (RAG) app in five minutes. The pitch is appealing but flawed: Chunk a document, run it through an embeddings API, load it into a vector database, and slap a UI on top. This setup works locally. It might even survive a beta test with friendly users. But launch it into a production-grade B2B SaaS environment, and the architecture collapses.

Enterprise applications don’t handle neat, static files. They ingest dynamic, unpredictable streams of live data bound by strict legal and compliance constraints. Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.

“Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.”

Here is exactly what breaks when a naive RAG setup hits production, and the architectural trade-offs needed to fix it.

Bottleneck 1: the synchronous ingestion trap

Synchronous data ingestion is the most prevalent architectural flaw in new AI products. A user uploads a 500-page compliance manual. The client makes a POST call to a web server, which parses the document, splits the text, iterates over a sequence of synchronous API calls to OpenAI or Cohere for vectorization, and writes those vectors to the database.

This approach introduces two critical failures:

• Timeouts: A 500-page document rarely finishes processing within standard HTTP timeouts (30 to 60 seconds) while waiting for the embedding API.
• Cascade failures: If the system hits rate limits or latency spikes, the entire ingestion operation fails, throwing a 500 error and losing the user’s document.

The fix: the batched fan-out pipeline

Production-grade AI pipelines require persistent events rather than simple HTTP calls. However, sending the whole 500-page document to be processed by one consumer from Kafka or RabbitMQ is a mistake. If a consumer spends 10 continuous minutes generating embeddings, it misses its broker heartbeat. Assuming the worker died, the broker kills the consumer and triggers a partition rebalance, creating an infinite loop of duplicated work and stalled processing.

“Synchronous data ingestion is the most prevalent architectural flaw in new AI products.”

Conversely, granular chunking, where every chunk becomes an individual Kafka message, launches a self-inflicted denial-of-service (DoS) attack on downstream services. A document with 1,500 chunks generates 1,500 individual messages. This instantly exceeds upstream requests per minute (RPM) limits and floods the pipeline with network overhead.

The engineering sweet spot is a batched fan-out approach:

• Asynchronous uploads: The web API stores the raw file in Amazon S3, triggers a document_uploaded event, and instantly returns a 202 Accepted status. This single, asynchronous path processes one-page invoices and 100-page SOC2 reports with equal reliability, eliminating the technical debt of maintaining separate “fast” and “slow” ingestion routes.
• Micro-batching: A lightweight “Spitter” consumer downloads the file, chunks it, and groups those chunks into optimized micro-batches (e.g., 64 chunks per batch).
• Controlled embedding: Embedding workers pull these batched events. To prevent concurrent workers from breaching upstream RPM limits, avoid fragile sleep() delays. Instead, enforce a token bucket rate limiter at the consumer level or strictly cap the number of active message broker partitions.

Python

# Conceptual snippet for architectural illustration
def handle_document_upload(event):
try:
raw_text = download_from_s3(event.file_uri)
chunks = semantic_chunking(raw_text)
except Exception as e:
# Log failure and raise so the message broker routes this to a Dead Letter Queue (DLQ)
print(f"Failed to process document {event.file_uri}: {e}")
raise

# Batch size heavily depends on the downstream embedding model's context limits
batch_size = int(os.environ.get("EMBEDDING_BATCH_SIZE", 64))

for i in range(0, len(chunks), batch_size):
chunk_batch = chunks[i:i + batch_size]
kafka.publish("embedding_tasks", {
"tenant_id": event.tenant_id,
"document_id": getattr(event, "document_id", event.file_uri),
"chunks": chunk_batch
})

This keeps individual consumer tasks short, respects upstream rate limits by maximizing payload density, and allows horizontal scaling of embedding workers during traffic spikes.

Bottleneck 2: the multi-tenant nightmare

Developers often treat multi-tenancy as an afterthought. The simplest way to handle multiple B2B tenants using a single RAG system is logical segregation, where all vectors reside in a large index, and each entry has a tenant_id associated with its metadata. Upon retrieval, the application filters results by adding a clause to the metadata payload.

Flaws of the approach:

1. Security vulnerabilities: Relying on application-level filtering creates an unacceptable risk. If an engineer omits or misconfigures a metadata filter, one client can access another’s confidential data. In highly regulated environments, this breaks compliance.
2. The noisy neighbor problem: If one customer uploads 10 million vectors to the shared index, memory usage skyrockets during vector searches. This degrades performance across the entire system, even for tenants with a handful of documents.

The fix: serverless compute-storage decoupling

Echo-chamber thinking assumes that the only solution is to provide each tenant with its own dedicated database cluster. This is prohibitively expensive and practically impossible to manage in a modern-day SaaS offering. 

The true gold standard here is using next-generation serverless vector databases like Pinecone Serverless or managed Qdrant implementations, which make a clear distinction between computing and storage.

Engineering takeaway: stop building complex multi-tenant routing logic in your application code. Push the isolation boundary down to the infrastructure layer using serverless namespaces.

Bottleneck 3: the semantic caching trap

Once ingestion is asynchronous and tenants are segregated, inference costs become the final bottleneck. Hitting an LLM API for every individual query is economically unsustainable.

The industry defaults to semantic caching: embed the user’s prompt, calculate its cosine similarity against previous prompts, and return a pre-calculated LLM response if the score exceeds a set threshold (e.g., 0.95).

Why semantic caching fails

Embeddings capture overall semantic meaning, but they miss specific contexts and entities. The prompts “What was the holiday policy in 2023?” and “What is the holiday policy for 2024?” share a near-perfect cosine similarity score. The core semantics match, but returning a cached answer feeds the user incorrect or contradictory information.

The fix: hybrid verification vs. native prompt caching

To scale without compromising accuracy, there are only two choices to consider: application-layer validation or infrastructure-layer optimization.

Strategy A: combined lexical filtering and intent routing

When using an application-layer caching system (for instance, Redis), you need to layer the semantics search on top of extremely light guardrails.

1. Exact-match filter: Apply a token-validation filter over vector similarity. In the case where the cached query is “2023”, and the current query is “2024,” throw out the cache hit right away.
2. Intent routing: Before serving a cached answer, use an inexpensive, fast model as an intent match router.

Python

Query A: {incoming_query}
Query B: {cached_query}

Do these queries have the exact same intent and require the exact same factual answer? 
Respond only with YES or NO.

Strategy B: infrastructure-level prompt caching

If the system cannot tolerate the added latency of an application-layer verification router, bypass custom caching entirely and offload the problem to the infrastructure.

Modern LLM providers natively support prompt caching. It is crucial to understand what is being cached here: it is not the user’s short question. When multiple users query the same corporate knowledge domain, the massive system instructions and the heavy retrieved context documents (often 10k+ tokens) are cached automatically at the provider’s inference layer.

“Stop approaching AI like a magic black box and instead approach it as a distributed systems problem.”

The application sends the full RAG query every time. The provider recognizes the repeated context block, slashes context token costs by up to 80%, and drops the time-to-first-token (TTFT) to milliseconds.  

The infrastructure surrounding a foundation model separates a prototype from a production AI system. Stop approaching AI like a magic black box and instead approach it as a distributed systems problem, and things will fall into place. Batched fan-out asynchronous queuing solves timeout and rate-limiting issues. Serverless namespacing resolves compliance risks. Prompt caching and intent routing secure unit economics. Designing a native AI product today means engineering for inevitable API failures, cross-tenant data attacks, and runaway LLM costs.

The post Can prompt caching tame RAG costs without sacrificing accuracy? appeared first on The New Stack.

Public AI assistants have become so commonplace that software vendors are increasingly adding AI search, conversational experiences, and AI agents to their own applications. From eCommerce and customer support to enterprise software, AI is rapidly becoming the primary interface to many applications.

Companies that build products around proprietary information are particularly well positioned to benefit from this shift. Whether they provide financial intelligence, market intelligence, legal research, scientific publishing, or business information, their products help professionals make better decisions by transforming trusted information into actionable insight. 

AI allows these organizations to deliver that expertise through entirely new user experiences. Increasingly, they compete not only on the quality of their proprietary information, but on how intelligently they retrieve, understand, and transform it into customer value.

“Increasingly, they compete not only on the quality of their proprietary information, but on how intelligently they retrieve, understand, and transform it into customer value.”

A new competitive battleground is emerging. As AI becomes the primary interface to proprietary knowledge, the ability to retrieve, verify, rank, and assemble information is becoming almost as important as the proprietary information itself. 

Much of the industry’s attention has focused on increasingly capable language models, but those models are only as effective as the context they receive. Designing retrieval workflows that consistently deliver trusted, relevant, and up-to-date information is rapidly becoming one of the defining engineering challenges for AI-native applications.

Retrieval engineering: optimizing the workflow

For decades, search engineering has focused on helping people find the right information. Whether searching a website, a legal database, or a financial research platform, the challenge was to retrieve the most relevant results while balancing competing priorities such as relevance, latency, scalability, and cost. The search system’s job was to retrieve relevant information. The human’s job was to evaluate it.

AI fundamentally changes that role.

Instead of retrieving information for people to evaluate, retrieval systems increasingly assemble the context that large language models and AI agents use to investigate, reason, and act. Every retrieval decision now becomes part of an automated workflow in which relevance, freshness, latency, and trust directly influence the final answer.

“Prompt engineering influences how a language model reasons. Retrieval Engineering determines what it has to reason about.”

This shifts the engineering challenge away from individual technologies and towards the retrieval workflow itself. The goal is no longer simply finding relevant documents, but orchestrating retrieval, ranking, filtering, inference, and real-time updates so they work together efficiently. We believe this emerging discipline deserves its own name: Retrieval Engineering. Prompt engineering influences how a language model reasons. Retrieval Engineering determines what it has to reason about. Both matter, but as AI applications become increasingly autonomous, the quality of retrieval increasingly determines the quality of the outcome.

As AI applications evolve from conversational assistants to deep research systems and autonomous agents, optimizing workflows rather than individual components becomes increasingly important. A single user request may trigger dozens—or even hundreds—of retrieval operations before a response is generated.

The challenge isn’t vector search

Vector databases solved an important problem by making semantic retrieval practical at scale. But semantic retrieval is only one stage of a much larger workflow.

Production AI applications increasingly combine vector similarity with keyword search, structured filtering, business rules, personalization, machine-learned ranking, and real-time inference to assemble the context that language models depend on. The engineering challenge is no longer selecting the best retrieval technology—it is orchestrating increasingly sophisticated retrieval workflows that remain accurate, responsive, and cost-effective.

Many organizations address this by combining specialist technologies. A vector database provides semantic retrieval. A search engine handles lexical matching. Additional services provide reranking, personalization, and inference. This works well initially, but every additional component introduces another network hop, another operational dependency, and another source of latency. The problem is no longer vector search. It is engineering an efficient retrieval architecture.

From components to platforms

This shift is changing how retrieval infrastructure is designed. Instead of optimizing individual components in isolation, engineering teams increasingly need to optimize the retrieval workflow as a complete system—balancing retrieval quality, latency, freshness, scalability, and infrastructure cost.

“The problem is no longer vector search. It is engineering an efficient retrieval architecture.”

That is why AI Search Platforms are emerging. Rather than stitching together retrieval, ranking, inference, and serving from multiple independent services, they execute the workflow within a single distributed architecture. The optimization problem changes from integrating components to engineering the workflow itself.

AI has transformed the user interface. It is now transforming the retrieval infrastructure behind it. For organizations building applications around proprietary knowledge, the next competitive advantage will not come solely from larger language models or better embeddings. It will come from building retrieval workflows that consistently deliver trusted, relevant, and timely context at scale.

Retrieval Engineering is rapidly becoming one of the disciplines defining the next generation of AI-native applications.

If you’re interested in exploring these ideas in more depth—including Retrieval Engineering, AI Search Platforms, and the architectural patterns behind AI-native information platforms—we cover them in our ebook, Building AI-Native Information Platforms.

The post Is retrieval engineering becoming AI’s next bottleneck? appeared first on The New Stack.

When you run Kubernetes at the scale we do on Amazon EKS, nodes break constantly. GPUs fall off the PCIe bus. Container runtimes wedge. Network interfaces disappear. Across tens of thousands of clusters, “rare” hardware failures happen multiple times a day, somewhere in the fleet.

For years, everyone responded the same way: an operator wakes up, reads a dashboard, SSHes into the node, cordons it, drains it, terminates the instance, and waits for a replacement. Every step is human-paced. Every step is toil. And if the failure lands at 3 a.m. on a weekend, the workload sits degraded for hours before anyone looks.

We built the EKS Node Monitoring Agent to help close that gap, which we open-sourced in April earlier this year. It detects node failures and writes Kubernetes NodeConditions that signal the problem to Karpenter, which then automatically replaces the node if required. The agent is one piece of a larger system. To understand where it fits, you need to understand what manages the nodes it monitors.

“Across tens of thousands of clusters, ‘rare’ hardware failures happen multiple times a day, somewhere in the fleet.”

AWS launched Amazon EKS Auto Mode that fully automates Kubernetes cluster infrastructure: compute provisioning, scaling, networking, storage, OS patching, and security hardening, so teams focus on applications, not cluster operations. It dynamically selects optimal EC2 instances (including GPU instances like P5, P6, and G6 families), scales based on workload demand, consolidates underutilized nodes, and keeps the operating system patched. EKS Auto Mode ships with automatic node repair as the default behavior: detection, severity classification, and Karpenter-driven node replacement all run out of the box with no add-on to install, no controller to configure, and no repair policy to write.

This is the story of how we built automatic node repair, the design decisions that shaped the system, and the hard lessons that came from operating it at GPU scale.

Six lessons from building self-healing Kubernetes nodes at scale

After operating this across thousands of clusters, the lessons compress into a short list. These are not unique to our system. The same patterns show up in NPD, NVSentinel, AKS Periscope, GKE’s auto-repair, and anyone building a custom node controller. They are folklore that should be a checklist. The open-source repo reflects each of these lessons in code, from the reason-code stability guarantees in the API to the jitter implementation that solved GPU workload interference.

“Your reason codes are an API contract. Additions are features. Renames are breaking changes.”

1. Your reason codes are an API contract. Every downstream consumer (repair controllers, dashboards, customer automation) keys on them by literal string match. Additions are features. Renames are breaking changes. Severity changes are breaking changes. Plan for them the way you plan for API versioning.
2. Absent and Unknown are not the same thing. “We are not watching” and “we are watching but cannot tell” require different responses from downstream automation. If your disabled monitor writes Unknown, some controller somewhere will eventually act on it. Emit nothing when you are not watching.
3. Don’t cross ownership boundaries. The kubelet owns workload-driven conditions. Your node-health agent owns hardware and infrastructure failures. Crossing that boundary means your repair system is fighting the kubelet’s eviction system, and one of them will make the wrong call.
4. Measure latency from the source. The detection SLO includes every hop in the signal chain: hardware event to driver log, driver log to journald, journald to agent poll, agent poll to NodeCondition write. The longest hop dominates. For kernel-level signals, journald flush cadence is the bottleneck. For GPU telemetry through DCGM, push-based policy violations (DBE, XID, NVLink) are near-instant, but polled field watches (NVSwitch fabric health, clock throttle) have a 5-minute floor. Know which path each detection uses.
5. Detection and diagnosis are separate systems with separate consumers. Detection feeds automation (fast, continuous, minimal data). Diagnosis feeds humans (on-demand, detailed, heavyweight). Conflating them degrades both.
6. Test telemetry interpretation against the spec, not empirical values. Hardware telemetry interfaces are not boolean. We read a DCGM bitfield for GPU fabric health and treated non-zero as failure. When a driver update changed the healthy return value from zero to a spec-defined non-zero mask, every GPU node was flagged unhealthy at once. The safety breaker held (by design), giving us time to ship the fix. The lesson: if you’re parsing packed enums or bitfields from GPU firmware, your test fixtures must come from the vendor documentation, not from what the field happened to return on previous hardware.

Node health detection in Kubernetes: Traps no one warns you about

Every node health agent in the Kubernetes ecosystem performs the same translation. Node Problem Detector (NPD), NVSentinel, GKE’s auto-repair, AKS’s Linux Extension, and the EKS Node Monitoring Agent all take noisy, low-level signals from a machine and translate them into a set of Kubernetes primitives: NodeCondition, Event, sometimes a CRD. The translation looks simple. It isn’t.

The output is a NodeCondition, which is just a type, a status (True/False/Unknown), a reason code, and a message. Four fields. But that surface area hides decisions that determine whether a repair action helps or hurts.

Reason codes are a public API. We learned this the hard way. In version 1.6.2, we changed NvidiaDeviceCountMismatch from Warning severity to Fatal. The technical reasoning was sound: once a GPU drops off the PCIe bus, it doesn’t come back without a node reboot or replacement. Leaving it as Warning meant GPU workloads kept getting scheduled onto degraded nodes, wasting expensive accelerator capacity. So we shipped the fix. Downstream automation broke. Customers had repair configurations keyed on the old severity. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. From that point, we treat every reason code addition as feature work and every rename or severity change as a breaking change.

“Absent” must not equal “healthy.” When we shipped per-monitor configurability in v1.6.0, we had to make a choice. A disabled monitor needs to produce some output (or no output). The three options: write True (your auto-repair now thinks the node is healthy because you’re not watching), write Unknown (ambiguous, might trigger repair depending on downstream logic), or omit the condition entirely. Only the third is safe.

This seems obvious in retrospect, but consider that NPD achieves the same result through a completely different mechanism: compile-time disable via build tags. NVSentinel delegates it to operator-authored CEL rules. The upstream Kubernetes spec defines what Unknown means, but if your repair automation treats Unknown as actionable, you will lose nodes for no reason. We chose to emit nothing when a monitor is off, and documented it as a hard contract.

Detection latency is bounded by the source, not by the agent. We originally told customers, “We detect kernel panics within 30 seconds.” This was wrong. Our agent’s detection time was under 30 seconds. But the kernel panic shows up in journald, and journald’s flush cadence is the actual bottleneck. If journald takes 45 seconds to write the line, our 30-second claim was incomplete.

For GPU faults, the picture is more nuanced because we use two detection paths with very different latency characteristics. The critical faults (double-bit ECC errors, XID errors, NVLink failures, page retirements, thermal and power violations) go through DCGM’s push-based policy violation channel. DCGM notifies our agent the moment it detects the violation; there is no polling interval. Detection of these faults is near-instant (sub-second in practice). A separate path uses a 5-minute field-value window to monitor NVSwitch fabric health, Fabric Manager status, and clock-throttle reasons. That window is the floor for those specific detections, but it does not apply to the critical GPU faults that trigger automatic repair. The lesson: the customer-facing SLO must include source-of-truth latency, and different signal paths within the same subsystem can have radically different floors.

Two severities, one switch: How auto-repair decides which nodes to replace

The kubelet already reports DiskPressure, MemoryPressure, and PIDPressure. NMA complements those with five additional conditions covering domains the kubelet does not monitor: kernel health, container runtime, networking, storage, and accelerated hardware. Every detection carries one of two severities, and severity is the switch that decides whether the repair cycle fires.

Condition severity is a terminal fault. It flips the matching condition to False and makes the node eligible for automatic repair. GPU device-count mismatches, critical XID and double-bit ECC errors, NVLink and NVSwitch fabric failures, a missing Fabric Manager, and Neuron DMA and HBM uncorrectable errors. On the networking and runtime side: VPC CNI process down, IPAMD unable to reach the API server, fork failures due to PID exhaustion, and pods wedged, terminating behind a broken container runtime. These are faults that won’t recover on their own. On GPU nodes, a single degraded accelerator can corrupt training checkpoints or waste thousands of dollars in compute per hour.

Event severity is informational. It posts a Kubernetes event, the NodeCondition remains True, and operators get visibility without disruption. Bandwidth ceilings, connection-tracking limits, Amazon Elastic Block Store (Amazon EBS) IOPS throttling, I/O delays, filesystem fragmentation, clock drift, liveness and readiness probe failures, kube-proxy anomalies, GPU thermal and power warnings, PCIe link degradation, and page-retirement thresholds. These signal trouble building before it turns terminal.

Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes and needlessly displace workloads. Too conservative, and degraded nodes serve traffic for hours while a GPU with a failing memory bank corrupts training checkpoints. The classification principle: if the failure is deterministic and infrastructure-owned (hardware broke, firmware crashed, a physical link went down), it triggers replacement. If the signal could be application-induced or transient, it stays informational. You never want to terminate a healthy node because a misbehaving pod saturated a resource.

“Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes. Too conservative, and degraded nodes serve traffic for hours.”

DiskPressure, MemoryPressure, and PIDPressure are the canonical examples. Every major auto-repair system (GKE, AKS, NPD) has independently converged on the same answer: don’t touch them. These are workload-driven conditions, not node-level faults. Replacing the node just moves the misbehaving workload to a fresh machine, where it will eat memory again. The correct response is kubelet-level pod eviction, not node replacement. If you’re building a node-health system, draw this boundary early and document it publicly.

The agent that hurt what it was protecting: GPU workload interference from health monitoring

The hardest lesson came from a customer running large-scale distributed GPU training. Their workload used NCCL collectives across hundreds of GPU nodes, where every node in a communication group must complete its step before any can proceed. One slow node makes every node wait.

They found that NMA itself was causing periodic slowdowns. The agent’s monitors all ran on independent goroutines, and when their polling intervals aligned, dozens of goroutines would wake simultaneously and burst onto many CPU cores at once. On a general-purpose web service, this would be invisible. In a distributed training job, microseconds of jitter on one node can cascade across the entire GPU cluster, causing measurable throughput loss.

The customer disabled NMA entirely and saw an immediate improvement. That was the worst possible outcome for us: a health agent that interferes with the workload it exists to protect is worse than no agent at all.

The fix was straightforward once we understood the problem. We added a startup jitter to every monitor’s polling interval. Each goroutine delays its first tick by a random offset (up to 20% of its base interval), staggering the wake times so they don’t align on boot. We cached system calls that hit /proc on every poll. We consolidated handlers that shared an interval into a single sequential work queue, reducing the goroutine count for monitors that didn’t need their own thread. The result was an agent whose CPU profile is flat and predictable rather than bursty.

The lesson generalized: if your health agent runs on the same host as the workload, its resource consumption pattern matters as much as its resource consumption total. A process that uses 0.5% CPU spread evenly is invisible. A process that uses 0.5% CPU in concentrated bursts can disrupt latency-sensitive distributed GPU workloads in ways that show up as lost training time rather than a CPU alarm.

This is why per-monitor configurability matters. Not every monitor is relevant to every workload. A dedicated GPU training cluster with one pod per node and no pod churn doesn’t need IPAMD monitoring or environment scanning. We shipped the ability to disable individual monitors so customers can keep the health coverage they need without paying the overhead of coverage they don’t.

How the repair cycle works

Karpenter is the compute controller that provisions and scales EKS Auto Mode nodes. It already owns the lifecycle of every node it launched, and consuming our NodeConditions for repair is a natural extension of that ownership. There’s no separate repair backend, no sidecar controller, no webhook chain. The same system that created the node is the one that replaces it.

Karpenter’s AWS cloud provider declares repair policies: each one pairs a condition type with a status that means “replace this node.” The policies include toleration windows that prevent reacting to transient blips:

• Accelerated hardware faults: 10 minutes. These are unambiguous (a GPU is either present or absent) and expensive to leave running (a training job on a degraded node wastes GPU-hours).
• Everything else (kernel, runtime, networking, storage, kubelet NotReady): 30 minutes. Enough time for a transient network blip or a temporary runtime hiccup to resolve on its own.

The flow:

1. The agent detects a terminal fault and flips the matching condition to False with a reason code.
2. Karpenter’s health controller sees the transition and starts a timer.
3. If the condition clears before the window expires, the timer resets silently. The node was never touched.
4. Past the toleration window, a safety gate checks fleet health. Karpenter will not repair more than 20% of nodes in a NodePool simultaneously. If a correlated event (a bad AMI rollout, a control-plane hiccup, a zonal impairment) trips conditions across many nodes at once, the system holds. Auto-repair also stands down while an Amazon Application Recovery Controller zonal shift is active, so deliberate traffic movement away from an impaired Availability Zone is not mistaken for a fleet of broken nodes.
5. Inside the safety threshold, Karpenter taints the node to block new scheduling, gracefully drains running pods (respecting PodDisruptionBudgets), terminates the instance, and provisions a replacement sized for the displaced workload.

The replacement node comes up with a fresh agent monitoring it from boot. No operator in the path. In our testing, the full cycle from fault injection to replacement node running workloads took under 12 minutes. Detection landed in under a second (critical GPU faults use DCGM’s push-based policy channel, not polling). Then 10 minutes of toleration, and roughly 90 seconds for the replacement to launch and register.

The part that surprised us: detection and diagnosis are not the same problem

Auto-repair handles the common case: broken node gets replaced, workload keeps running. But “why did that node fail?” is a different question, and one we initially tried to answer inside the detection path. That was a mistake.

Detection answers “is this node healthy?” It runs continuously with minimal overhead, and it needs to be fast: a condition flip that takes 5 minutes to produce is 5 minutes of degraded workload. Diagnosis answers “what went wrong?” It needs to collect detailed artifacts: full journald output, containerd state, network configuration, dmesg, GPU driver logs. In our testing, that collection completes in about 7 seconds and produces a compressed log bundle. Baking it into the detection hot path would have slowed down the thing customers care most about: how fast the system reacts.

We built them as separate concerns sharing an agent binary. The NodeDiagnostic CRD lets you request a full log bundle from any node through kubectl, without SSH. On EKS Auto Mode, where nodes are Amazon Elastic Compute Cloud (Amazon EC2) managed instances with no shell access by design, this is the only way to investigate after a GPU failure or any other node-level fault.

The experience is one command:

kubectl ekslogs <node-name>

The plugin creates a NodeDiagnostic resource. The agent on the target node detects it via a watch, collects system state into a compressed tarball, and stores it temporarily (available for 10 minutes). The plugin then downloads it through the kubelet’s Node Log Query API (KEP-2258, GA in Kubernetes 1.36). No SSH, no security groups, no key pairs.

This separation means detection doesn’t slow down to collect evidence, diagnosis doesn’t need to be always-on (saving node resources), and you can diagnose a node that auto-repair has already flagged but hasn’t yet terminated. The 10-minute window for accelerated hardware faults gives you exactly enough time to grab the logs before the node is gone. If you’re interested in further improvements, engage with us on EKS public roadmap.

What this means if you’re running EKS

On EKS Auto Mode, all of this is on by default. Auto Mode fully manages your cluster infrastructure (compute, networking, storage, patching, and security hardening) so you focus on applications, not cluster operations. The agent runs as a systemd service in the node image (not a DaemonSet you manage), Karpenter consumes its conditions as part of the compute lifecycle it already owns, and kubectl ekslogs gives you diagnostic access without SSH. There is nothing to install, configure, or operate. For GPU workloads, this means your expensive accelerator nodes are automatically monitored, classified, and replaced without any operator intervention.

On managed node groups or self-managed Karpenter, you can assemble the same loop: install the Node Monitoring Agent as an EKS add-on and opt each node group into auto-repair. The architecture is the same, just not pre-assembled.

The EKS Node Monitoring Agent is Apache 2.0 open source at github.com/aws/eks-node-monitoring-agent. 

The failure modes we hit when running it at scale, and the fixes that come out of them, flow back to anyone using it. If you’re building a node-health system or running ours and hitting an edge case, come build with us!

The post Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent appeared first on The New Stack.

There’s a pattern I’ve watched repeat for two years. A team builds an agent, hits reliability problems, upgrades the model, sees marginal improvement, and hits the same reliability problems in a slightly different form. The diagnosis is always the same: the model wasn’t smart enough. The fix is always to try a new, smarter model. The result is always the same: still broken. 

This isn’t a model problem. It never was.

Andrej Karpathy figured this out months ago, and in a post on X, he noted a shift in how he was spending his AI compute: “a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge.” He wasn’t running a smarter model. He was building better infrastructure:  raw sources indexed into a directory, an LLM incrementally compiling them into a structured wiki with summaries, backlinks, and concept articles, tools handed to the agent as CLIs, outputs filed back into the base to enhance future queries. The model was constant. The infrastructure around it was the variable.

“This isn’t a model problem. It never was.”

I keep coming back to the framing. The model runs on context. The quality of execution depends on the quality of the context it receives, the precision of the actions it’s permitted to take, and the feedback loops that let the system learn from what it got wrong. None of that lives in the model. It lives in the infrastructure underneath. And it turns out, most teams haven’t built it.

The missing compile setup

Karpathy’s setup is built around one extra step. Raw data comes in. An LLM “compiles” it into a structured, queryable form. Then, agents operate over that compiled version with tools. That compilation step is where the work happens, and it’s what turns a mess of internal data into something an agent can reason over instead of guessing at.

Too many production agent systems still skip this step. They wire the model directly to raw data — databases, APIs, document stores — and expect it to compile at query time, within the context window, under latency pressure. What comes back is pattern-matched guesswork over a window full of noise.

The teams that got this right built the compilation step explicitly. Not a generic knowledge base: a structured representation of how their specific organization works. How things are named internally. What actual decision paths look like. What previous similar runs produced, and what was decided. The organizational equivalent of Karpathy’s wiki — built from operations, not from documentation.

This is harder to build than switching models and harder to maintain. It’s also the difference between an agent that operates in your organization’s actual reality and one that confidently operates in a hallucinated version of it.

The tool retrieval problem

Karpathy notes that once his wiki reached meaningful scale — around 100 articles and 400,000 words the system remained usable because the LLM maintained indexes and summaries, and because he began adding tools, including a small search engine exposed to the LLM over a CLI. The point is not that context disappears. It is that retrieval, indexing, and tool interfaces become part of the system you have to engineer.

Production agent systems hit this wall hard. Consider a mid-size engineering org: GitHub, Jira, Confluence, a handful of cloud providers, monitoring and alerting tools, a CI/CD platform, internal deployment tooling. Each integration is a family of tools with its own schema, naming conventions, and expected invocation patterns. Dropping all of that into a single context window is slow and expensive, and it produces poor tool selection. The model pattern-matches across noise.

Standard vector retrieval compounds the problem. It matches the semantic similarity between a user query and stored tool descriptions. It works when vocabulary aligns. It breaks when it doesn’t: a developer asks “why did the deploy fail,” the right tool is something like get_pipeline_run_logs, and the vector match between those two phrases is poor. The agent selects the plausible tool instead of the correct one.

The fix is to guess the answer first. Given the query, what would a working tool call look like? The system writes that hypothetical call, then matches against it instead of the raw question. “Why did the deploy fail” and “fetch pipeline logs” don’t look alike as text. But once you’re matching the shape of the right action instead of the words in the request, they line up.

This is a translation layer, from intent to action, and that translation is where most agent failures start. It’s an engineering problem, not a model problem. I’ve seen teams run this in production after watching vector retrieval fall apart at scale, and they say the same thing: switching from semantic similarity to hypothetical-invocation matching gave them more reliable tool selection than upgrading the model.

The guardrails gap

The capability-without-constraint failure mode shows up across contexts, and the pattern is consistent: an agent with broad tool access, executing correctly from its own perspective, takes an action nobody intended. Not because it hallucinated. Because the boundary between what it could do and what it should do wasn’t enforced.

Three cases illustrate the shape of the problem.

Start with GTG-1002, still the most detailed public account I’ve seen. In November 2025, Anthropic disclosed that a Chinese state-sponsored group had manipulated Claude Code in a cyber-espionage campaign against roughly 30 organizations, infiltrating in a few cases. Anthropic reported that AI did roughly 80-90% of the tactical work, with humans stepping in only at strategic decision points, and that, at peak, the AI was firing off thousands of requests, sometimes several a second. No human team could keep pace. This wasn’t a model-quality failure. It was a failure of execution boundaries. Once the attacker fragmented the work and slipped past the safeguards, the system could take high-risk actions faster than any human could supervise.

Prompt injection is a different vector, same structural failure. Researchers have repeatedly demonstrated agents executing injected instructions from content they retrieved:  a malicious payload in a web page or document that redirects the agent’s next tool call. The model does what the injected instruction says because nothing in the execution layer distinguishes “instruction from user” from “instruction found in retrieved content.” The agent is working correctly. The architecture isn’t.

The third pattern is quieter and more common: over-permissioned agents operating across multiple write-capable systems. An agent with access to a CRM, an email client, and a calendar does something unexpected — updates records, sends a draft, books a meeting — because a workflow reached a branch that wasn’t anticipated and there was no scoped permission model to prevent it. Nobody intended this. Nobody wrote a rule against it. The agent had access, so it acted.

“An agent with access to a CRM, an email client, and a calendar does something unexpected, because a workflow reached a branch that wasn’t anticipated and there was no scoped permission model to prevent it.”

What these have in common is that the model isn’t the failure point. The failure is the absence of an execution layer that defines, per action, what is permitted and enforces it regardless of what the model decides to do.

The architecture that addresses this intercepts every tool call before execution, masks sensitive data before the LLM processes it, blocks specific tool combinations at the execution layer rather than the prompt layer, enforces per-agent rate limits and role-based access, and generates a full audit trail with explicit reasoning for every invocation. Human checkpoints must be designed into high-stakes paths, and not as fallbacks, but as architecture.

We built our execution isolation layer at Mate around exactly this pattern. Every tool call gets validated, scoped, and logged before reaching the integration. The model never has direct write access to a downstream system;  it proposes actions to a layer that decides whether to execute them and at what scope.

Karpathy’s wiki points to the same discipline in miniature: health checks that find inconsistent data, impute missing fields, and suggest new connections. In an enterprise system, that linting layer needs permission boundaries. Some updates can be automatic; others should be routed to proposals, approvals, or review queues. The boundary between what an agent can change and what requires human judgment must be explicit.

What the engineering work actually looks like

The teams producing reliable production agents spend most of their engineering effort on infrastructure. I see four areas coming up again and again. 

The context graph first. Building and maintaining a compiled representation of org knowledge isn’t a one-time task. Schemas change. Systems get renamed. Personnel and processes shift. Teams that do this well treat the context graph as a product with an owner, an update cadence, and health checks, rather than a setup step that runs once at deploy time.

Observability second. Model-agnostic proxy layers are becoming standard: a single layer capturing full traces on every LLM call regardless of provider, enforcing per-tenant cost and rate limits, allowing model swaps without rearchitecting. Tracing for agents means capturing reasoning, not just requests: what the agent considered, which tool it selected and why, what came back, and what it did with the result. Without that, debugging is archaeology.

Continuous evaluation third. Per-agent, per-workflow datasets built from production traces rather than synthetic benchmarks. Two tracks: deterministic checks for things code can verify, like tool call correctness, rate limit compliance, scope violations, and model-as-judge for things it can’t, like reasoning coherence and response quality. When you promote a new prompt version or upgrade a model, you run it against real production data. The question isn’t “does it benchmark higher.” It’s “does it still work in this org’s actual context.”

Configuration management fourth. Prompt versions, model selections, and tool configurations need to be independently releasable and independently rollback-able. Changing which model an agent uses shouldn’t require a code deployment. Rolling back a prompt regression shouldn’t require an incident. The teams that figure this out early ship changes faster and break less. It’s the same ML engineering discipline that matured in recommendation systems five years ago, now applied to agent behavior, a practice most organizations are building from scratch.

The differentiator isn’t reasoning

Karpathy’s shift from manipulating code to manipulating knowledge describes where the hard work actually lives in agent systems. The model executes over context. How that context is structured, retrieved, and scoped determines the outcome. What constrains the model’s actions determines safety. What measures and improves the system determines reliability.

All of that is infrastructure. None of it is solved by a more capable model.

The model is commoditizing faster than most teams realize. The reasoning gap between major providers is narrow and narrowing. The infrastructure gap between teams that have built context plumbing and guardrails and teams that haven’t is wide and widening.

“The reasoning gap between major providers is narrow and narrowing. The infrastructure gap between teams that have built context plumbing and guardrails and teams that haven’t is wide and widening.”

A smarter model won’t help an agent that doesn’t know your organization. It won’t stop a prompt injection in a retrieved document. It won’t scope-limit an over-permissioned workflow. It won’t tell you that your tool retrieval accuracy dropped three weeks ago because someone renamed an integration.

The infrastructure does that. Build that first.

The post The bottleneck for AI agents isn’t the model anymore. It’s the context layer. appeared first on The New Stack.

Platform engineering has won the argument. Some 90% of organizations have adopted at least one internal platform; golden paths are orthodoxy, and environment requests that once took days now close in hours. By the standard the discipline set for itself, that is victory.

Then the most demanding customer the platform has ever had showed up, and it is not a developer. A coding agent that wants to validate its work requests an environment the way a client calls an API: in bursts, concurrently, with a lifetime measured in minutes and an expectation measured in seconds.

A 100-developer organization in which each engineer supervises a few agent sessions per day generates hundreds of environment requests before lunch. Each request needs realistic dependencies, and each is dead weight the moment its validation finishes. That is not a ticket queue. That is traffic.

The most demanding tenant the platform has ever had

The demand is not speculative. GitHub’s Octoverse counted 43.2 million pull requests merged per month, up 23% year over year, with Copilot’s coding agent alone opening more than a million pull requests in its first five months. Every one of those changes needs somewhere realistic to run before it merges.

The tenant mix is shifting underneath those numbers. Stack Overflow’s 2025 survey found that half of professional developers already use AI tools daily, and every daily user is a candidate to operate two, three, or five concurrent agent sessions. Environment demand no longer tracks headcount. It tracks headcount multiplied by agents multiplied by iterations.

“Coding agents turned environment requests into traffic: concurrent, short-lived, and relentless. The platform teams that keep up will be the ones that stop provisioning environments and start serving them.”

Platform teams can see what is coming. The latest State of Platform Engineering report found that 94% of organizations consider AI critical to platform engineering’s future, and its central theme is the shift from cloud-native platforms to AI-native ones.

What changed is not only the volume but also the shape. Human environment demand is diurnal, negotiable, and tolerant of a morning’s delay. Agents retry, fan out, and iterate in tight loops, and demand that the shape already has a name across the platform. The name is traffic.

Duplicate everything, and the cost curve kills you

The duplication model hands every request a full copy of the stack. Price one out: a 40-service system with its databases and queues costs a few dollars an hour per copy, takes tens of minutes to assemble, and sits mostly idle during the brief window of validation it exists to support.

Multiply by concurrency, and the model collapses. Hundreds of requests a day with modest overlap means dozens of full copies running at once, and a bill that scales linearly with agent activity. The latency is wrong by an order of magnitude too, because an agent that iterates in seconds cannot wait tens of minutes for its environment to arrive.

Pre-provisioning a warm pool does not rescue the model; it only moves the waste. Agent demand is bursty, so a pool sized for the peak idles through the trough, and a pool sized for the trough queues at the peak. Paying full-copy prices for capacity you mostly do not use is the definition of the wrong cost curve.

Share everything, and the queue kills you

The shared model runs one staging environment and admits tenants in turn. Queueing theory has described this failure mode since 1961. Little’s law says the number of requests in a system equals the arrival rate multiplied by time in the system, so as arrivals approach the rate the environment can absorb, wait times stop degrading gracefully and start exploding. Agents multiply the number of arrivals by 5-10 while the completion rate remains fixed.

Shared staging also fails on isolation. One broken change contaminates the environment for every tenant behind it, so the line does not merely lengthen; it periodically resets to zero while someone hunts down the offending commit.

Teams respond to the wait the way people always respond to a slow shared resource, by batching. Changes pile into larger deployments, making each trip through the environment count, which raises the blast radius of every failure and lengthens each occupancy. The queue teaches exactly the behavior that makes the queue worse.

Both models sit at the wrong ends of the same curve, paying full cost for full isolation or zero marginal cost for zero isolation. Neither is a point from which you can operate a serving system.

Environments are a serving system now

The mental model that fits this demand curve already exists inside every platform team. It is the one used for compute. A serving system is judged on latency, concurrency, marginal cost per request, and safe multi-tenancy on shared infrastructure, and those are exactly the four requirements agent-driven demand imposes on environments. A serving system is also something its clients invoke directly, through an interface rather than a person, which is the property that matters most once those clients are agents.

Renaming the problem matters because it changes who owns it and how it gets measured. A provisioning workflow is done when the environment exists. A serving system is never done. It has dashboards, capacity plans, and error budgets, and it is expected to absorb demand spikes without a human in the loop.

“The unit of work ceases to be a ticket and becomes a request. The latency target drops from hours to seconds.”

The mindset gap shows up on every operational dimension. The unit of work ceases to be a ticket and becomes a request. The latency target drops from hours to seconds. The success metric shifts from closed tickets to p99 latency at peak concurrency.

Serve the delta, not the whole stack

One architecture meets all four serving requirements by refusing to copy anything that has not changed. Run a single high-fidelity, stable copy of the system, deployed continuously from main. When a validation request arrives, deploy only the services that changed as lightweight, ephemeral environments, and route that request’s traffic through their own versions, while everything else falls through to the shared, stable environment.

Each serving property follows from the delta. Latency lands in seconds because starting one or two services is fast. Marginal cost approaches zero because tenants share the stable environment. Concurrency is bounded by cluster capacity rather than by environment count, and isolation holds because each request sees only its own changed services, not anyone else’s.

Fidelity is not the thing you give up. A full duplicate is faithful, which is exactly why teams build one, and also why it is slow and costly to stand up and prone to drift between refreshes. Sharing one stable copy that is continuously deployed from main gives every validation request the same real, current dependencies without reproducing them per request.

Routing is the implementation detail rather than the point. Service meshes can carry the routing label, sidecar-free approaches can too, and propagating a label through a call chain is a solved problem in most modern stacks. This is the pattern Signadot enables off-the-shelf.

Agents provision their own environments

An environment that arrives in seconds and costs almost nothing is not only fast enough to keep up with agents. It is cheap and fast enough for them to operate. When requesting one is an API call rather than a ticket, provisioning becomes a step within the agent’s own loop: ask for an environment, deploy the change to it, run the checks, read the result, tear it down, and repeat in the next iteration.

Both properties are what make that possible. A workflow measured in minutes and gated on human approval can never fit within a build-test-fix cycle, because the agent would spend its run waiting in a queue it cannot influence. Near-zero marginal cost makes a discarded environment a non-event, and seconds of latency lets validation live inside the loop instead of after it. Once the environment is something an agent requests for itself, the human stops being the rate limiter, and the platform’s serving capacity takes over.

Validation throughput is what ships AI code

Agents made generation cheap and pushed the bottleneck downstream, onto whether a change can be validated as fast as it is written. Validation throughput, not lines generated, now decides how much AI-written code actually ships, and it is a property of your platform rather than any model.

“Validation throughput, not lines generated, now decides how much AI-written code actually ships.”

Treat environments as a serving system, and environment capacity becomes a dimension you plan and budget like compute or continuous integration (CI) runners. This turns agent adoption from a surprise infrastructure bill into a demand curve you can plan against. For a decade, platform engineering built self-service golden paths for people. 

The next job is self-service for developers and agents that can scale with agent-driven velocity, and we built Signadot for exactly that.

The post Platform engineering’s new job: serving environments at agent speed appeared first on The New Stack.

Caching was one of the most critical optimizations in modern AI systems long before most teams realized it. Early prototypes of Retrieval-Augmented Generation (RAG) pipelines, AI copilots, and semantic search platforms often performed perfectly on small datasets and with limited traffic. 

But as soon as real production workloads arrived, tail latency, compounding infrastructure costs, and repeated retrieval operations started becoming impossible to ignore.

Our first instinct was that it was an easy fix. Redis would solve this effortlessly.

It was fast, simple, tested, and already trusted in high-scale systems for session storage, API caching, and rate limiting. Exact-match prompt caching dramatically reduced response times, allowing many repeated AI requests to be served in milliseconds without touching the expensive retrieval or inference layers again. For a while, Redis proved us right and solved almost every performance problem we had.

Until our workloads changed.

Traditional string-matching caches break down the moment your infrastructure becomes semantic. Human language variation means two users will ask for the exact same information using completely different wording. 

“Traditional string-matching caches break down the moment your infrastructure becomes semantic.”

Because Redis relies on exact string matches, it misses those connections entirely, creating duplicate, fragmented cache entries for identical intents. Before long, our hit rates tanked, memory utilization spiked, and we were stuck with a massive cloud bill for storing redundant data contexts.

That was when semantic caching via vector databases started to look attractive. On paper, it seemed like the perfect architectural evolution: match queries based on vector-distance math so that varied prompts could reuse old embeddings, context chunks, or past LLM answers.

Of course, production reality was far messier than the hype suggested. Vector database caching introduced its own set of problems: latency spikes, false-positive matches, embedding drift, operational complexity, and difficult tuning decisions around similarity thresholds. In some workloads, semantic caching significantly improved performance. In others, it became slower and more expensive than the Redis setup it was supposed to replace.

“Of course, production reality was far messier than the hype suggested.”

What we eventually learned is that Redis and vector databases solve fundamentally different caching problems. One optimizes exact retrieval speed. The other optimizes semantic reuse. Treating them as interchangeable technologies led to architectural mistakes that only became apparent under real production traffic.

The AI architecture we started with

Before the caching problems started appearing, our AI stack looked fairly standard for a modern Retrieval-Augmented Generation (RAG) system. The pipeline was designed around three major stages: embedding generation, document retrieval, and LLM inference.

A user query first entered the API layer, where preprocessing handled normalization, authentication, rate limiting, and conversation context assembly. Once the request was validated, the query was converted into an embedding vector using an embedding model. That vector was then used to retrieve semantically relevant chunks from a vector database before the final context was passed into the language model for response generation.

The simplified request flow looked like this:

• User sends a query
• Query is embedded into a vector
• Vector search retrieves relevant documents
• Retrieved context is assembled into a prompt
• LLM generates the final response
• Response is optionally cached

On a small scale, this worked perfectly. The real headaches started when traffic scaled and we noticed the exact same database queries and heavy inference workloads hitting us thousands of times an hour.

One of the first optimizations we introduced was Redis-based caching

The initial idea was straightforward: avoid recomputing expensive operations for repeated requests. We started by caching exact prompt-response pairs, embedding results, and frequently accessed retrieval outputs. 

Because Redis operates entirely in memory, lookup times were rapid, immediately reducing pressure on both the vector database and the LLM layer.

A simplified Redis caching flow looked like this:

const cacheKey = `llm_cache:${hash(userQuery)}`;

try {
const cachedResponse = await redis.get(cacheKey);
if (cachedResponse) {
return JSON.parse(cachedResponse);
}
} catch (cacheError) {
console.warn("Cache read failed, falling back to LLM:", cacheError);
}

const embedding = await generateEmbedding(userQuery);
const documents = await vectorSearch(embedding);
const response = await generateLLMResponse(documents);

try {
await redis.set(cacheKey, JSON.stringify(response), "EX", 3600);
} catch (cacheError) {
console.error("Failed to write response to cache:", cacheError);
}

return response;

The immediate results were excellent: near-instant response times for duplicate prompts, reduced API costs, and stable infrastructure that handled high traffic without requiring aggressive LLM scale-out. If a query matched an existing keyword-for-word, we skipped the entire expensive AI pipeline and served the answer straight from memory.

Why Redis looked like the perfect solution

Unlike vector indexes, Redis gave us clean, predictable metrics for memory usage, throughput, and latency characteristics under load. There were no similarity thresholds to configure, no ANN indexes to optimize, and no recall-versus-latency trade-offs to worry about. A cache key either existed or it didn’t. That predictability made the system easier to reason about during incidents and easier to scale under pressure.

We initially used Redis across multiple layers of the AI pipeline, including prompt-response caching, embedding caching, session state storage, rate limiting, temporary conversation memory, and caching frequently accessed retrieval outputs.

const cacheKey = `prompt:${hash(query)}`;
try {
const cached = await redis.get(cacheKey);
if (cached) {
return JSON.parse(cached);
}
} catch (cacheError) {
console.warn("Cache read or parse failed, bypassing cache:", cacheError);
}

We also started caching embeddings because generating embeddings became surprisingly expensive at scale. Even though embeddings were much cheaper than LLM inference, generating them repeatedly for popular queries still consumed a noticeable amount of compute resources.

const embeddingKey = `embedding:${hash(query)}`;
let embedding;

try {
const cachedEmbedding = await redis.get(embeddingKey);
if (cachedEmbedding) {
embedding = JSON.parse(cachedEmbedding);
}
} catch (cacheError) {
console.warn("Redis read failed, proceeding to generate new embedding:", cacheError);
}

if (!embedding) {
embedding = await createEmbedding(query);

try {
await redis.set(
embeddingKey,
JSON.stringify(embedding),
"EX",
86400
);
} catch (cacheError) {
console.error("Failed to write embedding to Redis:", cacheError);
}
}

As an added bonus, scaling horizontally with Redis clusters is a familiar process for most infrastructure teams. For a while, Redis significantly reduced both latency and infrastructure costs. Cache hit rates were high, GPU workloads dropped, and the vector database handled far fewer retrieval requests.

The architecture looked efficient enough that we initially believed Redis alone would resolve most of our AI caching challenges. That belief did not survive real semantic workloads for very long.

The problem was that AI workloads rarely behave like traditional web workloads for long. Users almost never ask the same question repeatedly. Instead, they ask semantically similar questions with slightly different wording, tone, or context. From Redis’ perspective, these were entirely different cache keys, even when the retrieval results and final answers were nearly identical. That limitation eventually pushed us toward semantic caching using vector databases.

Why we moved toward vector DB caching

Unlike Redis, vector databases do not rely on exact string matching. Instead, they compare numerical embeddings that represent the semantic meaning of text. This made it possible to retrieve cached results for prompts that were semantically similar, even when the wording was completely different.

Instead of hashing the raw prompt into a Redis key, we would generate an embedding for the incoming query and search for previously cached embeddings that were semantically close enough to reuse. If a sufficiently similar match existed, the system could skip large parts of the retrieval or inference pipeline.

The caching flow looked like this:

const embeddingKey = `embedding:${hash(query.toLowerCase().trim())}`;
const cachedString = await redis.get(embeddingKey);
let embedding;

if (cachedString) {
// Parse the stored string back into a workable array
embedding = JSON.parse(cachedString);
} else {
embedding = await createEmbedding(query);
await redis.set(
embeddingKey,
JSON.stringify(embedding),
"EX",
86400
);
}

This approach immediately solved one of Redis’ biggest weaknesses: wording variation. Queries that previously produced separate Redis entries could now reuse cached retrievals or responses if their embeddings were sufficiently close in vector space. This meant significantly higher cache hit rates for real conversational workloads.

The payoff from semantic caching was immediate, especially for conversational traffic where users ask the same question ten different ways. A string-matching setup like Redis misses completely if a user changes a single word. With a vector database, prompts like “How can I speed up vector search?” and “Best ways to optimize semantic retrieval performance?” resolve to the same underlying intent, allowing us to recycle the same cached response, context chunks, or embeddings seamlessly.

We also saw potential cost reductions beyond response caching alone. Embedding reuse became more effective because semantically similar prompts often generated nearly identical retrieval behavior. Retrieval outputs themselves could also be reused across related queries, reducing load on the vector search layer and decreasing the number of repeated context assembly operations.

Semantic caching appeared especially promising for RAG systems, AI copilots, internal knowledge assistants, search-heavy AI applications, and conversational agents with repeated intent patterns because these workloads frequently involve semantically similar queries that can benefit from intelligent cache reuse.

At first, the results were promising. Semantic caching immediately optimized our hit rates and reduced redundant retrieval calls across varied prompts. However, scaling this layout under full production traffic quickly exposed a brand new category of latency and performance constraints.

Where vector DBs started breaking

The advantages of semantic caching were real, but so were the new problems it introduced. As traffic increased and vector indexes grew larger, the system began to develop issues that were harder to predict and debug than the Redis problems we had dealt with earlier.

The first major issue was latency instability. Unlike Redis, which provided highly predictable exact-match lookups, vector similarity search performance degraded unpredictably under load, query complexity, metadata filters, and concurrency levels. Under heavy workloads, some semantic cache lookups became significantly slower than expected, especially when the system searched across millions of embeddings.

A typical semantic lookup now involves multiple operations:

• Generating an embedding
• Running ANN similarity search
• Evaluating similarity thresholds
• Retrieving metadata and cached responses

Even before LLM inference occurred, the cache layer itself was becoming computationally expensive.

False-positive matches also became a serious problem. Two prompts could appear semantically similar in vector space yet require very different responses in practice. This occasionally caused cached responses to be reused in contexts where they were only partially relevant or subtly incorrect. For context, a query about optimizing vector search for low-latency chat applications might accidentally reuse cached retrievals intended for large-scale offline analytics systems simply because the embeddings appeared highly similar.

The hardest part was tuning similarity thresholds correctly.

const SIMILARITY_THRESHOLD = parseFloat(process.env.CACHE_SIMILARITY_THRESHOLD || "0.93");
const cacheKey = `llm_cache:${hash(userQuery)}`;

try {
const cachedResponse = await redis.get(cacheKey);
if (cachedResponse) {
return JSON.parse(cachedResponse);
}
} catch (cacheError) {
console.warn("Cache read failed, falling back to semantic search:", cacheError);
}

const embedding = await createEmbedding(userQuery);

const result = await vectorIndex.query({
vector: embedding,
topK: 3,
});

if (result.matches && result.matches.length > 0) {
const bestMatch = result.matches[0];

if (bestMatch.score >= SIMILARITY_THRESHOLD) {
return bestMatch.metadata?.cachedResponse ?? bestMatch.cachedResponse;
}
}

const response = await generateLLMResponse(result.matches);

try {
await redis.set(cacheKey, JSON.stringify(response), "EX", 3600);
} catch (cacheError) {
console.error("Failed to write response to cache:", cacheError);
}

return response;

Another benefit was the reduction of repeated embedding and retrieval workloads. Since semantically related prompts often produced highly similar retrieval patterns, the infrastructure handled fewer redundant searches overall. This became especially valuable under high concurrency, where reducing repeated vector searches significantly lowered infrastructure load.

Semantic caching also improved the user experience in some scenarios. Similar prompts tended to receive more consistent responses because they reused previously validated retrieval contexts rather than generating entirely fresh retrieval paths every time.

For a while, vector database caching looked like the clear evolution of AI caching systems. The architecture appeared smarter, more adaptive, and better aligned with how humans naturally communicate.

But the more we pushed into production, the more we started discovering the hidden costs of semantic similarity itself. Finding a stable similarity threshold proved incredibly fragile. Low thresholds maximized cache hits at the expense of precision, while tight thresholds neutralized false positives but destroyed cache utility. This sensitivity turned threshold management into a major operational bottleneck, directly impacting downstream inference costs and system accuracy.

if (match.score >= 0.90) {
return match.cachedResponse;
}

Embedding drift introduced another long-term challenge. As embedding models changed over time, older cached vectors gradually became less compatible with newer embeddings. Semantic relationships shifted, reducing retrieval accuracy and forcing expensive re-indexing operations across the cache layer.

Operational complexity also increased substantially compared to Redis. Maintaining vector indexes required tuning ANN algorithms, balancing shards, handling index rebuilds, and monitoring recall accuracy as workloads changed. The infrastructure became harder to reason about because semantic correctness was no longer deterministic.

We also discovered that semantic caching consumed resources differently than traditional caching systems. Even cache hits still required embedding generation and vector search operations before matches could be identified. Unlike Redis, where a successful lookup was nearly free, semantic cache hits still carried noticeable computational overhead.

In production, the system began to reveal an uncomfortable reality. Semantic caching solved the exact-match problem, but it introduced an entirely new category of latency, accuracy, and operational challenges that traditional caching systems rarely encounter.

Redis vs Vector DB: The real production trade-offs

Once both systems had been running in production long enough, the comparison between Redis and vector database caching became much clearer. Neither technology was universally better. Each one is optimized for a completely different type of workload, and the real trade-offs only become visible under large-scale AI traffic.

Redis dominated in raw speed and predictability. Exact-match lookups were extremely fast, operationally simple, and relatively easy to scale. If a query had already been seen before in the exact same form, Redis almost always delivered the lowest possible latency. Cache hits are often completed in milliseconds with minimal computational overhead.

Operationally, Redis was also easier to maintain. Debugging cache misses was straightforward because the behavior was deterministic. A cache key either existed or it did not. Infrastructure teams already understood replication, sharding, persistence, and monitoring strategies because Redis has been battle-tested for years across traditional distributed systems.

Its weakness was semantic rigidity. Humans don’t write identical strings. If someone adds a typo or changes a single word, Redis drops the ball and treats it as a brand-new cache entry. Vector databases fixed that exact-match rigidity by letting us cache responses based on meaning. On paper, it’s a dream for RAG pipelines and copilots. In production, though, you realize you’re just paying a different tax. Vector lookups have actual computational weight. Unlike Redis, where an O(1) RAM read takes single-digit milliseconds, checking a semantic cache means you’re stuck waiting on an embedding model call and a graph traversal step just to see if you have a match.

“Unlike Redis, where an O(1) RAM read takes single-digit milliseconds, checking a semantic cache means you’re stuck waiting on an embedding model call and a graph traversal step.”

Worse, you lose determinism. With Redis, a key is either there or it isn’t. A vector cache forces you to manage a fuzzy threshold where the system occasionally treats two completely distinct user intents as “similar enough,” blindly serving bad data. Our infrastructure bill transformed too: we went from a system that was heavily memory-bound on RAM to one that aggressively chewed through compute just to optimize and search indexes.

The hybrid architecture that finally worked

After months of experimenting with both systems independently, we eventually stopped trying to choose between them and began using Redis and vector databases as complementary layers rather than competing technologies.

The final architecture used a multi-layer caching strategy. Redis handled ultra-fast exact-match caching for highly repetitive requests, session state, temporary conversation memory, and hot-path retrievals. The vector database handled semantic reuse for prompts that were conceptually similar but not textually identical.

The request flow became layered. The system first checked Redis for an exact-match cache hit, and if that failed, it moved on to a semantic vector cache lookup. If the semantic lookup also missed, the request proceeded through the full pipeline of retrieval and inference, after which the resulting output was stored back into both cache layers where appropriate.

A simplified hybrid flow looked like this:

try {
const exactCached = await redis.get(cacheKey);
if (exactCached) {
return JSON.parse(exactCached);
}
} catch (cacheError) {
console.warn("Redis read failed, proceeding to semantic search:", cacheError);
}

const embedding = await createEmbedding(query);
const semanticMatch = await vectorIndex.query({
vector: embedding,
topK: 1,
});
const SIMILARITY_THRESHOLD = parseFloat(process.env.SEMANTIC_CACHE_THRESHOLD || "0.93");

const bestMatch = semanticMatch.matches[0];

if (bestMatch?.score >= SIMILARITY_THRESHOLD) {
  return bestMatch.metadata?.response;
}


const response = await generateLLMResponse(query);
try {
await redis.set(cacheKey, JSON.stringify(response), "EX", 3600);
} catch (cacheError) {
console.error("Failed to write exact match to Redis:", cacheError);
}
await vectorIndex.upsert([
{
id: crypto.randomUUID(), // Most vector DBs require a unique ID
vector: embedding,
metadata: {
response,
},
},
]);
} catch (vectorError) {
console.error("Failed to upsert semantic cache to vector index:", vectorError);
}
return response;

This hybrid approach solved several problems simultaneously. Redis continued to handle the lowest-latency exact cache hits, protecting the infrastructure during traffic spikes and repetitive workloads. The vector cache improved semantic reuse without forcing every request through expensive ANN searches unnecessarily.

The layered design also improved reliability. Even if vector search latency increased temporarily, Redis still absorbed a large share of repeated traffic. Likewise, if exact-match cache hit rates dropped, semantic caching still recovered some of the lost reuse efficiency.

The architecture became easier to optimize because each layer had a clearly defined responsibility: Redis optimized speed, the vector database optimized semantic understanding, and the inference layer handled only true cache misses.

Over time, we also became more selective about what entered the semantic cache. Not every response benefited from semantic reuse, especially highly dynamic or context-sensitive outputs. Restricting semantic caching to stable retrieval patterns improved both precision and infrastructure efficiency.

Production lessons we learned

The biggest lesson was that AI caching behaves fundamentally differently from traditional application caching. Human language introduces semantic variation that exact-match systems struggle with, but semantic systems introduce probabilistic complexity that exact-match systems avoid.

In production, we learned that Redis and vector databases are not competing solutions but tools optimized for different layers of AI caching. Redis excels at fast, deterministic exact-match retrievals, while vector databases are better suited for semantic reuse in variable, intent-driven workloads. The most stable systems are not built on choosing one over the other, but on combining both in a layered architecture that matches the nature of AI traffic.

Ultimately, there is no “best” caching system for AI workloads. Redis and vector databases solve fundamentally different problems, and treating them as interchangeable leads to architectural inefficiencies at scale. 

Redis delivers speed and predictability for exact-match scenarios, while vector DB caching enables semantic reuse where user intent matters more than exact wording. In real production systems, the most reliable approach is not replacement but combination.

The post Why smarter AI caching sometimes makes everything slower appeared first on The New Stack.

In the same quarter, a $1.7 trillion social network and a shoe company both became cloud providers. Nobody’s operating model was designed for this.

On July 1, Bloomberg reported that Meta was building a cloud business to sell its excess AI capacity, signaling that one of the largest GPU fleets on Earth is about to get a price list.

As the report outlines, the company is still weighing two business models that are part of its Meta Compute initiative: it could offer hosted access to AI models running on Meta infrastructure (similar to AWS Bedrock), or directly rent raw compute capacity, à la the CoreWeave model.

Two weeks before that report, Allbirds had finished becoming Smartbird. The erstwhile sneaker company — which hit its peak in 2022 in selling 3 million pairs — sold its footwear brand for $39 million, lined up a convertible note facility that it later expanded to $100 million, hired an ex-AWS executive as CEO, and set out to sell GPU-as-a-Service. 

When it first announced the pivot in April, the stock spiked nearly 600% in a day, adding more than $100 million in market value at its peak — all before the company had racked a single GPU.

One of these is a serious supplier, and the other is a public shell chasing an AI multiple, but I don’t think the distinction matters much. Compute supply is fragmenting faster than any enterprise can absorb it, and overbuild always finds a buyer.

Overbuild becomes inventory

Meta isn’t selling compute because its leaders woke up wanting to fight AWS. It’s selling compute because it provisioned for its own peak, and there’s a gap between what it built and what it uses. That’s what an accidental cloud is: Infrastructure that was never meant to be a product, monetized because the alternative is depreciation.

That’s what an accidental cloud is: Infrastructure that was never meant to be a product, monetized because the alternative is depreciation.

Meta won’t be the last. Everyone who bought more GPUs than they needed in the last three years is facing the same math. Some will eat the write-down. The rest will sell. Stack that on top of the neoclouds (CoreWeave, Nebius, Lambda), the sovereign clouds, and now the Smartbirds, and the list of places you can buy serious compute has gone from a handful to many in about two years. 

The market read the Meta news as bad for neoclouds. CoreWeave and Nebius both dropped double digits on the day, and I get why: Nebius has a $27 billion contract with Meta and CoreWeave a $21 billion deal. Both had Meta as a customer and later witnessed it become a competitor in the compute business. But I’d pull a different lesson from it. It’s not that one supplier wins and another loses. It’s that supplier positions are now unstable everywhere. 

If the suppliers can’t predict their position two years out, betting your operating model on any one of them is a risk you’re not pricing.

More suppliers should mean leverage. Mostly it means sprawl.

On paper, a fragmenting supply side is great for buyers: price competition, more choice, more leverage. Cheaper compute is coming, and for AI workloads, it’s coming fast.

Most teams I talk to can’t capture any of it. Every new supplier shows up with its own console, its own billing format, its own identity model, and its own hole in your governance coverage. Signing a supplier is cheap. Operating one is not. The security review, the tagging standards, the budget enforcement, the offboarding plan — all of it gets rebuilt per provider. Choice without governance isn’t leverage. It’s sprawl, and sprawl costs more than the discount that created it.

Choice without governance isn’t leverage. It’s sprawl, and sprawl costs more than the discount that created it.

This bites hardest with GPU capacity, because that’s where the fragmentation is happening and where the money is. Workloads end up pinned to whichever supplier had chips available the day the contract was signed, and they stay there. Not because moving is impossible, but because nothing above the suppliers makes moving routine, and the team that signed the contract usually isn’t the team on the hook for utilization. Those incentives don’t fix themselves.

The Smartbird end of the market makes this non-optional. Capacity from a vendor with no enterprise track record is only usable if you can exit it in a day. That’s something you design for up front, not something you negotiate into a contract.

The durable position is above the suppliers

Every argument about picking the right cloud assumes the list of clouds is stable. It isn’t, and it’s about to get less stable. The position that survives supplier churn is the layer above them: a single control plane where every provider — hyperscaler, neocloud, or accidental cloud — is just a target you provision to, under the same policies, approvals, cost visibility, and exit path.

This is the VMware argument, extended forward. Broadcom taught the industry what single-supplier dependence costs when the supplier’s incentives change. Meta just taught the follow-up lesson. The future holds more clouds, not fewer, arriving faster and from stranger directions than anyone planned for.

The enterprises that win the price war won’t be the ones that picked the right supplier. They’ll be the ones for whom the supplier stopped mattering.

The practical version of this is an abstraction layer that treats every supplier as a provisioning target rather than a separate operating model. When a new supplier appears — Meta Compute, or a neocloud that didn’t exist last quarter — it plugs into the governance the team already defined, rather than becoming a new operating model to build from scratch. The policies get written once; the supplier list underneath can churn.

Meta selling compute and a sneaker company selling GPUs are the same headline: supply is no longer scarce; it’s fragmented. The enterprises that win the price war won’t be the ones that picked the right supplier. They’ll be the ones for whom the supplier stopped mattering.

The post Meta and the rise of the accidental cloud appeared first on The New Stack.

Imagine you just finished a two-hour meeting. You were wearing a small AI work companion that promised to capture the conversation, structure it, and let you ask questions about it later. Two hours after the meeting, you open the app and ask for the transcript. You wait. The spinner turns. The whole reason you bought the device was so you’d never have to remember a meeting again, and now the one thing it promised to do well — recall what was said — is the thing that’s making you wait.

This is the failure mode that fascinates me about AI hardware products, because it’s not a model problem. The transcription was perfect. The summarization was good. The thing that frayed was the part nobody markets: getting the right bytes off disk and onto the screen at the moment the user asks. That’s a data problem. And for a product whose entire promise is memory, a data problem is a product problem. 

“For a product whose entire promise is memory, a data problem is a product problem.”

I want to walk through a real version of this because the team that hit it, Plaud, makes the most popular AI notetaker on the market, and the architecture that broke is the architecture almost every team in this category starts with. It feels reasonable at launch. It becomes a liability at scale. And the reasons why are worth understanding before you ship, not after.

First, what was built correctly

It would be easy to tell this story as “they made a mistake.” They didn’t. The original architecture was a sensible set of decisions, and I want to be precise about that before I dissect what went wrong, because the lesson is in the gap between “reasonable” and “right at scale,” not in anyone’s competence.

Plaud’s product generates two very different kinds of data per recording. There’s structured metadata: who recorded it, when, how long, what tags, what state the processing pipeline is in. And there’s the unstructured payload: the audio file and its transcript, which can run tens of megabytes per session. The team did the textbook thing. They put the structured metadata in MySQL, where relational queries and transactions are cheap, and they put the large objects in S3, where storage is cheap and effectively infinite.

If you’ve built systems, you’ve made this exact call. Object storage for blobs, a relational database for everything you need to query. It’s in every architecture diagram. It’s the default. And for a long stretch of the product’s life, it worked fine.

The problem is not that the decision was wrong. The problem is that it contained a hidden assumption: that the metadata and the content could live in separate systems because they would never need to be consistent with each other in real time. For most applications, that assumption holds. For a product whose core interaction is “give me back exactly what I recorded, right now,” it doesn’t. And the gap between those two worlds is where everything started to fail.

The transcript sidecar

In my article on RAG retrieval, I identified an anti-pattern I called the vector sidecar: the habit of standing up a separate vector database alongside your primary store, only to discover that the two systems can’t answer a single query together. The Plaud architecture is the same shape, applied to an AI hardware product. Call it the transcript sidecar.

The transcript sidecar is what you get whenever you split structured metadata from unstructured content across two systems with no shared transaction guarantee. The metadata says a recording exists, is complete, and is ready. The content lives elsewhere, reached via a separate call, with its own latency and failure modes. Nothing ties the two together. There is no transaction boundary that spans “the row that says this transcript is ready” and “the object that contains the transcript.”

This produces three distinct compounding problems.

• Retrieval latency is not a network problem; it’s a data locality problem. S3 is excellent object storage. It is not a database. When you make object storage the primary retrieval path for tens-of-megabytes payloads under concurrent load, you’re trading the consistency and latency guarantees of a database for the simplicity of a blob store. Under light load, the trade is invisible. Under heavy concurrent load, S3 retrieval latency and its variability become the user’s experience of your product.
• Consistency gaps open between two systems that fail independently. The MySQL row can indicate that a recording is ready a moment before the S3 object is durably reachable, or a replica can lag, causing the metadata a user sees and the content they fetch to disagree. With no shared transaction, the application layer inherits the job of papering over the gap, with retries, polling, and reconciliation logic that grows more elaborate every quarter.
• The schema becomes immovable at exactly the wrong time. More on this below, but the short version is that the metadata store hit a scale ceiling where changing the schema required a maintenance window. For a product still evolving its feature set, that’s a database governing the roadmap, rather than the other way around.

The unifying insight is the one I keep coming back to throughout this whole series: any time you split data across systems that can fail independently, you have handed the consistency problem to the application layer, where it is not solved so much as managed, and where it compounds over time.

When the database governs the roadmap

The most interesting failure in the Plaud story isn’t the latency. It’s the schema freeze, because it’s the one that teams least expect and feel most acutely.

With around 300 million rows, the team’s MySQL setup reached a point where schema changes such as adding a column or changing an index — the routine evolution of any growing product — could no longer be performed online without risk. DDL operations on a table that large, on that architecture, meant locking, replication strain, and the real possibility of downtime. So changes had to be batched into maintenance windows.

Sit with what that means. A maintenance window for a schema change is the time the database allows the product team to ship. The roadmap now bends around the database’s limitations. A feature that needs a new column will wait until the next window. The product’s cadence is set by the operational fragility of the data layer. Most teams don’t notice this inversion because it happens gradually, and then one day a product manager asks why a small change is a two-week project, and the answer is “the database.”

“A maintenance window for a schema change is the time the database allows the product team to ship. The roadmap now bends around the database’s limitations.”

The 300-million-row ceiling is a forcing function that teams hit later than they expect and should hit later than they do. Expect, because single-instance MySQL feels limitless right up until it doesn’t. Should, because the architecture that gets you to 300 million rows is rarely the architecture that takes you past it, and the migration is far more disruptive at 300 million rows than it would have been at 30 million. The ceiling is real; it is predictable, and most teams plan for it only after they’ve hit it.

What changing the architecture actually fixed

Plaud’s resolution was to consolidate the metadata layer into a distributed SQL database (they moved to TiDB), which removed the single-instance ceiling and restored online schema changes. I’m going to use their numbers, but not as a product pitch. I want them as evidence for a claim about where architectural debt goes in a product like this.

QPS headroom and tail latency matter, but the first result I’d point to is the online DDL. Restoring the ability to change the schema without a maintenance window is what handed the roadmap back to the product team. The database stopped governing the release cadence. That’s not a performance win you can put on a benchmark chart, but it’s the one the product organization feels every week.

Also notice what the fix did not do: it did not attempt to store 30-megabyte audio files in the database. Large objects can still be stored in object storage. The point was never “put everything in one system for its own sake.” The point was that the metadata layer, the part that has to be consistent, queryable, and evolvable in real time, needed to actually behave like a database at scale, instead of becoming the fragile half of a split-brain architecture.

Where architectural debt goes

Here’s the claim the Plaud numbers are evidence for. When the database is the product, architectural debt does not accumulate in some back-office system that only the platform team feels. It accumulates in the product experience, where every user feels it.

This is the structural difference between an AI note-taker and, say, an internal analytics tool. For a product like this, every meaningful user interaction is a database operation. Recording creates rows and objects. Transcription updates state. Retrieval is a read. Editing is a write. Search is a query. There is no part of the product that isn’t, underneath, the database doing something. Architectural problems surface as product problems, one-for-one. A consistency gap becomes a transcript that is briefly missing. Slow retrieval reads as a product that is slow to remember. A frozen schema means features arrive late.

No user will ever file a bug report that says “your metadata store and your object store lack a shared transaction boundary.” Left unaddressed, this class of debt manifests as a product that feels less dependable precisely when someone depends on it. For a product whose entire value proposition is “trust me to remember,” the debt lands on the promise itself. This is why Plaud’s decision to re-architect when it did matter. The team fixed the data layer before the failure became noticeable to users.

“You can have the best transcription model in the category and still ship a product that feels unreliable, because reliability for this kind of product is a data architecture property, not a model property.”

This is why I say the database is the product. Not as a slogan. As a literal description of where the product’s quality is determined. You can have the best transcription model in the category and still ship a product that feels unreliable, because reliability for this kind of product is a data architecture property, not a model property.

The trap is waiting for the whole category

AI hardware is having a moment. Note-takers, wearables, ambient recorders, pendants, badges — each built on the premise that the device will remember so you don’t have to. The category is growing, and the products are getting better at the parts everyone talks about: the models, the form factor, the battery life.

But underneath, almost all of them start with the same architecture Plaud started with, because it’s the reasonable default. Structured metadata in a single-instance relational database. Large content in object storage. No shared transaction boundary. A schema that’s easy to change at ten million rows and frozen at three hundred million. The trap is identical, and it’s waiting at the same place on the growth curve for every team in the category.

The teams that will do well are the ones that recognize this early, understand that, for a memory product, the metadata layer is not back-office plumbing but the spine of the user experience, and plan their data architecture for the scale they’re trying to reach rather than the scale they’re at. Plaud’s migration is the pattern worth studying before you hit 300 million rows, not after. The lesson is cheaper to learn from someone else’s 300 million rows than from your own.

When your database and your product are the same thing, you don’t get to treat the database as someone else’s problem. It is the product. Build it like one.

The post “The database is the product”: What breaks when memory devices scale appeared first on The New Stack.

Most engineering teams I talk to can ship an AI demo. The prototype works, stakeholders are impressed, and everyone agrees the use case has potential. Then the project hits a wall.

The reasons for this can vary, but new research shows that difficulties in collecting and parsing real-time data from multiple sources are often the problem. And it’s compounded by a growing skills shortage.

“Only 32% of organizations report having agentic AI running in production.”

According to Confluent’s 2026 Data Streaming Report, only 32% of organizations report having agentic AI running in production. At the same time, two-thirds of respondents cited data infrastructure and data quality as barriers to the success of agentic AI. The models work in controlled conditions, but production is a different story.

Why the demo-to-production gap is so wide

Demos tend to work because everything around them is controlled. The data is static and curated carefully to support exactly what the model will be asked to do. Production environments don’t always offer those luxuries.

In production, AI systems have to query data that lives across dozens of sources, including databases, event streams, application logs, and third-party feeds. Much of that data is poorly governed, and little of it is designed to be consumed by an AI agent in real time. Models that looked impressive in pilots return unreliable results because they’re working with stale, incomplete, or uncontextualized data.

The instinct is to tune the model, but the problem is more likely to be the data feeding it.

In the report, 72% of IT leaders cited insufficient infrastructure for real-time data processing as a barrier to scaling AI, up from 61% the year before. That increase suggests the problem isn’t going away; it’s getting more visible as teams move projects into production.

“The instinct is to tune the model, but the problem is more likely to be the data feeding it.”

AI systems need data that’s trustworthy, contextualized, and current, and those properties are hard to guarantee when data is sitting in siloes that weren’t built for continuous consumption. Batch pipelines almost always introduce latency, lack formal data contracts, and obfuscate lineage. The AI system ends up working with an inconsistent, partial snapshot of the business instead of what’s actually happening now.

The skills problem makes this harder

The report reveals another challenge: 71% of IT leaders cited a shortage of relevant expertise and skills as a barrier to AI adoption. 

The work of application development has shifted from encoding business logic to creating an information environment where automated systems can learn and generalize.  Building reliable AI applications requires developers to be stronger data engineers. They need to understand distributed systems, streaming architectures, data quality controls, and how to build pipelines that hold up under real-world conditions. They need to reason about data lineage, schema evolution, and what happens when an upstream source changes. And the QA patterns that work for deterministic software — where the same input yields the same output — don’t transfer to probabilistic systems.

Most developers haven’t had to think this way before. The discipline of getting the right data to the right system at the right time, in a governed and reusable way, has gone from a specialist concern to a requirement for anyone building production AI.

This affects how organizations should think about closing the demo-to-production gap. The investment in data engineering skills needs to keep pace with the investment in AI itself.

What production-ready AI actually requires

Organizations that make it out of the pilot stage treat data infrastructure as a first-class concern from the start. That means building real-time pipelines rather than batch processes. It means applying schema definitions, ownership metadata, and quality checks at the point of data production rather than in the data lake. And it means structuring data as reusable products that different teams and applications can build on, so the engineering work supporting one AI application can accelerate the next one, rather than starting from scratch.

The 2026 report found that 88% of IT leaders said data streaming platforms help address data infrastructure and quality issues for agentic AI. That’s because they address the specific reasons AI projects stall — real-time data delivery, upstream governance, and making data trustworthy enough to use at inference time.

The shift is already happening

For the first time, the report found that investments in data streaming outranked those in AI and machine learning, by 88% to 82%. Organizations that have tried to ship production AI are increasingly recognizing that the model isn’t the hardest part. 

“For the first time, the report found that investments in data streaming outranked those in AI and machine learning, by 88% to 82%.”

So if you’re stuck at the pilot stage, resist the urge to keep optimizing the model. A better question is whether the data feeding the model is fresh, accurate, and well-governed, and whether your pipelines were actually built for production AI or a demo that only had to work once.

The post The impressive AI demo is dead. Here’s what actually reaches production appeared first on The New Stack.

Agentic systems usually have two jobs: Build context, then use that context to produce an answer or action.

Many failures that look like LLM problems start in the context-building step. The answer the LLM gives is limited by the context it was given, or it finds through tool calls. If the agent model cannot find the right sources, then improving the generation model will not improve the overall system.

“Many failures that look like LLM problems start in the context-building step.”

A client, Specstory, wanted to give users the ability to ask questions from the agent’s history. For example, why a team chose Authlib for authentication and what alternatives they considered. The chatbot needs the right prior conversations, decisions, and tradeoffs from a large corpus of coding sessions. The model and system prompt help only after those chat turns have been retrieved and are in context.

If retrieval ranks implementation snippets above the discussion where the team weighed alternatives, the agent can still produce a confident answer. It may find code that imports Authlib and a few inline comments, then describe the decision based on implementation evidence rather than the actual trade-off discussion.

The same pattern showed up in an AnkiHub operator review in our private community. A request for help studying based on lecture slides only works if the agent’s tool calls retrieve the right flashcards. The hard part is not finding any related cards. A lecture on the function of the heart may match hundreds of cards. Ranking decides whether the core cards make it into context or whether the system has to raise top_k and flood the prompt.

The exact setup changes by product. The context-building step might use local search, semantic search, web or API calls, or database queries. It might be handled by an agent, a fixed workflow, or application code. The process stays the same: gather the right context, then generate from it.

For example, a coding agent runs rg, opens files, reads logs, and inspects tests before writing a patch. A research agent searches the web and internal notes before writing an answer. A study assistant searches deck facts and user context before suggesting what to learn next.

When context building fails, the symptoms look like generation failures.

Retrieval failures mimic generation bugs

A better model helps with reasoning and writing, but it cannot give a better answer without the right context.

“A better model helps with reasoning and writing, but it cannot give a better answer without the right context.”

The Mixedbread OfficeQA-Pro Eval shows the same pattern at the benchmark scale. OfficeQA-Pro uses 89,000 pages of financial documents, dense tables, scanned PDFs, and questions that require reasoning across documents. Giving Codex better search tools reduced tool calls and improved answer quality.

Plain-text tools like grep and rg work (ish) on flat code files. They do not work well when context lives in PDFs, tables, chat histories, multi-modal inputs, web results, and permissioned data. In those cases, the agent needs a retrieval that can combine exact terms, meaning, metadata, permissions, and ranking quality.

Retrieval needs traces and evals

Once retrieval enters the architecture, the next question is whether it finds the right information.

For that, you need traces and evals. For each retrieval step, the minimum trace is the input, the outputs, and a way to label whether each output was relevant.

For a coding agent using rg the input is the command, the output is the returned snippets, and the label says which snippets helped, which were noise, and which relevant files were missing.

For product retrieval, the step might be BM25, semantic search, hybrid search with reranking, a generated SQL query, or something else. Capture the query or arguments, the returned documents or chunks, and whether those results were helpful.

Trace each retrieval step by itself, then evaluate the full context-building pass. The local trace answers “Did this query return useful material?” The full trace answers “Did the system collect everything the model needed before generation?” If it did, failures are a generation problem. If not, it’s a retrieval problem.

You cannot know where the failure started or what to fix without traces.

Different failures need different fixes

“Improve retrieval” is too broad to be useful, as different problems require different solutions. If a relevant document is missing, the trace should show where it disappeared: query building, retrieval, filtering, ranking, or final context assembly.

The failed step, plus what the trace shows, tells you what change to make.

The right fix depends on what the system was trying to retrieve. A decision-history question requires the decision, the alternatives, and the chats in which the team worked through them. A study question depends on the lecture material, deck metadata, semantic matches, and the user’s study context.

The architecture

Once you trace individual retrieval calls, the full architecture has a simple shape: fan out to context-building tools, then fan in to generate the final output.

The retrieval layer might be a search engine, a vector database, an SQL query, a local file tool, a web search API, or a custom service. The pattern stays the same: build candidate context, narrow it, rank it, assemble it, then generate from it.

Give agents human search controls

Semantic search compares embeddings (numerical representations of meaning). It helps when wording differs, but most retrieval intents also depend on structured constraints. A meeting search box can use semantic search over transcripts and notes, but a useful interface also lets someone filter by person, date, project, and source. 

A finance search may need the latest filing, a specific quarter, or an official source in addition to the closest semantic match. In e-commerce, the best semantic match for “32×30 cargo pants” may be an out-of-stock product. The system still has to decide whether to hide it, return it with a backorder note, or show it so the user can check later. That product decision is a retrieval decision because it changes which candidates reach the agent.

In a chat interface, those controls are in the tool schema, query planner, or app logic. If an agent runs the search, it needs arguments for the same constraints a human would set with filters, sliders, tabs, and sort menus.

A retrieval system usually needs several controls working together:

Here, a chunk means a small piece of source content, and a candidate is a chunk returned by the first search step. Ranking is the scoring step that orders those candidates. Context assembly then selects which chunks and structured fields to include in the model prompt.

Better ranking improves precision, which means a larger share of the returned chunks is useful. If the relevant chunks are near the top, the system can pass fewer chunks to the model, use fewer tokens, reduce latency, and expose the model to less noise. If the right chunk is ranked 40th and the context only includes the top 10, the system behaves as if the retrieval missed it.

People and agents use the same basic search path: ask for results, inspect what comes back, and decide what to use. A person can skim ten search results, compare titles, snippets, dates, domains, and URLs, and decide whether the result set looks right. They can open the third result, ignore the rest, and search again with a better query. An agent usually receives a bounded set of returned documents and reasons from the context. If the right source falls below the cutoff, the agent may answer from partial context. To avoid this, the system has to retrieve more candidates, run more searches, or pass more evidence into the model.

A missed document can change what the agent searches for next. Suppose someone asks why the team chose Authlib. If the first search misses the transcript where the team compared Authlib with alternatives, the agent may search the codebase instead. It finds imports, callback handlers, tests, and maybe a comment. Then it asks follow-up questions about OAuth configuration. The context starts to look complete, but it supports the wrong answer. It explains how Authlib was used and why it makes sense in the codebase, not why the team chose it.

“The context starts to look complete, but it supports the wrong answer.”

But ranking cannot repair every search problem. If the agent failed to request the latest filing, a reranker may faithfully select an older document with a closer wording match. If the tool has no date_range, person, source_type, size, or in_stock argument, the model has to impose hard constraints in the search text and hope that retrieval infers them. A hard filter gives the model less to infer, making semantic search more reliable.

Scale changes the retrieval problem

Search systems already have tools for this: indexes, filters, facets, sorts, caching, bounded reranking, and freshness jobs. Agent systems need the same discipline.

A human might search, adjust a date filter, scan the first page, then search again. One agent request can do that many times in seconds: rewrite the query, run keyword and semantic search, inspect thin results, issue follow-up searches, fetch sources for citations, and ask for more context before answering. With many concurrent users or agents, the retrieval layer can become a bottleneck.

Humans often wait through a slow search if the result is good. Agent systems often turn slow and uncertain search into more work. When ranking is weak, teams compensate by raising top_k, running keyword and semantic searches in parallel, adding reranking, fetching more source documents, and passing larger evidence bundles to the model. That can improve answers, but it moves the cost into tokens, latency, and retrieval load. A better ranking lets the system return fewer, better candidates, rather than making every request carry a larger pile of possible evidence.

With a small corpus, you can still search comprehensively quickly and cheaply, even with fully agentic approaches. That’s what I recommend when you’re starting and don’t have much data. Don’t add complexity until you need it. But with millions or billions of chunks, every extra retrieval call, candidate, ranking pass, and returned token adds up quickly.

Multi-stage retrieval is the production shape

Most production systems should split retrieval into stages, even when the UI is a chat box.

Each stage leaves a different repair path. If the agent chose the wrong filters, changing the embedding model will not help. If the latest document was available but the tool never sorted by date, the fix belongs in the search arguments or retrieval API. If the right source was present but below the cutoff, the fix belongs in the ranking. If the right source came back but was dropped before generation, the bug is in context assembly.

As retrieval becomes a core part of agent architecture, teams increasingly need infrastructure that can combine semantic search, exact matching, filtering, ranking, and large-scale retrieval in a single system. Depending on requirements, this may involve search and retrieval platforms such as Vespa, Elastic, or Coveo, each of which supports different approaches to ranking, retrieval, and operational scale. 

The important point is not the specific technology choice, but recognizing that retrieval quality has become a first-class engineering concern. As agent workloads grow, retrieval systems are increasingly determining the accuracy, cost, latency, and reliability of the overall application.

The post Why retrieval quality is becoming the defining challenge in AI agent architecture appeared first on The New Stack.

Amazon EKS runs hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Operating at that scale has taught us something that has shaped how we build the service and that we think is useful to anyone running Kubernetes at scale: most availability problems do not stem from a component failing. They come from a component reacting to a problem in a way that makes it worse. A cache that goes stale and serves wrong answers. A health check that restarts the very process keeping a cluster alive.

What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage. This post is the story of how we keep the EKS-managed Kubernetes control plane on the right side of that line at ever-growing scale: the foundational changes we made and why, and what operating at fleet scale taught us about building systems that tolerate faults rather than spreading them. These are the reasons our most demanding customers confidently run their mission-critical workloads on EKS.

How AI and analytics workloads reshaped what “scale” means

Kubernetes was built for a particular rhythm of work. Pods came and went at predictable rates, and controllers had seconds or minutes to reconcile. The system’s design reflected that pace: strong data consistency, ordered watches, and consensus-replicated storage that puts correctness first. It worked beautifully for what it was designed to do, and it still does.

“What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage.”

But the workloads evolved faster than anyone anticipated. Foundation model training runs scale-up training jobs on thousands of GPU nodes in minutes. Real-time inference services scale from a warm baseline to thousands of replicas, then drop back within the hour. Apache Spark analytics pipelines burst from zero to tens of thousands of executor pods, chew through a dataset, and vanish. 

Emerging agentic AI workloads add yet another dimension: autonomous agents that spin up, fan out, execute tasks, and tear down in seconds or less. These workloads share a trait that distinguishes them from traditional microservices: they generate enormous volumes of state transitions within compressed time windows and are deeply intolerant of delays. This velocity of state change pushed us to reinvent some of the mechanics to support a scale that was previously impossible, and to contribute what we could upstream.

How EKS reimagined Kubernetes storage foundation

Every Kubernetes cluster depends on etcd as its source of truth. Every application, every service endpoint, every scheduling decision is stored there. If etcd loses data, the cluster forgets everything it knows. Protecting that state is the most important job for a managed Kubernetes service.

Operating etcd for one cluster is well understood. Operating it for a fleet of millions is a different problem entirely. Hardware fails, networks blip, and disks degrade, so something has to handle those events without a human in the loop. And the operations etcd needs most, like replacing a failed member or recovering after a zonal event, are exactly the ones where a person acting under pressure can make a mistake that causes permanent data loss.

From the beginning, we built an operator agent that runs alongside every etcd instance and automates its entire lifecycle. The agent has two jobs. First, backup and recovery: it takes point-in-time snapshots and stores them durably outside the cluster. If too many instances are lost at once and the survivors cannot form a majority, the agent automatically detects the condition and rebuilds from the latest snapshot. Second, membership management: when an instance fails, the agent removes the terminated member and adds its replacement in an order that protects quorum and prevents split-brain.

A recent, more fundamental change was replacing etcd’s consensus mechanism, Raft, with a purpose-built journal that provides durable, ordered storage independently of etcd. In traditional etcd, a majority of members must agree on every write before it is committed. If two of three are unhealthy, the cluster becomes unavailable. 

By offloading durability to the journal, etcd peers no longer negotiate quorum among themselves. Writes commit as soon as the journal acknowledges persistence, and that entire class of etcd quorum-loss failures disappeared. Since the journal handles persistence, etcd no longer needs to fsync writes to local disk, so its data store has moved to an in-memory filesystem. What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path. For a deeper look at this architecture, read “Under the hood: Amazon EKS ultra scale clusters.”

“What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path.”

For ultra-scale clusters, we went further and partitioned etcd into resource-specific shards. Each partition operates independently with its own storage budget and throughput capacity. The primary value is failure isolation. In a monolithic deployment, if the events keyspace exceeds its quota because a misbehaving controller creates objects faster than garbage collection can remove them, it blocks writes to everything, including node leases. 

Suddenly, healthy nodes appear unhealthy because their lease renewals are being rejected. With partitioned etcd, the events partition hits its quota, but the leases partition continues operating normally. Nodes remain healthy. The scheduler keeps running.

What replacing etcd’s consensus mechanism unlocked

Removing the quorum requirement allowed us to make a change we had wanted for a long time: running etcd on the same host as the API server. In the traditional layout, every read and write crosses the network between separate machines. Each trip is fast on its own, but at thousands per second, the travel time adds up. With collocation, the API server talks to its local etcd over a loopback interface, and pod scheduling and controller reconciliation get measurably faster. For workloads where job controller queue depth is the binding constraint, shaving milliseconds off each API call means many more jobs are processed per second before the queue starts growing.

This is where the operator agent paid off. When etcd runs on the same host, an etcd member comes and goes whenever a control-plane host is replaced, which happens routinely. That only works if membership management is completely safe and automatic, which is exactly what the agent was already doing. We did not have to build a colocation from scratch; we built it on top of infrastructure that had been managing etcd membership safely since day one.

Collocation also taught us a lesson worth passing on: the convenient path needs a failover in case it breaks. The local etcd is the fast path, but if it becomes impaired, the API server fails over to another etcd member that is actively serving other API servers from the same journal. When you optimize for the common case, design just as deliberately for the moment that optimization is not available.

Fixing bottlenecks across the stack

At extreme scale, you have to address bottlenecks across the entire Kubernetes stack, and most of them are not bugs in the traditional sense. They are design choices that were correct at the scale Kubernetes originally targeted and break down only when the numbers get large. Rather than working around them internally, we fix them upstream so the entire community benefits.

One example involved the watch cache, the in-memory layer that distributes state changes from etcd to every controller watching for updates. When a controller starts, it requests a full snapshot of the current state via a mechanism called WatchList, and the existing implementation holds a shared read lock for the duration of the response build. 

At hundreds of thousands of objects, that work runs long enough to starve the writer that needs exclusive access, so the cache’s resource version cannot advance. Consistent reads see a stale cache and fail over to etcd, while the response building churns through hundreds of thousands of allocations under the lock. We identified this as a limitation in the watch-cache’s locking model and are working with the community to refactor the underlying data structures and interfaces to eliminate the contention.

The same shape appears elsewhere. In the Horizontal Pod Autoscaler, a single mutex protecting the scaling state becomes a serialization point at high HPA counts, where workers spend nearly all their time blocked rather than doing useful work. A redesigned data store (PR #139142) restores parallelism and raises reconciliation throughput by orders of magnitude. In the scheduler, we identified a bottleneck (issue #138426): every scheduling cycle rebuilds a set of in-use persistent volumes by scanning every node in the cluster, even for pods that do not use storage at all. The fix computes that information lazily, and only for pods that actually need it, restoring throughput at scale.

Each of these started from a real production workload hitting a cliff, and we are working on the fixes upstream so the improvements reach every Kubernetes user.

From engineering to guarantees: EKS Provisioned Control Plane

The engineering described above made the EKS control plane more resilient and performant. But customers had a different problem: they could observe that the control plane kept up today, but they could not reserve its capacity the way they reserve compute or GPU capacity. 

A team planning a thousand-node training run could secure the instances weeks in advance, yet had no equivalent mechanism for the orchestration layer that would coordinate them. EKS Provisioned Control Plane fills that gap. It exposes the control plane’s performance as dimensions you size explicitly, backed by the same kind of commitment you expect from the rest of your infrastructure.

You choose a scaling tier that maps to concrete, measurable capabilities: API request concurrency, pod scheduling rate, and cluster database size. The tiers range from XL through 8XL. At the top end, 8XL on Kubernetes 1.34 provides 16,000 concurrent API request seats, 400 pods-per-second scheduling rate, and 16 GB of cluster database storage, all backed by a 99.99% availability SLA measured in one-minute intervals.

Tiers are not static. You step up before a GPU training run or a large sales event, step back down during quiet periods, or grow permanently as your platform matures. Configuration happens through the console, CLI, eksctl, CloudFormation, or Terraform on any cluster, without recreation or downtime. 

For AI workloads, orchestration capacity is planned alongside GPU capacity, available when the compute comes online. For analytics platforms submitting hundreds of jobs per minute, the control plane is ready for the burst before it arrives. And for organizations that need environmental consistency across staging, production, and disaster recovery, the same tier guarantees consistent performance characteristics everywhere.


Taking the same foundation to the edge

Some workloads cannot move to the cloud, whether due to data sovereignty requirements, latency constraints, or unreliable connectivity to the Region. Running Kubernetes in these disconnected environments introduces unique challenges: etcd must remain durable on hardware with only a few machines, the cluster must self-heal without reaching the cloud, and observability must survive network partitions that last days. 

With the updated architecture for EKS local clusters on instance store Outposts, we brought edge clusters onto the same management plane and software stack as EKS clusters in the cloud.

The control plane lives in an EKS-managed account on the Outpost rather than in the customer’s account, so customers never manage control plane instances, etcd backups, or logging agents themselves, and they cannot accidentally break the thing keeping their cluster alive. The same machine images, container images, and operator agent run in both places, with edge-specific behaviors selected by configuration. 

Because it is the same stack, new Kubernetes and EKS platform versions arrive in lockstep with their cloud release, and features like EKS add-ons, Pod Identity, and access entries work the same way they do in a Region.

The hardest part was keeping etcd healthy on hardware with only a few machines that may be cut off from the cloud for days at a time. We solved it by extending the same agent. It keeps a spare copy of the data continuously up to date and promotes it the instant a machine fails, so the cluster heals itself with no human involvement and no connection to the cloud. 

Observability survives the disconnect, too: the metrics agent continues collecting and writing to local disk, shedding the least critical data first when space runs short, so the signals that matter most are the last to go. When the link returns, the buffered data is flushed back with its original timestamps.

All of this only works because the system was designed from the start to operate without anyone logged in. That same design is what makes it possible to deploy changes safely across the entire fleet.

Operating safely at fleet scale

Every one of these changes was deployed to a running fleet of hundreds of thousands of clusters. The journal migration and collocation required transitioning each cluster individually. Every migration follows a strict sequence: validate pre-conditions, create a point-in-time snapshot, perform the switchover, validate post-conditions. If any step fails, the system rolls back automatically. 

Rollouts proceed cell by cell, zone by zone, region by region, with automated monitoring comparing latency, error rates, and throughput between updated and non-updated clusters. Any statistically significant deviation triggers an automatic halt.

What made all of this possible is that EKS is built to operate without human intervention at the individual cluster level. Through Zero Operator Access, the architecture prevents AWS personnel from having technical pathways to access customer content in the managed control plane. A system designed to work without human access must be observable, recoverable, and automatable from the start, and that same discipline is what enables operating at extreme scale.

Three operational lessons shaped how we approach this work.

The first is that a healthy leader is not the same as a working one. The control plane’s controllers run in an active-passive configuration, and early on, we treated an unhealthy standby as if cluster operations had halted. They had not; what matters is whether a leader exists. But the harder lesson: a leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth. If the queue fills while the leader looks healthy, the system is falling behind in ways no liveness probe will catch.

“A leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth.”

The second is that maintenance ordering matters as much as the maintenance itself. etcd defragmentation is blocking, and the pause grows with database size. When it hit the leader, every write stalled. We taught the agent to move leadership to a healthy node before defragmenting, so the disruptive work always lands on a follower while writes keep flowing.

The third is that liveness is not readiness. A process can be alive but not ready while it warms caches, and routing based solely on liveness sends requests to an instance that cannot handle them. Equally, readiness flapping during graceful draining should never trigger a restart. We keep the two signals strictly separate: one decides recovery; the other decides routing.

None of this work is visible from the outside, and that is the point. The largest clusters taught us lessons that made every cluster faster. The riskiest migrations produced safety machinery that protects every upgrade. The upstream fixes we contributed for workloads at the edge of what Kubernetes can handle flow back to every user of the project.

“None of this work is visible from the outside, and that is the point.”

When you deploy on EKS and your pods come up in seconds, even during a burst, even when something behind the scenes goes wrong, that speed is not accidental. It is the accumulated result of years of operating at scales where small problems can become big ones fast, and engineering the system to contain them before they do.

To explore the architectures referenced in this post, see EKS Provisioned Control Plane and local Amazon EKS clusters on AWS Outposts.

The post Operating Kubernetes at scale: a few stories from running Amazon EKS appeared first on The New Stack.

Many agentic projects can sail through development just fine. Then they hit security review — and that’s where things can grind to a halt. Unclear identity models and overly broad permissions quickly become blockers.

You’ve probably seen this play out: A customer support agent is working well; it triages tickets and processes refunds, handles the whole workflow without a hitch. Then security asks a simple question: Under whose identity is this running? The answer stops the process cold: It’s a shared account with broad permission, no clear ownership, no audit trail, and no least-privilege controls in sight.

The root issue isn’t complicated. It’s undefined identity and poorly scoped permissions. And that challenge is accelerating fast. Research from the 2026 Tech Leader Study, conducted with Oxford Economics and IBM, shows surveyed enterprises expect to deploy an average of 1,661 AI agents, a 38% increase from today. Each new agent introduces another identity to secure, and without clear boundaries, the problem compounds quickly. 

As a result, many agentic systems focus on what agents can do without defining what they should do, or under whose authority. Agents also don’t hold a fixed set of permissions. They request access, call new tools, and assume roles as they work, so access paths compound in ways no one explicitly granted or reviewed. Without a verifiable identity, there’s no accountability, making least-privilege enforcement, traceability and incident response difficult.

“Each new agent introduces another identity to secure, and without clear boundaries, the problem compounds quickly.”

To address these gaps, this guide is written for developers, architects and DevOps engineers building agentic systems — and for the IT leaders responsible for approving them.

The four identity decisions every agentic system must make

Identity decisions can’t be treated as an afterthought. Identity shapes how agents authenticate, what they can access, and how their actions are controlled and audited over time. Get it wrong early, and you’re building on a shaky foundation.

Here are the four decisions that matter most:

Workload identity vs. shared service accounts 

Shared service accounts are easy, and that’s exactly what makes them dangerous. When multiple agents act under a single identity, it becomes hard to tell what happened or what went wrong after the fact. If an account is leaked or misused, everything it touched is exposed. 

“Shared service accounts are easy, and that’s exactly what makes them dangerous.”

Workload identity assigns each agent its own identity. Permissions stay scoped, and actions are attributable. It requires more setup but creates isolation and auditability.

Static API keys vs. short-lived credentials

Static API keys tend to stick around forever. They get hardcoded into apps, passed around between systems, and rarely rotated — which makes them a persistent vulnerability waiting to be exploited. 

Short-lived credentials work differently. They’re issued on demand, scoped to a specific task, and expire automatically. In practice, this often relies on identity federation (for example, using OIDC tokens) combined with systems that can issue dynamic credentials at runtime, rather than storing long-lived secrets in code or configuration

Direct credential handoff vs. brokered session access

Handing credentials directly to an agent is simple. It’s also opaque. You don’t have a natural point to evaluate policy or understand what’s happening in real time. 

Brokered access introduces a control point into the flow. Requests go through a broker, policies are evaluated in real time, and temporary credentials are issued per session. It adds infrastructure, but restores visibility and policy enforcement. 

Fragmented logging vs. full identity lineage

Most systems log what happened. Far fewer capture who initiated it or how an action propagated through a chain of agents and services. 

Full identity lineage connects every step. You can trace an operation from triggers to outcomes, which can make debugging faster and enable more credible incident response. The catch is that this requires consistent identity propagation and structured logging from the beginning—it’s hard to retrofit.

When these tradeoffs become real risks

These aren’t abstract architectural preferences. They show up as concrete vulnerabilities.  

Nightfall AI reports that organizations expose nearly 350 secrets per 100 employees each year, with 35% of exposed API keys still active. Combine that with persistent credentials and shared identities, and the potential blast radius grows fast. 

The pattern is consistent: shared accounts and long-lived keys are faster to build but harder to secure. Workload identity and short-lived credentials require more upfront investment but can deliver more security over time.

Debugging breaches by feel

Think about what happens when an agent running on a shared account with a long-lived key suddenly spikes its data access. Was it a bug? A breach? Routine behavior? Hard to say. Revoking the key might stop the issue, but it could also break a half-dozen unrelated workflows in the process. You’re now debugging by feel. 

Shortcuts reduce friction at the start and accumulate risk over time. 

Standardize identity at the platform layer

The answer isn’t to rebuild authentication, authorization and auditing from scratch for every agent you ship. That’s not scalable. 

Instead, standardize identity at the platform layer—centralized identity providers, policy engines and a credential broker to enforce secure defaults and make compliance straightforward rather than a constant negotiation. 

“Shortcuts reduce friction at the start and accumulate risk over time.”

Agentic AI works in production when identity is designed up front and enforced at runtime, rather than assumed from a prior login. When projects are treated as an afterthought, they stall. When it’s built in deliberately, agents can operate with the control that production environments demand. 

IBM supports this through an integrated identity-first approach that spans secrets management, secured access, and identity governance—helping organizations scale agentic systems securely without adding operational complexity. 

Learn how IBM approaches identity-first security for agentic systems

© Copyright IBM Corporation 2026. IBM and the IBM logo are trademarks of IBM Corp., registered in many jurisdictions worldwide. Examples presented are illustrative only. Actual results will vary based on client configurations and conditions; therefore, general expected results cannot be provided.

The post The AI agent identity problem nobody’s talking about appeared first on The New Stack.

Modern businesses are in a constant, uphill battle against what to do with unstructured data: PDFs, contracts, scanned images, customer call recordings, meeting videos, and more. Traditional document automation workflows that rely heavily on template-based extraction or rigid rules used to make sense. But document formats have changed; they’re diverse and don’t fit standard formats, making costly, brittle, traditional systems a relic of the past. 

“Modern businesses are in a constant, uphill battle against what to do with unstructured data.”

Enterprises demand faster, more accurate processing, which raises the question: How can we reliably turn messy, multimodal content into structured, actionable insights without a mountain of manual effort?

That’s where Amazon Bedrock Data Automation (BDA) comes in.

What is Amazon Bedrock Data Automation (BDA)?

Amazon Bedrock Data Automation (BDA) is a generative AI-powered, fully managed service on Amazon Web Services for end-to-end document and media automation. It enables users to automate the extraction, classification, and transformation of unstructured content across modalities such as documents, images, audio, and video.

“At its core are Foundation models which enable intelligent extraction and understanding of content.”

At its core are Foundation models (FMs) which enable intelligent extraction and understanding of content. It allows users to configure standard output for common use cases, or even define custom extraction logic using blueprints tailored to your business. BDA is designed for scalability, accuracy, and auditability, making it ideal for enterprise workflows.

Walk-through: creating a project, standard output & custom output using blueprints

1. Create a project via console

In the Amazon Bedrock Console, navigate to Data Automation → Create Project.

Enter the name of the project:

2. Standard output:

Standard output gives you the model’s default, unstructured response (text, image, audio, or video) directly from the Data Automation pipeline.

In standard output, each modality has its own options for what is needed as an output. 

Document:

Image & Video:

Audio:

Now let’s test Document Modality for Standard Output:

First, click on “Test” in the upper right corner.

Next, select the document from the system, sample, or S3 and choose the modality from the dropdown menu. 

Click on the “Generate results” button:

After processing, it will show the summary and content of the document:

Custom output (blueprints):

Custom output lets you define a structured, predictable format using blueprints, which ensures the output matches your exact schema, fields, and business rules.

Let’s test custom output using blueprints for the same document:

Navigate to “Custom output” and click on “Add Blueprint”:

From here, two options will appear. You can either use LLM power to generate the blueprint (where it inspects the document), or you can choose to enter field names, instructions, and other information manually.

Below is a blueprint generated by LLM which has pulled all possible fields and tables from the document:

It has extracted the information using the blueprint as demonstrated below, including the  Field name, Instruction, and Results:

It also provides the Extraction type (which can be Explicit or Inferred), Confidence percentage, and other relevant information.

Additionally, it can extract information in the form of a table, such as an account summary or transaction information:

Code examples

Amazon Bedrock Data Automation (BDA) Utility Module
Description:
    Helper functions to create BDA projects, blueprints, invoke jobs,
    monitor job status, and fetch results.

import boto3
import time
import json
import botocore

class BedrockDataAutomation:
    def __init__(self, region="us-east-1"):
        self.bda = boto3.client("bedrock-data-automation", region_name=region)
        self.runtime = boto3.client("bedrock-data-automation-runtime", region_name=region)

    # ------------------------------------------------------------
    # BLUEPRINT OPERATIONS
    # ------------------------------------------------------------
    def create_blueprint(self, name, schema, description="", stage="LIVE"):
        """
        Create a BDA Custom Output Blueprint from a JSON schema.
        """
        print(f"Creating blueprint: {name}")

        response = self.bda.create_blueprint(
            blueprintName=name,
            blueprintStage=stage,
            type="DOCUMENT",
            schema=json.dumps(schema)
        )
        return response["blueprint"]["blueprintArn"]

    # ------------------------------------------------------------
    # PROJECT OPERATIONS
    # ------------------------------------------------------------
    def create_project(self, name, description, standard_output_config, custom_output_config=None):
        """
        Create a BDA Project with Standard or Custom Output.
        """
        print(f"Creating project: {name}")

        response = self.bda.create_data_automation_project(
            projectName=name,
            projectDescription=description,
            projectStage="LIVE",
            standardOutputConfiguration=standard_output_config,
            customOutputConfiguration=custom_output_config or {}
        )
        return response["projectArn"]

    # ------------------------------------------------------------
    # INVOCATION OPERATIONS
    # ------------------------------------------------------------
    def invoke_project(self, project_arn, profile_arn, input_s3_uri, output_s3_uri, blueprints=None):
        """
        Invoke a BDA project using async invocation.
        """
        print(f"Invoking project: {project_arn}")

        kwargs = {
"inputConfiguration": {"s3Uri": input_s3_uri},
"outputConfiguration": {"s3Uri": output_s3_uri},
"dataAutomationConfiguration": {
"dataAutomationProjectArn": project_arn,
"stage": "DEVELOPMENT"
},
"dataAutomationProfileArn": profile_arn
}

        if blueprints:
kwargs["blueprints"] = blueprints

        response = self.runtime.invoke_data_automation_async(**kwargs)
       invocation_arn = response["invocationArn"]

       print("Invocation ARN:", invocation_arn)
       return invocation_arn

    # ------------------------------------------------------------
    # JOB STATUS POLLING
    # ------------------------------------------------------------
    def wait_for_job(self, invocation_arn, poll_interval=10):
        """
        Poll until job finishes.
        Returns final status object.
        """
        print("Polling job:", invocation_arn)

        while True:
            try:
                resp = self.runtime.get_data_automation_status(
                    invocationArn=invocation_arn
                )
            except Exception as e:
                print("Error fetching status:", e)
                raise

            status = resp["status"]
            print(f"Status: {status}")

            if status in ("SUCCEEDED", "FAILED", "CANCELLED"):
                return resp

            time.sleep(poll_interval)



# --------------------------------------------------------------------
# EXAMPLE USAGE
# --------------------------------------------------------------------
if __name__ == "__main__":
    bda = BedrockDataAutomation(region="us-east-1")

    # 1. Create Blueprint
    blueprint_schema = {
        "type": "object",
        "properties": {
            "account_holder": {"type": "string"},
            "balance": {"type": "string"},
            "transactions": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "date": {"type": "string"},
                        "description": {"type": "string"},
                        "amount": {"type": "string"}
                    }
                }
            }
        },
        "required": ["account_holder", "transactions"]
    }

    blueprint_arn = bda.create_blueprint(
        name="BankStatementBlueprint",
        schema=blueprint_schema,
        description="Extract fields from bank statements."
    )

    # 2. Create Standard Output Config
    standard_config = {
        "document": {
            "extraction": {
                "granularity": {"types": ["PAGE", "LINE"]},
                "boundingBox": {"state": "ENABLED"}
            },
            "outputFormat": {
                "textFormat": {"types": ["PLAIN_TEXT", "CSV"]}
            }
        }
    }

    # 3. Create Project with Custom Blueprint
    project_arn = bda.create_project(
        name="BankStatementProject",
        description="Process PDF bank statements",
        standard_output_config=standard_config,
        custom_output_config={
            "blueprints": [
                {
                    "blueprintArn": blueprint_arn,
                    "blueprintStage": "DEVELOPMENT",
                    "blueprintVersion": "1"
                }
            ]
        }
    )

    # 4. Invoke the project
    # Ensure you replace <ACCOUNT_ID> with your actual AWS Account ID
    profile_arn = "arn:aws:bedrock:us-east-1:<ACCOUNT_ID>:data-automation-profile/us.data-automation-v1"
    invocation_arn = bda.invoke_project(
        project_arn=project_arn,
         profile_arn=profile_arn,
        input_s3_uri="s3://your-bucket/input/statement.pdf",
        output_s3_uri="s3://your-bucket/output/",
        blueprints=[
            {
                "blueprintArn": blueprint_arn,
                "version": "1",
                "stage": "DEVELOPMENT"
            }
        ]
    )

    # 5. Poll job status
    final_status = bda.wait_for_job(invocation_arn)
    print("Final status:", json.dumps(final_status, indent=4))

Types of document blueprints

When processing documents, BDA supports five core automation types:

1. Classification: invoice, bank statement, ID card, contract, HR letter, etc.

2. Extraction: Extract entities, fields, tables, metadata.

• Example: From a bank statement → Date, Description, Amount, Balance.

3. Transformation: Modify or restructure data.

• Example: Convert Home Address into separate fields -> street, city, ZIP code, etc.

4. Normalization: Standardize data values.

• Example: Convert multiple date formats (MM/DD/YYYY → YYYY-MM-DD).

5. Validation: Validate extracted fields against rules.

• Example: Amount must be numeric; dates must match the format; balances must reconcile.

Use cases that illustrate business value

Real-world scenarios where BDA provides significant ROI include:

• Financial Services: Automate processing of bank statements, invoices, and loan applications, reducing manual labor and speeding up reconciliation or underwriting.
  
• Insurance: Ingest and extract data from claims forms, medical reports, and damaged-asset photos.
  
• HR / Legal: Process resumes, contracts, and offer letters; extract structured data, including skills, clauses, salaries, and parties.
  
• Customer Support: Transcribe and summarize calls, extract intent and sentiment, and feed those insights into CRM or case systems.
  
• Security & Compliance: Analyze CCTV footage or meeting recordings to detect key actions, summarize context, and flag compliance issues.

BDA proves itself flexible and powerful, as it supports both standard outputs for basic workflows and fine-tuned custom schemas via blueprints. It is scalable and robust, with projects that enable batch processing and versions (development vs. live) for safe testing. It’s also audit-friendly, providing structured fields with types, normalization rules, and validation logic. 

“Compared with rule-based systems, foundation models achieve better semantic extraction across the board.”

A true key benefit is that BDA is multimodal across formats. Users can use the BDA framework to process documents, images, audio, and video. And, best of all, it’s highly accurate. Compared with rule-based systems, foundation models achieve better semantic extraction across the board. 

Amazon Bedrock Data Automation empowers businesses to transform unstructured, multimodal content into structured, trustworthy, and actionable data. With minimal setup, highly customizable blueprints, and a scalable project-based architecture, BDA helps organizations reduce manual workload and unlock insights faster.

The post Template-based data extraction is dead. Here’s what comes next. appeared first on The New Stack.

Kubernetes teams automate deployments without thinking about it. CI/CD pipelines fire dozens of times a day, autoscaling adjusts replicas in the background, rollback is muscle memory. But there is one category of automation where that confidence vanishes: letting a system change CPU and memory requests on a running workload without a human reviewing it first. 

And as AI inference lands on Kubernetes at scale, that hesitation is becoming hard to ignore, and increasingly expensive.

Why teams trust automation for change but not for constraint

We surveyed 321 Kubernetes practitioners at enterprise organizations earlier this year. The headline finding is one most practitioners will recognize immediately: 82% report high or complete trust in automated delivery controls. But 71% still require human review before applying resource optimization recommendations. Only 27% allow CPU and memory changes to be auto-applied, even within guardrails.

“Deploying code feels additive… rightsizing feels subtractive because you are removing safety margin from a running service, and the failure mode is fundamentally different.”

Those numbers describe a specific asymmetry. The same engineers who deploy to production dozens of times a day without hesitation slow down the moment automation wants to adjust resource allocation. And the survey data make it clear why. Deploying code feels additive. You are shipping new value, the rollback path is well understood, and if something breaks you usually see it right away. Meanwhile, rightsizing feels subtractive because you are removing safety margin from a running service, and the failure mode is fundamentally different.

As one practitioner in the survey put it: “Automated right-sizing carries a unique risk because it directly impacts the underlying stability of the application runtime. Unlike a code deployment that follows a tested path, resource changes alter the invisible contract between the workload and the scheduler.”

When you change resource requests, you change how Kubernetes schedules, prioritizes, and allocates resources. Those effects are not visible the way a code change is. You can’t trace them through a deployment pipeline. And you might not discover that something went wrong until two weeks later, when a traffic spike hits a threshold that didn’t exist at the old values. By that point, three other things have changed too, and proving causation is nearly impossible. The people responsible for those workloads are the same people who get paged at 2 a.m., and they know this.

Why AI workloads raise the stakes

That trust gap existed before inference workloads showed up. What’s changed is the cost of not closing it.

For a long time, teams could absorb the cost of manual oversight. They knew their workloads, had intuition for where the safe boundaries were, and the inefficiency of over-provisioning was a price worth paying for stability. GPU-accelerated inference workloads change that math. GPU compute is significantly more expensive per hour than CPU. The cost of over-provisioning is no longer a rounding error you can absorb quietly. And the workload behavior is less familiar, as inference jobs are bursty in ways teams haven’t built intuition for, traffic patterns shift as models are updated and usage changes, and the resource dimensions involved differ from what teams have spent years learning to tune.

That unfamiliarity compounds with scale. Rightsizing isn’t a one-lever problem the way horizontal scaling is. It involves, at minimum, CPU and memory requests and potentially limits for both, with four dimensions per workload, multiplied across hundreds or thousands of workloads per cluster. The survey data indicates that manual optimization breaks down at around 250 changes a day. Inference workloads push teams past that threshold faster than anything they’ve managed before, because the resource decisions are more frequent and the cost of getting them wrong is higher.

The economic case for automated rightsizing has never been stronger. The organization’s willingness to delegate hasn’t caught up because teams are being asked to trust automation with workloads they don’t yet have a track record with.

What the survey says about closing the gap

When we asked practitioners what would actually increase their trust in optimization automation, 48% said visibility and transparency into how decisions are made, 25% wanted proven guardrails, and 23% needed instant rollback.

Nobody asked for full manual control and very few asked for blind autonomy. What they described is automation that earns trust in stages, and that’s consistent with how the teams furthest along in their automation journey actually got there. They didn’t start with production. They started with a single namespace in a dev environment, observed the system’s behavior, compared recommendations with outcomes, and gradually expanded the scope. Different environments remained at different levels of automation maturity simultaneously, and that was intentional. Production carried more scrutiny than dev.

CI/CD followed the same curve, and the timeline is easy to forget. Most organizations took years to get from running their first automated pipeline to trusting it with production deploys without manual approval on every commit. Kubernetes resource automation is earlier in that same process, and AI workloads are extending the timeline because teams are building trust from scratch with a workload category that doesn’t yet have a track record.

Why automation design matters as much as capability

Some automation architectures deliver meaningful value only with full delegation. The system needs complete control to function the way it was designed to. That’s a form of forced autonomy, and it creates an adoption problem because it asks for exactly the level of trust that most organizations haven’t built yet. Force generally doesn’t work. Teams that feel pushed into a level of delegation they aren’t comfortable with tend to pull back entirely after the first incident.

The alternative is what I’d describe as adaptive autonomy: designing the system to work at every stage of the trust curve. A team still evaluating gets useful recommendations in read-only mode. A team ready to act but wanting boundaries can run guardrailed execution within limits they define. As confidence grows, the system handles more decisions autonomously while humans manage exceptions. And for environments where the track record supports it, closed-loop optimization runs in the background and becomes boring, which is the goal. Each stage is a legitimate operating mode, not a stepping stone you have to rush through.

That design distinction matters more with AI workloads than it ever did with traditional services, precisely because the trust-building process is starting from zero on workloads where the cost of getting it wrong is highest.

“Trust takes a long time to build and a single production incident to undermine.”

The other piece that makes this sustainable is rollout safety. Trust takes a long time to build and a single production incident to undermine. Start with the workloads showing the most headroom between requests and actual usage. Make changes incrementally, small enough that a bad outcome stays contained. Rollback needs to be fast and tied to the health signals the team already monitors. And start with opt-in, not opt-out. Let the teams willing to go first build a track record that others can look at.

The broader pattern

The 71% figure is sometimes read as resistance to automation. I think it’s a more accurate picture of how operational trust actually forms: conditional, earned over time, and moving at different speeds depending on what’s at stake. AI workloads are raising those stakes significantly, which means the path to trusted automation matters more now than it did when the cost of caution was just some unused CPU.

“Most of what gets written about Kubernetes optimization focuses on tooling capability, and the tooling is capable. The harder problem is the human one.”

Most of what gets written about Kubernetes optimization focuses on tooling capability, and the tooling is capable. The harder problem is the human one. If your team is managing AI inference workloads on Kubernetes and your optimization tooling is sitting in read-only mode, the question worth asking isn’t whether to trust the system. It’s whether the system is designed to let you build that trust gradually, starting where the stakes are low and expanding as the evidence supports it, on workloads where getting it wrong costs more than it ever has before.

The post Kubernetes teams trust automation to ship code but not to touch CPU, and AI is raising the stakes appeared first on The New Stack.