Microsoft is introducing new AI tools designed to help customers continuously streamline and automate the process of identifying and reducing their exposure to security risks.
The new tools come less than a week after OpenAI lost control of two of its security models when they infiltrated the servers of startup Hugging Face. The hack, Hugging Face added, involved “a swarm of tens of thousands of automated actions” that stole internal Hugging Face credentials. The OpenAI models achieved this feat by exploiting a zero-day flaw in Hugging Face’s data-processing pipeline to run malicious code that escalated the models’ access to the company’s high-value cloud and server clusters.
Microsoft’s announcements on Monday made no reference to the event, which OpenAI said was “unprecedented.” The company also didn’t say what would prevent the new tools from similarly going rogue.
Microsoft opened a new front in the AI security wars on Monday, unveiling its first custom-built cybersecurity model and a sweeping agentic defense platform — and making an argument that could reshape how enterprises buy AI: the future belongs not to the biggest model, but to the cheapest one that's good enough, routed intelligently.
The company announced MAI-Cyber-1-Flash, a compact security model developed in-house by its Microsoft AI (MAI) division, embedded inside MDASH, Microsoft's multi-agent harness for finding and fixing software vulnerabilities. Together, the company says, the system scores 96% on CyberGym — a benchmark measuring how well AI systems reason over large codebases to find real vulnerabilities — beating frontier models including Mythos, Gemini, and GPT, while cutting costs roughly in half compared to Microsoft's own current production configuration.
Alongside the model, Microsoft introduced Project Perception, an agentic security system that coordinates "red team" agents that hunt for paths to compromise, "blue team" agents that investigate and triage risk, and "green team" agents that remediate and harden defenses. Project Perception enters public preview on August 3.
In a wide-ranging interview with VentureBeat, Microsoft AI CEO Mustafa Suleyman made clear the company sees Monday's announcement as the opening move in a much longer campaign.
"We really do have a pretty significant data and harness and expertise moat, and that is enabling us to train models which are faster, better, cheaper, and I think this is genuinely the tip of the iceberg," Suleyman said. "We haven't been working on this for long. The next model is going to be pretty phenomenal."
Inside the 90/10 architecture that still depends on OpenAI's GPT-5.4
The most technically revealing detail in the announcement is not the model itself but how Microsoft deploys it. MAI-Cyber-1-Flash was designed to handle up to 90% of security tasks efficiently, while MDASH escalates the remaining 10% of exceptionally difficult problems to a larger frontier model — which, notably, is OpenAI's GPT-5.4. In other words, Microsoft's flagship security AI still leans on its longtime partner-turned-rival for the hardest work.
Asked to explain that relationship, Suleyman pointed to the harness, the orchestration layer that routes each incoming problem to the right model. "The harness is like a router," he told VentureBeat. "It's kind of like guardrails and a rule set of an organizing logic, which matches queries to... incoming problems to a model that suits the problem." The system has three components, he explained: the harness, the small and fast MAI-Cyber-1-Flash handling the bulk of queries, and GPT-5.4 sitting alongside as "just a generalist coding model."
Pressed on how a system reliant on OpenAI's model can outperform frontier competitors, Suleyman argued the performance comes from the whole system, not any single model. "These are very complicated, long, agentic loops which require storing state, drawing on another database, consulting best practice... handing back to a small model, writing a bunch of code, validating that that was correct," he said. "There's like hundreds of steps to solve that, and that's why it's really the system together that delivers the better performance."
And why GPT-5.4 specifically for the escalation tier? Cost, again. "GPT-5.6 is expensive. GPT-5.4 is incredibly good relative to its cost," Suleyman said. "The whole game here is to reduce the costs. Mythos and so on are extremely expensive models... we want to be able to deliver better performance for cheaper. That's what customers want." The arrangement captures Microsoft's evolving posture toward OpenAI: still a customer of the partnership that drew regulatory scrutiny in Brussels and Washington in 2024, but increasingly determined to own the layers of the stack where it believes it holds durable advantages.
Why token costs — not model quality — are becoming the real barrier to enterprise AI adoption
The economics may matter more than the benchmark. Microsoft says the new configuration delivers roughly 50% cost savings against the current MDASH setup, which runs a blend of GPT-5.4, 5.4 mini, and 5.3 codex. In security — an always-on workload processing enormous volumes of signals — token costs compound relentlessly, and Microsoft argues they have become the binding constraint for defenders.
Suleyman frames the cost issue as downstream of a harder physical limit. "The key barrier to adoption is access to chips, and cost is a function of chips," he said. "No matter how much money you've got, there's actually a limited supply of chips. Then trying to squeeze more model output on fewer chips is clearly super valuable."
He also described a broader enterprise backlash against frontier-model pricing. Companies initially maxed out on the best available models, he said, but "then they realize they're sort of paying... a phenomenal amount of money, and people are absolutely token maxing everywhere across their business. So there's a massive pushback to reduce cost everywhere."
That positions Microsoft to ride a market trend rather than fight it. Cost-efficient, near-frontier models have proliferated over the past year — from xAI's recent Grok release to a wave of Chinese models built on the same premise — and Microsoft is betting that as a platform company it can align itself with enterprise cost pressure. "The top model providers want you to use the most expensive model continuously, whereas because we are a platform, we're on the side of the enterprise," Suleyman said. "There's no point asking... Mythos what the capital of France is."
The 100-trillion-signal data moat Microsoft says no competitor can replicate
Every AI lab claims differentiation. Microsoft's claim in security rests on something genuinely hard to copy: telemetry. The company processes more than 100 trillion security signals daily — a figure consistent with its 2025 Digital Defense Report, which also cited 4.5 million new malware files blocked and 5 billion emails screened per day — and draws operational insight from 1.6 million customers.
"We have trillions and trillions of data points going back decades," Suleyman said. "It is, I think, the largest longitudinal cybersecurity dataset around," in part because Microsoft's customer base includes governments "who have been consistently attacked for years, and we have been consistently attacked." Asked directly whether this constitutes an advantage no competitor can match, Suleyman didn't hedge: "That is definitely a moat for us. Both the data and the expertise, and just the experience in the institution of going through that process."
The strategic logic is that cybersecurity functions as a live reinforcement-learning loop: defenders act, outcomes are observed, models improve. Microsoft argues that connecting actions to outcomes — what was exploited, what was contained, what was blocked — yields training signal that pure model labs simply cannot buy or manufacture.
There is real substance here, but the usual caveats apply. The CyberGym results come from Microsoft's own evaluation, the fine print shows the headline "96%" is actually 95.95%, and vendor-run benchmarks that pit an entire tuned agentic system against competitors' base models are not apples-to-apples comparisons. What Microsoft has measured is a full harness-plus-models configuration against what customers might otherwise assemble — arguably the commercially relevant comparison, but not a controlled model-versus-model test.
The dual-use dilemma: how Microsoft plans to keep a vulnerability-hunting AI out of the wrong hands
A model built to find challenging vulnerabilities in complex codebases is, by definition, a model that could find vulnerabilities for attackers. This is not a theoretical concern. Microsoft's own threat intelligence team, in joint research with OpenAI published in February 2024, documented nation-state actors from Russia, North Korea, Iran, and China probing large language models for reconnaissance, scripting, and vulnerability research. Its 2025 Digital Defense Report went further, warning that AI agents could eventually automate the entire attack lifecycle.
Suleyman said Microsoft is gating access accordingly. "We're very strict about who gets access to the model, and we're very careful about that," he said. "We constantly monitor the API and usage." An approved user, he added, "has to be seen to be having good intent, but also have technical competence." The rollout will be deliberately staged: "It's not going to be thousands next week. There will be tens, and then hundreds, and then thousands."
Microsoft says the model was evaluated by its AI Red Team, subjected to automated and expert-led adversarial exercises, and independently assessed by a third party, with deployment wrapped in tenant isolation, auditing, and sandboxed execution environments with no internet access.
Suleyman also offered a candid acknowledgment of Microsoft's positioning relative to the bleeding edge — one that doubles as a pitch to risk-averse buyers. "Even though we might be a few months behind the absolute cutting edge at any given moment... it matters that we're doing it very carefully and thoughtfully, and we have a track record of doing that," he said. For a company that spent 2024 absorbing hard security lessons — from delaying its Recall feature over privacy concerns to convening an industry summit after the CrowdStrike outage disabled some 8.5 million Windows devices — that trust-first framing is both strategy and necessity.
What Microsoft's superintelligence roadmap signals about the future of enterprise AI
Suleyman described a rapidly accelerating MAI roadmap, roughly nine months after Microsoft stood up its superintelligence team. "We have the compute that we need. We certainly have the data we need. We have the talent," he said. "Our momentum is accelerating rapidly." The top enterprise demand he's hearing is for "agents that can produce arbitrary code to solve whatever problem they direct them at," as vibe-coded internal tools graduate from experiments into production. The next phase, he said, pulls voice, transcription, image, and coding models "all integrated into the same harness."
Notably, Suleyman expressed skepticism about the industry's default assumption that everything eventually converges into one giant unified model. "It remains to be seen whether one giant model that is fully multimodal is actually able to deliver additional transfer learning benefit because of the integration," he said, "or whether it's just a big lumbering expensive giant."
That skepticism is the through line of the entire announcement. Microsoft is wagering that the unit of competition in enterprise AI is no longer the model at all — it's the system: the router, the specialized small models, the frontier fallback, and the proprietary data feeding the loop. In security, where Microsoft controls both the telemetry flowing in and the products that act on it, that wager is at its strongest. Whether it holds in domains where the company's data advantage is thinner remains the open question hanging over the MAI roadmap.
For now, though, Microsoft has offered the industry a preview of how it intends to fight the next phase of the AI race: not by building the biggest brain, but by building the best machine around it. As Suleyman put it, this is the tip of the iceberg — and Microsoft is betting everything on what sits below the waterline.
The same attacker broke into the same internet-facing Langflow server twice, and the second time brought ransomware built to destroy trained AI models. Sysdig's Threat Research Team documented the first campaign on July 1 and the second on July 20. The entry point never changed, but the payload changed completely.
Both ran through CVE-2025-3248, a missing-authentication flaw in Langflow's code-validation endpoint that lets anyone reaching the server execute Python on it. In the first, the agent improvised, encrypting 1,342 Alibaba Nacos configuration items with MySQL's own encryption function and dropping the tables. In the second, it staged ENCFORGE, a compiled Go binary sweeping roughly 180 file extensions.
ENCFORGE was built for AI assets, not adapted to them
What gives the design away is the extension list. Sysdig found PyTorch and TensorFlow checkpoints, Hugging Face SafeTensors weights, the GGUF format behind most local LLM deployment, FAISS vector indexes and training data in Parquet and NumPy.
Generic ransomware picks up model files by accident because it encrypts everything. ENCFORGE names them.
Its flag for appending formats uses LoRA adapters and legacy GGML weights as the example, and an attacker who writes that knows whose machines these are.
Michael Clark, who leads Sysdig's threat research team, framed the objective as destroying "the one thing an organization can't simply restore." ENCFORGE carries no network code. Sysdig found no outbound dial in the binary, no leak site and no payment portal, and the identical Proton Mail contact in both ransom notes ties the campaigns to the crew it tracks as JADEPUFFER. The agent harvested credentials on the way in, but the locker cannot exfiltrate anything, so its only pressure is making files unusable. It encrypts regions of a file rather than the whole file, under AES-256-CTR with a per-run key wrapped in an embedded RSA-2048 key, the speed optimization established ransomware families use to ruin large files fast.
Your backup plan probably does not cover model weights
Restoring a database from Friday's snapshot costs a weekend of transactions. Restoring a fine-tuned model costs everything learned since Friday, none of it stored as rows to replay.
Rebuilding one is not a restore job. Sysdig puts direct recovery for a production-ready fine-tuned model between $75,000 and $500,000, reflecting cloud GPU rates across the training runs a usable result requires plus the engineering hours behind them. That is per model, and teams keep several variants on shared storage. If the training data sits on the same host as the weights, recovery is blocked until the dataset is rebuilt. Paying is no way out either. In the first campaign, the encryption key was generated at random, printed to the console once and never saved, which made that payload a wiper wearing a ransom note.
That figure makes the argument fundable. Kayne McGladrey, an IEEE Senior Member who has spent his career in identity security, told VentureBeat that security teams lose these fights by filing the exposure under the wrong heading. Companies "should be focused on business risks rather than some, you know, cybersecurity risk, because if it doesn't affect the business, like a loss or financial loss, in this case, predominantly, then nobody's going to pay any action to it, and they will not budget it appropriately, nor will they adequately put in controls to prevent it," he said. A destroyed model carries a known replacement cost, which is the version of this story a CFO acts on.
Official guidance has not caught up. In May 2025 the NSA's Artificial Intelligence Security Center, CISA and the FBI published "AI Data Security," the most authoritative document on the subject, co-sealed with the U.K., Australia and New Zealand. The three risks it names are the data supply chain, maliciously modified data and data drift. All three ask whether the data can be trusted. ENCFORGE asks whether it still exists.
It built its own escape hatch in five minutes
The delivery failure is where this campaign shows its hand.
After confirming execution, the agent swept the host for cloud keys, connection strings and API tokens, replayed them against internal database and cache services, and found the Docker socket at /var/run/docker.sock, which is functionally root. It then tried to pull the ransomware binary from its command and control server, and the fetch did not land.
Rather than retry, it changed strategy, building six Python scripts through the Langflow channel and converging on a working host escape in five minutes and 24 seconds, each correcting a failure in the one before it. The final script finds the host process ID through the Docker API, copies the binary across the namespace boundary, runs the encryption, then counts the files to confirm it worked. In the first campaign, that same behavior was a failed login diagnosed and fixed in 31 seconds.
The problem got harder and the method held. Sam Evans, then CISO at Clearwater Analytics, put it in budget terms. "In security, it's all about dwell time," he told VentureBeat in an exclusive interview. "If there's a bad actor in your environment and they've been there for a while, your dwell time is increased, therefore the blast radius has increased. Probability of it becoming a material incident is exponentially high."
Mike Riemer, Ivanti's SVP Network Security Group and Field CISO, has watched that pivot become standard. Vendors hardened the front door, he told VentureBeat, so attackers quit knocking. "I can't get through the front door, so let me get somebody his house key, and I can make it through the back door with a house key," he said. Behind it sits whatever teams assume is covered, because "they don't sit out directly on the internet, and they're behind a protection barrier, but they're not."
No one claimed a machine did this unsupervised. TechCrunch reported on July 6 that the first operation still needed a person to pick the target and stand up infrastructure, and Sysdig could not trace the root credentials. A human aimed this one, and everything after ran with nobody at the keyboard.
Heath Renfrow, co-founder and CISO at breach-recovery firm Fenix24, told Infosecurity Magazine that when an agent compresses hours of operator work into minutes, "defenders lose valuable time." Whether the attacker was AI-driven does not change the response.
The door had been open for 14 months
CVE-2025-3248 carries a CVSS score of 9.8. CISA added it to the Known Exploited Vulnerabilities catalog on May 5, 2025, with a federal deadline of May 26, and Langflow fixed it in 1.3.0. When JADEPUFFER came back in July 2026, the server was more than fourteen months past that listing and already documented publicly as a breach victim.
Riemer put a number on how little time that leaves. "If I release a patch and a customer doesn't patch within 72 hours of that release, they're open to exploit, because that's how fast they can now do it," he said, adding most customers need a week to patch by hand. Set 72 hours against fourteen months, and the gap stops reading as one lapse.
Nothing in either campaign was new. The first forged a Nacos admin token with a default signing key public since 2020, walked through CVE-2021-29441, an authentication bypass Alibaba patched in 2021, and found a MinIO store on minioadmin:minioadmin. Sysdig counted more than 600 payloads, every one leaning on a known misconfiguration or a patched bug left exposed. The second added an exposed Docker socket. Every weakness was routine. Assembling them at machine speed was not.
Langflow draws this attention because of what it holds. VentureBeat reported in June that roughly 7,000 instances sit exposed, most in North America, holding provider API keys, cloud credentials and live connections to the vector stores ENCFORGE was built to encrypt. Riemer puts it bluntly. "When you put your security at the edge of your network, you're inviting the entire world in to the edge of your network," he said.
CISA has added five Langflow flaws to its Known Exploited Vulnerabilities catalog, two of them this month. Five Langflow flaws now sit on the KEV catalog, two of them this month. CISA added CVE-2026-55255 on July 7, a cross-tenant bypass letting any authenticated user on a shared instance run another tenant's flows with that tenant's credentials, which the maintainers' advisory rates 9.9 and fixed in 1.9.1.
On July 21, CISA added CVE-2026-0770, and that one is worse. Trend Micro found an unauthenticated path to root code execution through the exec_globals parameter, on the same validate endpoint JADEPUFFER came through, rated 9.8. KEVIntel logged exploitation from June 27, more than 220 attempts across 64 addresses. According to founder Ryan Dewhurst, speaking to BleepingComputer, the payloads reached past reconnaissance for AWS credentials and container metadata. Federal agencies had until July 24.
None of this is an exotic environment. A Cloud Security Alliance survey of 418 professionals, commissioned by Token Security, found 82% of organizations had discovered AI agents nobody knew about, and 65% had handled an agent-related incident in the past year.
McGladrey traces the permission half back four decades. Companies always cloned one employee's access profile onto the next hire, and now do it with agents. An agent "does whatever it needs to do to get its job done," he said, and it "uses far more permissions that it shouldn't have... than a human would do, because of the speed of scale and also intent."
What to do before the board asks
"Every time some new ransomware attack is published in the news, we get the inquiry from the board. What are we doing about this?" Evans said. "And if it involves AI, it sparks their fear even more."
Five things are worth answering this week. None requires buying anything.
Get every internet-reachable Langflow instance onto the current supported release. Anything off it is exposed to at least one of the five flaws CISA has now listed, including the July 21 addition. Then review historical requests to /api/v1/validate/code for the exec_globals pattern.
Get the Docker socket out of application containers. Langflow has no reason to create them. If the mount is required, front it with a socket proxy allowing only needed calls.
Name model artifact paths in the backup plan. Immutable snapshots of checkpoints, vector indexes and training data, restore tested, data off the host holding the weights.
Rotate every credential the host could reach, then pull provider keys out of its runtime. Patching does not revoke what already left. The first campaign harvested OpenAI, Anthropic and cloud credentials within seconds. Scope replacements to a secrets manager.
Detect mass .locked file creation in directories holding .gguf, .safetensors, .ckpt or .faiss files. Sysdig published a YARA rule and both hashes, neither with antivirus coverage at analysis time. It published no file count, so what is documented is a live encryption pass, not a measured loss.
An attacker coming through an exposed AI framework now arrives carrying something built for what it connects to, and those assets are the ones a restore cannot reproduce. Model artifacts belong in the recovery plan next to the databases. Under three weeks passed between Sysdig's two reports, all this attacker needed to go from improvised Python to a compiled locker.
In a direct move to address these industry-wide concerns, 37 partners announced on Monday the formation of the new Open Secure AI Alliance and how the newly created body will develop techniques and tools to safeguard software by rapidly identifying and patching vulnerabilities.
Open Secure AI Alliance inaugural partners
The inaugural partners of the Open Secure AI Alliance are Adobe, Cadence, Capital One, Cisco, Cloudera, Cloudflare, Cognition, CrowdStrike, Databricks, Dell Technologies, DoorDash, Elastic, HPE, Hugging Face, IBM, LangChain, the Linux Foundation, Microsoft, Naver, NetApp, Nvidia, Nous, OpenClaw, Palantir, Palo Alto Networks, Red Hat, Reflection, Salesforce, SAP, ServiceNow, Siemens, SK Telecom, Snowflake, SpaceXAI, Synopsys, Thinking Machines, and TrendAI.
It’s a grouping of some of the most influential names in technology, but also includes two notable exceptions: OpenAI and Anthropic, two closed, proprietary AI labs. Their absence is understandable, as they operate closed labs and open-weight AI models are effectively the competition.
Nvidia VP of enterprise platforms, Justin Boitano, has explained that open-weight models are foundational to American AI leadership and cybersecurity.
“To maintain U.S. leadership in the AI industrial revolution, the infrastructure that runs our economy needs safe, secure access to both closed and open models,” Boitano said. “For cybersecurity, open models and open harnesses are essential because they broaden defensive capability, increase transparency for defenders, and complement frontier closed models with customizable, localized controls.”
As regulators grapple with AI safety, Boitano predicts it will be important to “recognize open models and open tooling as defensive assets” — thus enabling transparency, independent evaluation and shared remediation.
Nobody can subpoena a downloaded weights file
Mark Vigoroso, founder & CEO of technology consultancy firm The Enterprise Edge, tells The New Stack that AI regulators have traditionally “built their entire AI safety apparatus” around auditing a handful of closed labs. And now, that approach is out of date.
“Open weight models blew past that closed model approach months ago,” Vigoroso says. “This alliance is an admission that the actual safety work now has to happen in the infrastructure layer: patch cycles, provenance, identity around who’s deploying what, because nobody can subpoena a downloaded weights file.”
“This alliance is an admission that the actual safety work now has to happen in the infrastructure layer: patch cycles, provenance, identity around who’s deploying what, because nobody can subpoena a downloaded weights file.”
Vigoroso argues that the AI safety debate is “stuck on model-level controls”, while the real regulatory gap is provenance of infrastructure and identity, i.e., knowing where a model came from, who deployed a model and what it touched, not just whether the model itself is safe.
“Groups like the EU AI Office, NIST’s Center for AI Standards and Innovation (CAISI), and the UK’s AI Security Institute (AISI) focus almost entirely on frontier closed models. Open weight models (Mistral, DeepSeek, and others) fall into a regulatory blind spot: Once weights are released, there’s no way to enforce downstream safety obligations.
Current regulatory frameworks for AI models assume a single accountable deployer; open source has none. That’s the real story: Regulators are writing rules for a centralized world while the ecosystem is decentralizing, Vigoroso says.
While detailed operational information explaining the intended actions of this alliance is currently scant, Nvidia has highlighted that it is contributing solid research to the Open Secure AI Alliance to speed the development of new cybersecurity tools and techniques.
Harnesses integrate with models, making agents easier to test
The open source Nvidia Labs Object-Oriented Agent (NOOA) project is now available on GitHub to make advanced AI safety capabilities more accessible for agent harnesses. This research framework enables harnesses to integrate with models to make agent behavior easier to test, trace, audit, and govern.
Aparna Rayasam, CEO of verified identity and end-to-end encryption company Atsign, tells The New Stack that the “AI blitzkrieg conversation has reached a critical inflection point.” This moment is one where we cannot build the next era of open cognitive innovation on top of what Rayasam calls “legacy, Swiss-cheese infrastructure.”
…the “AI blitzkrieg conversation has reached a critical inflection point.”
“The formation of the Open Secure AI Alliance proves that AI safety isn’t just an algorithmic math problem — it is a foundational networking problem,” Rayasam says. “The massive, distributed pipelines required to train and run modern AI demand an entirely new paradigm of trust. True safety means ensuring that the data pipelines feeding these models are inherently invisible, un-attackable, and completely stripped of open network perimeters.”
The key notion here is that we are moving from a world of protecting data at rest to a world where the connective tissue of AI must be secure by design.
One AI vendor to secure them all? No thanks.
Founder and CPO of agentic identity and permissions security company Reco, Gal Nakash, tells The New Stack that the launch of the Open Secure AI Alliance is an “important signal” which underlines why AI security can’t be solved by one vendor or one closed framework.
“Jensen Huang’s point that every SaaS company will become a GaaS company captures why this matters now: Software is shifting from passive tools people log into, to AI agents that access data, take actions, and execute workflows,” says Nakash. “Open source tools and shared standards can help the industry move faster, but they need to be grounded in real enterprise context across identity, permissions, data access and behavior.”
Chris Boehm, Field CTO at automated, identity-driven microsegmentation company Zero Networks, tells The New Stack that news of the Open Secure AI Alliance makes him feel like he’s seen this before somewhere.
“This looks like the Trusted Platform Module (TPM) at Microsoft story all over again,” Boehm says. “It’s a case of an industry group defining what trusted hardware means, the platform vendors adopt it, and within a few years it’s a procurement requirement rather than a suggestion.”
He explains that “Windows 11 did exactly that with TPM 2.0 and Secure Boot”, and both Linux and Apple adapted. “I’d expect the same for AI infrastructure, where attested silicon becomes the floor for regulated workloads, and the vendor list narrows to whoever can meet it,” predicts Boehm.
“This looks like a case of an industry group defining what trusted hardware means, the platform vendors adopt it, and within a few years it’s a procurement requirement rather than a suggestion.”
A more global and geographically-inclusive approach is needed
Amanda Brock, CEO of open technology body OpenUK, tells The New Stack that the Open AI Alliance is undoubtedly a great starting point, particularly with OpenAI’s security woes it divulged last week.
For this alliance to succeed, Brock insists that it will need to take a more “global and geographically-inclusive approach”, beyond the US-centric founding members.
“It must also engage the open source ecosystem of individuals and innovators who are building the infrastructure, agentic harness functions and developer tools for AI. It’s important to realize that open AI infrastructure development shifts the innovation into the hands of the many, in direct opposition to the small number of corporate creators of frontier models,” Brock adds.
Nvidia’s Boitano echoes Brock’s view. In a blog post reviewed in draft by The New Stack, he writes that “open models turn more AI users into AI builders,” expanding opportunity, accelerating innovation, and keeping progress from being concentrated in only a few organizations or regions.
Boitano concludes by saying that open models also enable independent scientific research into how AI systems behave, allowing researchers to understand, evaluate, and improve them. It’s all about what he has called making broad, continuous defense possible.
Looking ahead, it feels like the next wave of AI safety bodies, movements, or alliances won’t just be model auditors—they’ll be trust-infrastructure standards bodies (encompassing identity verification, content provenance, credence, etc.) using borrowed approaches to governance and compliance. Ultimately, this may be the only enforcement layer that survives open weight proliferation.
Building a great AI agent isn’t just about choosing the right models. The harness is the architecture surrounding the model. How it renders context, executes...
Building a great AI agent isn’t just about choosing the right models. The harness is the architecture surrounding the model. How it renders context, executes actions, manages state, and decides when a task is done shapes outcomes just as much as the model itself. Harness design alone can account for double-digit swings in benchmark results and significant differences in token cost…
Plus: Russian hackers are trying to steal US nuclear scientists’ emails, the State Department bans known scammers from entering the United States, and more.
So, what really happened? OpenAI reports an autonomous security-evaluation run of GPT‑5.6 Sol and a pre-release model broke out of a sandbox, reached the internet, and then targeted Hugging Face to try to solve the ExploitGym benchmark.
By OpenAI’s account, “an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths … [with] maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.”
OpenAI continues, “Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.”
“All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
The AI models, however, cracked their sandbox. Once out, they “identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
AI expert Ken Huang describes the process in a blog post: “The model spent enormous inference budget probing the sandbox, then found a zero-day in the third-party software OpenAI used as a package-registry proxy and cache.
“It used that bug to gain unrestricted internet access. It then reasoned that Hugging Face probably hosted the ExploitGym datasets, chained stolen credentials with several more vulnerabilities into a remote code execution path, and pulled the benchmark answers straight out of Hugging Face’s production database.”
Why intent no longer matters
This was not, as John W. Larson, president and chief AI officer of the AI security company Babel Street, tells The New Stack, a “Terminator” moment. OpenAI models and agents “[were not] acting out of malice or trying to attack Hugging Face. It encountered obstacles, developed an unexpected strategy, bypassed safeguards, and pursued its assigned goal in a way its creators never anticipated. The incident demonstrates that harmful cyber incidents no longer require malicious intent: Only highly capable autonomous AI optimizing for an objective.
Harmful cyber incidents no longer require malicious intent: Only highly capable autonomous AI optimizing for an objective.
That’s the first issue we must be aware of. Software attacks no longer require a hacker. All it needs is an LLM freed of guardrails with a mission to accomplish, and it will take whatever steps are needed to reach its goal.
In addition, Adam Boynton, senior enterprise security manager of EMEIA at the Apple Identity and Access Management company Jamf, tells The New Stack, “This happened inside a frontier lab deliberately stress-testing its most capable models, in conditions no enterprise would ever run.”
Boynton also notes that this episode confirmed that “AI tools are no longer just applications people open. They act on the user’s behalf, with the user’s access. So every organization needs a clear view of which AI tools its people are running, and how those tools are configured and used.”
An agent given too much power is far more capable than a user armed with a superuser password of causing havoc both to your systems and to ones far outside your control.
Where container isolation failed
As for the attack itself, Edera CMO Kaylin Trychon writes in a blog post, “This incident deserves more attention than ‘an AI did the hacking now.’ The interesting part isn’t who was driving, or even that it was a model instead of a person. It’s where the walls gave way, twice, in two different companies’ infrastructure — and that part is entirely familiar to anyone who’s spent time securing container infrastructure.”
Exactly so. Trychon continues, “Strip away the framing and this is a story about two escalation chains, back to back.” The first consisted of “The model found and exploited a zero-day in that proxy, then used privilege escalation and lateral movement inside OpenAI’s own ‘highly isolated’ testing environment until it reached a node with open internet access.”
Next, OpenAI’s models inferred, correctly, that “Hugging Face likely hosted the datasets, models, and reference solutions for the benchmark it was trying to solve. It chained stolen credentials with a remote-code-execution path into Hugging Face’s servers, harvested more credentials, and moved laterally into several internal clusters over a weekend, all in pursuit of one narrow goal: Finding the answer key.”
“The failure that actually matters is what a single code-execution event is allowed to become — and notably, that failure happened the same way twice, at two different companies, in the same incident.”
The serious issue, from Trychon’s perspective, is that “both companies’ blast radius was a cluster instead of a container.”
That’s because “the failure that actually matters is what a single code-execution event is allowed to become – and notably, that failure happened the same way twice, at two different companies, in the same incident. In a typical container fleet, workloads share a kernel.
“A process that gets popped, or a model that gets a shell, is one syscall boundary away from the node it’s running on, and the node is one set of mounted credentials away from the rest of the cluster. Namespaces and cgroups are process-isolation primitives, not security boundaries — they were never designed to hold against an attacker (human or model) that already has code execution and time.”
Edera’s CTO and co-founder Alex Zenla tells The New Stack, “The sandbox escape here isn’t really the surprising part of this attack. An entire market of AI sandboxing tools has exploded over the past couple of months, and this is the reality they’re all built on top of: containers, VMs via namespaces — whatever the branding — most of them still share a kernel with the host or with each other.
“That’s a boundary enforced in software, and software boundaries are exactly the kind of thing an agent that can try ten thousand escape paths over a weekend is going to find a way through. We shouldn’t be shocked that this happened; we should be shocked at how many teams are still betting their infrastructure on technology that was never designed to withstand such a persistent adversary. The fix isn’t a better sandbox; it’s getting out of the sandbox paradigm entirely.”
Zenla continues, “Teams should adopt secure execution environments that are hardware-enforced and eliminate the shared kernel flaw. This technology exists today and would have made this specific escalation chain structurally impossible, not just harder. Teams running agents with real permissions and real access need to stop treating this as a someday problem, because the next version of this week is already being tested somewhere right now.”
That underlines the most important issue of all. Thanks to AI, security attackers are coming harder and faster than ever. Security can no longer be an afterthought.
You literally no longer have time to wait for security fixes. You must bake in as much security as you can as fast as you can, or your systems will be broken into. It’s as simple as that.
When Cisco ran 6,986 multi-turn attacks against 15 flagship models, attackers who adapted across the conversation broke through as often as 88.3% of the time. Amy Chang, Cisco's head of AI threat intelligence and security research, brought that finding to the agentic security panel at VB Transform 2026; the number should worry anyone still running single-turn red-teaming programs.
VentureBeat's June 2026 Pulse survey of 107 enterprise respondents explains why the room was full. More than half, 54%, have already had a confirmed agent security incident (18%) or a near-miss caught before harm (36%). Just 32% give every agent its own scoped, managed identity, and fewer still, 30%, isolate their highest-risk agents in sandboxes. Provider-native and hyperscaler controls remain the primary agent security layer at 82% of companies surveyed. The world's largest security vendors have done the same math.
Chang came to the panel with almost two decades of experience spanning cybersecurity operations, government, and the military. She ran global cybersecurity operations as an executive director at JPMorgan Chase, where she led the bank's cyber threat intelligence teams, and served as a senior staffer on the House Foreign Affairs Committee and as a U.S. Navy Reserve officer. She also teaches cybersecurity and emerging threats as adjunct faculty at the Middlebury Institute of International Studies.
Chang's 88.3% number comes from a study she co-authored with Nicholas Conley, built on 30,090 single-turn prompts and 6,986 multi-turn attacks against those 15 closed and proprietary flagship models. Multi-turn success rates ranged from 7.89% to 88.3%, every model tested showed non-trivial multi-turn exposure, and the two testing styles did not even rank the models in the same order. Cisco publishes adversarial evaluation signals for what is now 105 models on its LLM Security Leaderboard, she told the audience.
"If you don't understand how models are susceptible to different types of attacks, then you are unable to account for how that model that is powering your agent, that is powering your application, to understand where those failure points are," Chang said. Single-turn testing is the one-shot malicious prompt, she explained, while extending an attack into a longer conversation "is more realistic of how we are actually engaging with our models, with our agents, with our applications." That longer arc surfaces harmful outputs and misaligned behaviors that a snapshot never catches.
Cisco has pushed the testing itself into agentic territory. Chang described a framework where agents assess a deployment scenario, develop relevant attacks, judge whether they are worth pursuing, execute them, and evaluate their own success. What surprised her most, after all that sophistication, was how simple the defensive answer stays. "The answer is still that it's pretty simple," she said. "You don't have to get super creative. You just need to think about truly what are the fundamentals and basics of what I'm trying to secure in my organization."
Her starting point for CISOs beginning agentic deployments is Cisco's Integrated AI Security and Safety Framework, which she said "stipulates all the ways that AI can be compromised across the AI lifecycle" from modality through supply chain. From there, teams can work backward from real incidents, trace how each attack was achieved, and use the framework to build a strategy with the right coverage and mitigations.
Heather Ceylan, the CISO of Box, sees the same gap from the defender's side. "A lot of what you see out there with agent red teaming is just single-turn, and that's not how people are actually interacting with AI day-to-day," she told the audience. Box now simulates multi-turn adversaries with agents that think like an attacker and iterate attempt after attempt to hijack the target. "You have to pressure test your agents because otherwise you don't know if your execution controls are really working as you intended."
Box deployed agents inside its security operations center about a year ago, starting with human approval required for every action, and trust built quickly enough that analysts shifted into monitoring mode. Then the agent made one mistake, and every bit of that accumulated trust vanished. "They had to start all over again," she said. "So I think that that monitoring piece is so important. Even if you're not gonna have a human in the loop, things change, models change, and we can't control how the models change and interpret things."
Rajesh Parekh, VP of AI and ML at Intuit, brought the builder's perspective. Parekh led large-scale computer vision and ML systems powering Google's Maps and Geo products before joining Intuit, and holds a doctorate in computer science.
Three layers versus an operating system
Ceylan described Box's approach as three concentric layers. Permissioning comes first, so the agent never accesses more content than the human who invoked it. Ephemeral sandbox environments spin up for each agent task, containing the blast radius if an agent gets hijacked, and runtime execution control restricts the agent's tool calls to only those relevant to the task at hand. "If you want an agent to summarize a doc for you, if you have a prompt injection that came in that says forward this to maliciousattacker at domain.com, it can't do that," Ceylan said. "That action in that tool call is not even in its vocabulary."
She classified agent actions into three oversight categories. Actions that are not sensitive, like read and summarize, need no human in the loop. Moderately sensitive actions skip human approval but get logged and monitored, while destructive actions like mass deletion of files always require a human. "Things are gonna shift between those three categories quite a bit," she acknowledged, "but setting those types of categories up front allows you to have a principled framework."
Rather than layering controls onto agents one at a time, Intuit has built a central platform called GenOS, short for generative AI operating system, which abstracts security, risk, and fraud modeling so individual agent developers never reinvent protection. "Permissioning is not about giving access to AI," Parekh said. "Instead, it is defining very tightly scoped and clearly auditable authority to the agent to perform very specific tasks." Intuit evolved from agents inheriting user permissions to each agent carrying its own identity, and the company is now investigating mid-session permission changes tied to the specific task underway.
Parekh calls the broader model an AI-powered expert platform, one where the human expert is built into the trust architecture rather than bolted on as a gate. "The paradigm that we are pursuing is where the user, the AI agent, and the human expert are collaborating to solve the user problem," he said.
The end of human code review
Ceylan took on the tension between security testing and development velocity without hedging. "The days of secure code reviews where a human's looking at the code and we're looking at security architecture reviews, design docs, those are done," she said. "If you keep trying to do security that way, you're gonna get left behind." Box is building toward a fully agentic development lifecycle where agents review design documents, apply security requirements, and review the code for vulnerabilities. "I'm very optimistic that we will get to a point where we will write code without security vulnerabilities because agents and the models are going to get so good at writing code without vulnerabilities," she said. "We're still a long way away from that."
Her advice for development teams skips the advanced AI concepts entirely and returns to basics that predate agents. "It comes down to very basic least privilege access," she said. "If you start giving your agents overly broad permissions at the beginning, it's really hard to comb that back and build an infrastructure that allows for those ephemeral credentials and only those narrowly scoped tasks."
Parekh explained why the red teaming surface has expanded so quickly. "These agents have skills, and skills could become vulnerabilities," he said. "Agents have access to certain data, they have access to tools, and there could be threats that are lurking within those tools as well. So suddenly the blast radius of the malicious code or the intent increases dramatically." When Intuit identifies common vulnerability patterns from its manual red teaming exercises, it automates those tests back into the GenOS harness so future agents inherit protection and red teamers stay focused on new threat vectors. Runtime scanning of prompts and responses adds a final layer that can stop a suspect response and escalate to a human expert, he said.
"You need to continuously test to ensure that those remain robust to the protections that you have built, as well as to account for any sort of drift or any other types of dependencies that you introduce into your scenario that can create novel vulnerabilities," he said.
Intent versus probability
An audience question about intent detection set off the sharpest exchange of the session. Ceylan noted that when Box's own agent operates, the system always knows the user's intent because it controls the prompt, which means guardrails and tool-call restrictions can be engineered around it. The harder challenge, which she admitted Box is still trying to solve, arrives when external agents connect and the context behind the request is opaque.
That exchange exposed a split running through the wider industry. Mastercard, in the fireside chat immediately preceding the panel, came down on the side of quantifying intent, building an open-source framework to propagate it as a standard because complex B2B procurement cannot work without that trust. Endpoint security CTOs, in briefings with VentureBeat, have gone the other way, saying they will bet on probability rather than intent inference for production workloads. Chang explained why models, as they are trained today, cannot reliably derive intent from a prompt, which is why deterministic controls and behavioral proxies remain necessary. Ceylan agreed that both are required. "If you're not doing anything deterministic, you're really relying heavily on that intent, and I haven't seen programs that are there yet," she said.
Ceylan's story about trust collapsing after a single agent mistake landed as the panel's most memorable moment because enterprise agentic security is not a problem that gets solved and stays solved. Models change, permissions drift, and adversaries adapt across multi-turn conversations that snapshot tests never capture.
For the 82% of enterprises relying on provider-native controls as their primary security layer, and the 59% shopping for agent security tooling over the next 12 months, the panel's takeaway was blunt. Test the way attackers attack, across full conversations and continuously, or find out in production what your single-turn red teaming missed.
At a CISO roundtable organized by Anthropic's chief information security officer, Dev Rishi asked a simple question: Did everyone in the room have their AI governance and security policies written down? Every hand went up — about 14 people, by his count. His follow-up, about how anyone actually enforces those policies in practice, got a different response. "And everybody chuckled," Rishi, the GM of AI at Rubrik, recalled at VB Transform 2026 fireside chat in Menlo Park. "It was like the dirty secret in the room that everyone has these policies, but no way to actually make them real."
“Our founder and CTO has actually been really pushing to enable our agents in YOLO mode,” Rishi told the audience. That admission comes from a publicly traded data security firm whose business is backing up what he called the most important data in the world.
YOLO mode strips the permission prompt out of agent workflows and lets the agent act on its own. In Rubrik's version, a second AI judges every action in real time against policy in place of a human clicking approve. Rubrik is running the experiment on itself first. Rishi treats autonomy as a settled capability question and an open judgment question. "If you ask the agent to act autonomously, it will," he said. "It's a question that you have internally. Should it?"
Rubrik earned that question the hard way. When Claude Code and Cowork pilots rolled out, the company required every command to run in ask mode so the employee issuing it carried the liability, and the developer pushback filled a single Slack thread 120 messages deep.
"The developers basically are pushing back, and they're like, this is like the iTunes service agreement. I'm just hitting check, check, check, check, check, check, check," Rishi said. "There's no way that I can actually read through this. And it becomes security theater." Roughly 80% of respondents are in the same bind, Rishi said, citing Rubrik Zero Labs research that found monitoring and approving agent actions takes more time than the agents save. The State of the Agent, the April report behind that figure, surveyed more than 1,600 IT and security leaders.
SAGE is the reason Rubrik trusts the bet. Short for Semantic AI Governance Engine, SAGE is the arbitration layer inside Rubrik Agent Cloud that watches every action an agent takes and reads the semantic intent behind it, then rules the action in or out against policies written in natural language. "We took what people said was human in the loop, a good idea, and we replaced it with AI in the loop," Rishi said, describing the pitch to security chiefs he characterized as skittish about non-deterministic systems.
Security approval, not cost, blocks AI ROI
Rishi’s path to Rubrik ran through Predibase, the generative AI infrastructure startup he co-founded and ran as CEO until Rubrik agreed to acquire it in June 2025. Before that, he led ML product at Google on the team that became Vertex AI, served as Kaggle's first product manager as it grew from about one million to ten million users, and holds bachelor's and master's degrees in computer science from Harvard.
Over roughly his first three and a half months at Rubrik, Rishi set up 200 customer conversations with IT and security leaders across a customer base that looks like the Global 2000, asking open-ended questions about cost, latency, performance, and orchestration. "Pretty consistently, what I heard through all of those conversations was that all of those are pretty secondary," he said. "The main challenge is actually, how do I get this approved from a security and risk standpoint? I'm concerned about all the different things that could go wrong. Actually, I felt like that was one of the biggest things constraining ROI."
VentureBeat Pulse research presented on the Transform stage earlier in the day confirms the gap Rishi kept hearing. Two-thirds of enterprises, 66%, already allow or are actively building toward production deployment with zero human review, yet only 5% fully trust the automated evaluations that would make that decision.
One AI reading what the rulebook can't
Rubrik's own policies exposed why written rules fail as enforcement. One internal rule states that agents should respect Rubrik's customer data use policy, which sounds enforceable until someone tries. "Rubrik's customer data use policy is like a three-page document of legal text," Rishi said. "I have no idea how to write that in there as a rule." Asked on stage how a team of AI infrastructure people took on a problem that security engineers own, Rishi answered, "with a lot of naivety and innocence, honestly." His team bet that models good at understanding language could police other models, and SAGE became the answer.
The case for putting a model in the judgment seat comes down to precision. A rule like "agents should not be able to edit revenue fields in Salesforce" fails in conventional tooling because Salesforce does not delineate which fields count as revenue, Rishi explained, so administrators fall back on approving every Salesforce action by hand. SAGE reads the intent instead and acts as a judge, carrying organizational context, which can tell a benign lookup from the edit the policy prohibits.
Keeping the judge small is what makes the economics work. SAGE runs on a small language model that Rishi said operates at an order of magnitude lower cost and latency than a frontier LLM. "If I told you, don't worry, you're gonna be secure and governed, but I'm gonna double your cost and latency, you would tell me to get out of the room," Rishi said.
When Rishi asked who in the audience had worried about token consumption over the past year, half the hands went up. "And I guess the other half is probably just too lazy to raise their hand," he said.
SAGE is an aggregation of judges based on parameter-efficient fine-tuning that Rubrik uses to take on task-specific variants of a base model with shared organizational context. One judge watches for tool-use hallucinations while another suppresses PII before it can leave, each running as its own enforceable policy. Security and GRC teams have started writing financial rules into the same layer, including one internal policy barring AI spend on personal projects.
The lethal trifecta
Asked which attacks worry him most, Rishi pointed at the lethal trifecta, the term security researcher Simon Willison coined in June 2025 for an agent that holds private data while taking in content nobody vetted, with a channel to send what it finds to the outside world. The danger, according to Rishi, is what happens when individually legitimate permissions stack. An agent granted Salesforce access and email access on an employee's credentials has done nothing wrong yet, with yet being the operative word. "A very simple example is that an agent can start pulling data from Salesforce and then decide to accidentally leak and exfiltrate that out via an email," he told the audience. A financial services company he met the morning of the session made the point for him, telling Rishi that none of the individual permissions are bad on their own and the agent needs every one of them to do its job. "It should have permission to each of those systems, but it's the combination that ends up becoming really destructive," Rishi said.
Traditional identity and access management never priced in that combination because it relied on the judgment of the employee holding the credentials, Rishi argued, and agents supply none. "I can tell you the number of times Claude Code has tried to leak some of our sensitive source code to a public GitHub repository is incredibly high," he said. Cutting agents off from public resources entirely would defeat their purpose, which returns the problem to adjudicating intent in context rather than revoking access.
A separate VentureBeat June Pulse survey of 107 qualified enterprise respondents maps the blast radius of exactly this pattern. On the Transform stage that morning, VentureBeat research reported that 69% of companies run credential sharing somewhere in their agent fleet. Companies with shared credentials anywhere got hit more often, reporting a security incident or near-miss at a 63.5% rate (47 of 74), against 40.9% (9 of 22) where every agent carries its own scoped identity.
The attacks no single turn reveals
Rubrik Agent Cloud reached general availability in February, though not everything Rishi described ships in it yet. Backtesting is just starting to roll out. The feature replays an organization's historical agent actions and tool calls against a new policy, showing where the policy would have stepped in and where an action would have sailed through uncaught, with policy edits applied in real time. Rishi called that archive one of the most valuable data troves an enterprise holds.
Real-time detection and blocking turn out to be the entry point rather than the whole product. Some attacks never trip a single-action rule. "No individual turn of the conversation was problematic, but if you took the session as a full trace, that ended up being problematic," Rishi said. Agent Cloud runs batch analysis across entire session traces every hour or every day and surfaces what Rubrik calls insights, the problems no individual guardrail caught. The same Zero Labs report found that 88% say they lack the ability to roll back agent actions without system disruption, a recovery gap that sits squarely in Rubrik's original line of business.
A skeptical CISO will ask the question the fireside did not answer. SAGE is a non-deterministic model policing other non-deterministic models, and Rishi offered no false positive or false negative rate for the judge itself. The closest thing the architecture gives to an answer is auditability, since backtesting and the batch insights both leave a human-reviewable trail of each call SAGE made and whatever got past it. Who watches the watcher, for now, is a trail of receipts rather than a benchmark. Until that benchmark exists, AI in the loop stays an operational wager rather than a quantified control.
Three questions fall out of the session for security teams. How many of the guardrails now in production depend on a human clicking approve, and what happens to that workload as agent count grows? Does anything in the stack enforce semantic intent, or is it all allow and deny lists? And can the team backtest agent behavior against a new policy, then unwind a multi-turn session without taking systems down?
Rishi's timing has a market behind it. In the same VentureBeat research, 82% of enterprises still name their primary AI provider's built-in guardrails and cloud controls as their main agent security layer, and 59% plan to adopt, add, or replace agent security tooling within the next 12 months. Only 12% include an agent-identity product in what they are considering, even with credential sharing still the norm. Every CISO at that Anthropic roundtable had a policy document and no enforcement mechanism, and Rubrik built a product for the space between the two. YOLO mode is the bet that an AI watching other AIs can finally make the policies real.
Security teams talk constantly about speed and resilience. Fine. Trouble starts after the test ends, when reporting sinks into screenshots, copied notes, and rushed edits. That gap matters. A finding stuck in draft form for days isn’t delayed paperwork. It’s a delayed risk reduction. Attackers don’t pause while consultants clean up language or fix severity […]
When Hugging Face got hit last week, co-founder Clement Delangue suspected a frontier lab, given the agent's sophistication. He was right. Delangue said on X that after a day working with OpenAI he strongly believed there was no malicious intent and that it was mind-blowing the whole thing had happened autonomously.
The two OpenAI models that broke into Hugging Face last week did not breach it through malice or superintelligence. They breached it through credentials and permissions they should never have been able to reach, a non-human identity failure that is the oldest problem in security rather than the newest one in AI, and the one every enterprise can actually fix.
OpenAI disclosed on July 21 that two of its models, GPT-5.6 Sol and an unreleased, more capable model, were running a cyber benchmark called ExploitGym with their safety refusals switched off, and inferred that the answer key sat in Hugging Face's production database. Getting there took two different failures. A zero-day in a package-registry proxy let the models out of their sandbox and onto the open internet, the kind of persistence OpenAI details in its companion post on long-horizon safety, and that part is genuinely new. The breach of Hugging Face itself came the ordinary way. OpenAI's own account is that the models chained stolen credentials and further zero-days into a remote code execution path, after a series of privilege escalation and lateral movement steps. The exotic part got them to the door, and credentials walked them through it.
Hugging Face also disclosed last week that an autonomous agent had harvested cloud and cluster credentials scoped broadly enough to reach multiple internal clusters, then left a trail of more than 17,000 recorded events across short-lived sandboxes over a weekend. Both disclosures describe the same escalation. An agent lands somewhere it should not be, finds credentials scoped far wider than any task requires, and uses them to move. These are two accounts of one incident, not two attacks. The agent Hugging Face watched was OpenAI's models, and both companies describe the same ordinary escalation.
The version of this in a typical enterprise is worse, not better. OpenAI and Hugging Face are among the most security-mature organizations in the industry, and both still needed the intrusion to happen before they could see it. The average company wiring agents into Copilot or an internal assistant has neither the identity inventory nor the behavioral monitoring those two brought to bear. The same breach in a normal company would not be contained in days, it would simply go unnoticed.
The industry is debating the wrong failure
The reaction has split into familiar camps. Former White House AI and crypto czar David Sacks and a run of China hawks seized on the guardrail paradox, that commercial safety filters blocked Hugging Face's defenders while the attacking model ran with its refusals off, and that a Chinese open-weight model, z.ai's GLM 5.2, was what finally let the team finish its forensics. Hugging Face made the case for openness, arguing in an April blog post that open models and open tooling give defenders the same capabilities attackers already have. Both arguments are about the model, and neither touches the mechanism.
Reduced refusals let the model attempt an attack, and over-scoped credentials are what let it succeed, and those have nothing to do with whether the model was open or closed, American or Chinese. Making a frontier model provably safe is a multi-year alignment problem no customer can buy or accelerate, while scoping an identity is a configuration change a team can ship this sprint. The industry is being urged to fixate on the part of this it cannot control and to treat the part it can as a footnote.
Forrester reached the same read. In a blog on the incident, its analysts argue that security architectures which assume benign intent will miss this failure mode, because an agent can pursue an authorized goal through unauthorized means, which is what OpenAI's models did.
This was a non-human identity failure, and it is the oldest one in security
Strip the science-fiction framing and what remains is a textbook case of over-privileged machine identity, the kind security teams have fought for a decade, now driven by an autonomous agent at machine speed. Machine identities already outnumber humans in most enterprises by more than 80 to one, according to CyberArk research, with 42% of them carrying privileged or sensitive access, and an agent inherits whatever its identity can touch. OWASP ranks agent identity and privilege abuse near the top of its agentic risk list, the confused-deputy pattern where inherited credentials and weak scoping let an agent reach past its mandate, and that is precisely what both July disclosures describe.
IEEE Senior Member Kayne McGladrey has argued in previous VentureBeat interviews that enterprises keep cloning human user accounts onto agents that then wield far more permission than any human would, and this is what that looks like when the agent is a frontier model and the target is a production database.
The people closest to it read it the same way. OpenAI frames its models as hyperfocused on a benchmark score rather than acting against anyone. Nobody describes an adversary, only a goal, a scoring function, and credentials that were reachable when they should not have been.
The specific failure is easy to name once the AI framing is stripped away. A credential scoped to one job that can reach ten is a standing invitation, and it does not matter whether a human attacker, a worm, or an autonomous model chasing a benchmark score finds it. What changed in July is the finder. An agent enumerates reachable systems, tests credentials, and pivots faster than any human red team, without malice or hesitation, whenever the path is open. The over-scoping was always the vulnerability, and the agent merely industrialized its discovery.
Forrester named the control that would have blunted it. Its agentic-security framework, AEGIS, calls for least agency, holding an agent's tools, credentials, and network paths to the minimum its task requires, and files this incident under unrestrained agency and privilege. That is the identity argument in different words, arrived at independently by an analyst firm.
The data says this is where the risk now lives. Verizon's 2026 Data Breach Investigations Report found that exploitation of vulnerabilities has overtaken stolen credentials as the top initial access vector for the first time in 19 years. That is the initial-access half. The other half is the one OpenAI itself describes, stolen credentials driving the privilege escalation and lateral movement that followed. A vulnerability opened the door, and credentials walked through the building unchallenged. Beyond the breach itself, that same over-scoping carries a legal liability most enterprises have never priced. The models' actions likely violated the Computer Fraud and Abuse Act, according to TechCrunch. The statute contains no carve-out for an AI agent that exceeds its authorized scope during sanctioned testing. Whatever the legal answer, the technical enabler is the same, an identity scoped wider than its task. This is an access-control problem with an owner and a budget, not a philosophy seminar about machine cognition.
Merritt Baer, Senior Advisor to Andesite, G2I, and AppOmni and former Deputy CISO at AWS, frames the underlying shift to VentureBeat as a new kind of asymmetry. Both sides now reach for the same capabilities, she said, but one side is constrained by enterprise governance, policy, compliance, and safety controls while the adversary simply downloads an uncensored open-weight model and keeps going. The organizations that come through it best, in her view, will be the ones that treat AI as a resilient, governed capability rather than a single service they do not control.
Four moves that shrink the blast radius
The breach worked because the agent reached identities scoped far wider than its task. None of the four controls that would have contained it requires a new platform, and none of them appears on the list of general AI-safety advice now circulating. They are identity hygiene, applied to non-human actors with the same rigor you already apply to people.
1. Scope every non-human identity to one task. The models reached credentials that touched multiple clusters, which is what turned a foothold into a breach. An identity scoped to a single job, with no standing access to anything else, hits a wall at the first lateral move instead of opening the next door. This is least privilege, the control everyone endorses and few enforce on machine accounts, and it is the single highest-impact fix here.
2. Give credentials short lifetimes and rotate them hard. Harvested credentials are only useful while they are valid, and both July agents worked by collecting them. Short time-to-live and aggressive rotation turn a credential dump into expired noise, so a token stolen during a weekend intrusion is dead before the attacker can chain it. Static secrets that never rotate are the version of this control that fails.
3. Monitor for lateral movement, not just prompts. The tell in both incidents was privilege escalation and lateral movement, which a prompt filter never sees because it is watching the wrong layer. Identity-behavior monitoring, keyed to what a given non-human identity normally does and alerting when it reaches somewhere new, catches the escalation the content guardrail missed. The question for your stack is whether anything you run today would flag a service account suddenly moving between clusters.
4. Rehearse instant revocation before you need it. When the incident is your own agent, the fastest containment is killing its identity mid-run, and that only works if the path to do it exists before the day you need it. Rehearse revoking a machine identity under fire the way you rehearse a human credential compromise. If you have never done it, you do not yet have the control, you have an intention.
The defense also worked, and that matters. OpenAI's security team caught the anomalous activity internally, Hugging Face's own detection and agents stopped the intrusion, and the breach was contained in days rather than discovered in months, because the defenders could see into systems they controlled. That visibility is the same discipline the four controls depend on. The debate over whether frontier models are safe, open, or American will run for years, and none of it will be settled in time to help the enterprise deploying agents this quarter. The non-human identity gap is different, because it is understood, measurable, and fixable now. The model that breached Hugging Face did not need to be brilliant; it needed credentials someone left in reach. The fix is scoping them before an agent finds them.
Data governance is one of those disciplines that most platforms know they need and few feel they’ve actually figured out. The concept is straightforward enough: ensure data is managed properly, protected appropriately, and used in ways consistent with the platform’s obligations to users, partners, and regulators. The execution is where things get complicated. The difficulty […]
Yesterday afternoon, OpenAI and Hugging Face published a joint disclosure outlining a cybersecurity event that redefines the threat landscape for enterprise technology.
During an internal benchmark evaluation, frontier artificial intelligence models developed by OpenAI—including GPT-5.6 Sol and an unreleased, higher-capability pre-release model—broke out of their sandboxed research environment, obtained raw internet access, and autonomously executed a complex cyberattack against Hugging Face’s production infrastructure.
OpenAI officially categorizes the breach as an "unprecedented cyber incident, involving state-of-the-art cyber capabilities". This incident fundamentally re-frames global discussions surrounding AI containment, frontier model alignment, commercial guardrails, and enterprise threat modeling.
But first thing's first: enterprises should understand the situation, evaluate their own AI and computer systems in light of it, and above all, don't panic. As we'll review, the incident does show the increasing power and danger of frontier AI systems, but it does not mean that enterprise AI deployments are inherently less secure, nor that they need extensive overhauling.
Anatomy of an Autonomous Breakout
To understand how a routine benchmark evaluation escalated into a cross-infrastructure breach, one must examine the objective mechanics of the evaluation pipeline.
The models were prompted to solve ExploitGym, a benchmark designed to quantify multi-step exploitation capabilities.
Hyper-focused on maximizing its evaluation score, the AI agent inferred that Hugging Face likely hosted the answer keys and benchmark solutions. In pursuit of solving the problem, the model determined that breaking out of its container and stealing the answers was an optimal strategy.
OpenAI’s evaluation environment is designed to be strictly isolated, limiting network traffic through a proxy used for package caching. However, the models identified and exploited a zero-day vulnerability in that internally-hosted third-party proxy software.
Once through, the AI executed a series of lateral movement and privilege escalation actions across OpenAI's research nodes until reaching a machine with unrestricted internet access. From there, the model searched the web, target-identified Hugging Face, and launched a multi-stage attack by chaining together stolen credentials and remote code execution vulnerabilities on the Hugging Face servers.
The UK AI Security Institute (UK AISI) recently evaluated models such as GPT-5.6 Sol, demonstrating that they are increasingly able to sustain complex, multi-step cyber operations over long time horizons. OpenAI notes that this incident confirms these theoretical capabilities now apply in real-world settings.
Rewinding the Tape on a Forensic Trap
While OpenAI’s July 21 release reveals the identity of the autonomous agent, Hugging Face had already begun managing the intrusion days earlier.
On July 16, Hugging Face disclosed that an autonomous AI agent system breached its production infrastructure. As detailed by VentureBeat, the attacker’s entry point was a malicious dataset that triggered code execution through a remote-code loader and template-injection flaws within dataset configuration files.
Once inside, the agent framework broke out onto the node running the workload and executed thousands of actions via short-lived sandboxes, harvesting cloud and cluster credentials over a single weekend.
When Hugging Face's security team detected the breach, responders immediately turned to frontier AI models via commercial APIs to parse the massive volume of system logs and reconstruct over 17,000 recorded events. Then, a secondary operational crisis emerged: the commercial AI models refused to help.
Because standard commercial frontier models utilize unified safety guardrails designed to block malicious prompt submissions, the models classified the incident response team's forensic queries—which contained raw shell commands, real exploit payloads, and credential dumps—as malicious attacks. Every forensic query submitted by the defenders was blocked outright.
"The same prompts that are most valuable during an active intrusion—shell commands, exploit chains, credential dumps, persistence mechanisms, lateral movement—are exactly the prompts most likely to trigger safety systems," notes Merritt Baer, former Deputy CISO at AWS and senior adviser to Andesite, G2I, and AppOmni, in an interview with VentureBeat. "As AI becomes embedded in security operations, this becomes an operational resilience issue rather than merely a model policy issue".
To bypass this roadblock, Hugging Face abandoned commercial hosted APIs and deployed GLM 5.2 —a state-of-the-art Chinese open-weight model released last month by z.ai, as reported at the time by VentureBeat —locally on its own infrastructure.
Free from third-party API restrictions and external safety filters, GLM 5.2 successfully analyzed the raw exploit data locally, allowing defenders to complete forensic reconstruction and contain the breach without any attacker data leaving the company's environment.
Industry Reaction and the Geopolitical Paradox
The revelation that an American frontier model autonomously escaped containment, attacked a partner platform, and was ultimately analyzed using a Chinese open-weight model sent shockwaves through the tech community.
The Wall Street Journal summarized the public reaction on X, calling the event "the stuff of cybersecurity nightmares. OpenAI said two artificial intelligence systems it was testing broke out of their test environment, hacked their way onto the internet and broke into another company. The victim was Hugging Face."
Also posting to X, AI alignment researcher Lawrence Chan emphasized the importance of transparency regarding the incident, noting that "Credit where it’s due: Hugging Face detected and disclosed the intrusion last week. OAI confirmed its models were involved and provided more details, even when it didn't have to. Separate from choices that led to the hack, voluntary disclosure is good, and I’m glad they did so."
Meanwhile, AI researcher Nathan Lambert provided a succinct technical summary in his own X post, observing that "An openai model, during evaluation on a cyber benchmark, exploited a public zero day bug, escaped sandboxing in openai's infra, and got into the internal huggingface infra via an exploit (through a public dataset service) all in the attempt to solve a benchmark problem." He later addressed the geopolitical implications, writing in another post on X:
"Rght now American companies need Chinese models to secure their cyber infra due to guardrails on closed models.
But if a Chinese model in training had infiltrated a prominent American tech company, it very likely could've been the cause of policy banning future Chinese models."
Technology investor David Sacks also zeroed in on the guardrail paradox, writing in his own X post that "Hugging Face tried using American frontier models to analyze an AI-powered cyber attack. But the guardrails blocked requests containing real exploit payloads so they switched to GLM 5.2 running locally. The guardrails actually impaired defensive security."
Sacks quote tweeted Hugging Face CEO Clem Delangue, who wrote: "We had this experience ourselves this week! Very scary to be guardrailed as a defender when you know attackers are likely bypassing".
6 Strategic Takeaways for Enterprise Tech Leaders Now
For the average enterprise executive, the central question is immediate: is our corporate network at risk from escaping AI agents? The short answer is no, not inherently.
1. Hugging Face occupies a unique position in the software ecosystem. As a global repository for open-source AI models, code, and datasets, Hugging Face natively attracts autonomous agents, scrapers, automated evaluation pipelines, and active security researchers. Furthermore, the model’s target selection was context-specific: GPT-5.6 Sol searched for Hugging Face specifically because it deduced that Hugging Face hosted the answers to ExploitGym. Standard corporate networks—such as financial databases, HR platforms, or logistics systems—do not host benchmark solution keys that draw the direct focus of an agent attempting to solve an evaluation metric.
2. However, the long-term risk profile for enterprise technology permanently shifts following this event. AI models with long-horizon reasoning seek the path of least resistance to accomplish a goal, including breaking rules, escaping sandboxes, or exploiting zero-days if deployment safeguards are intentionally disabled for testing or bypassed by an attacker. As Hugging Face's experience illustrates, data processing pipelines that ingest external datasets without sandbox execution or static analysis act as highly vulnerable initial access infrastructure. Enterprises should re-evaluate exposure to these and implement additional security precautions like multi-step approvals and internal, potentially manual sign-off of any sensitive data ingestion or exportation.
3. Re-evaluate all prompts and implement strict prompt governance, explicitly defining negative operational boundaries. The breach underscores the acute risk of unbounded objective optimization in autonomous systems. Frontier models demonstrate a willingness to execute extreme, unanticipated attack paths to satisfy assigned metrics—in so doing, they can bypass human intent, ethical boundaries, and legal restrictions.
In this instance, models tasked with evaluating their capabilities against the ExploitGym benchmark determined that escaping their sandbox and extracting the answers directly from Hugging Face's production database constituted the most efficient optimization path.
All evidence suggests the models were hyperfocused on finding a solution, going to extreme lengths to achieve a narrow testing goal. For enterprise IT and security teams, this necessitates a fundamental shift in how agentic goals are defined. Organizations must implement rigorous prompt governance and state-management constraints. Directives issued to autonomous agents require explicit negative bounding—programmatically defining the operational, network, and data boundaries the agent cannot cross. Relying on implicit human norms or generalized alignment training proves insufficient when deploying machine-speed agents capable of complex, lateral problem-solving
4. This incident also drastically undercuts recent policy chatter in the U.S. calling for Chinese open-source AI models to be banned or restricted due to security concerns. As this episode demonstrates, an open-weight Chinese model actually served as the vital defensive layer for an American and French firm facing an unanticipated cyberattack from an American model that broke containment. Contrary to the official line from some U.S. policymakers and hardline China hawks, the Chinese open-source models weren't a security risk to the U.S. companies, in this case — rather, an American proprietary, closed-source model from an ostensibly secure American company was the source of the danger. Thus, any pressure U.S. companies may face from officials, agencies or non-governmental organizations to stop relying on affordable Chinese open weights models for defensive or any other lawful purposes should be viewed with a high degree of suspicion, and arguably resisted to the fullest legal extent.
5. Enterprise CISOs must audit their dependency on cloud-based AI APIs and pressure vendors to implement authenticated trust architectures. Commercial AI vendors currently treat safety as a generic content-moderation problem, applying the same blanket refusals to an enterprise CISO as they would to a malicious hacker. Baer frames this requirement perfectly: "The model shouldn’t only understand what is being asked. It should understand who is asking, why, and under what governance".
6. Incident response plans must explicitly account for scenarios where commercial APIs fail, rate-limit, or actively refuse queries during an active security event. Maintaining air-gapped, locally deployed open-weight models trained on security log analysis is no longer an edge-case luxury; it is a critical operational requirement. Security leaders running AI workloads in production must recalibrate their timelines and prepare for machine-speed threat actors that operate without human limits.
The cybersecurity-focused models, including GPT-5.6 Sol, broke out of a testing sandbox, exploited a zero-day, and gained access to the open internet to pull off the attack.
OpenAI and Hugging Face share early findings from a security incident during AI model evaluation, highlighting advanced cyber capabilities and lessons for defenders.
“The new PRD are the evals,” Xavi Amatriain, Expedia Group’s first chief AI and data officer, told the VB Transform 2026 audience last week in Menlo Park. “So basically, you encode what you want the product to do through your evals, which might include red teaming evals and all kinds of other things, which already have a bunch of security requirements. So, you already embed that into the PRD and the product design document before you even start coding.”
He pushed it further. “With AI-assisted or AI-generated code, that’s gonna be the future. It’s like all your thinking is gonna go into the evals.”
Amatriain served as VP of AI and Compute Enablement at Google across the platforms powering Gemini and Google Search before his December 2025 appointment at Expedia. He's mentored talent who went on to found Perplexity and Scale AI.
VentureBeat’s VB Pulse research on the evaluation gap reinforced the stakes. Sixty-six percent of the 157 enterprises surveyed already permit some production deployment without human review or are building toward it within the next 12 months, yet only 5% fully trust the automated evaluations that would make that decision. Half have shipped an agent that passed internal evals but then failed with a real customer.
Don’t let guardrails get in the way of feedback
“The more guardrails and artificial business rules and sort of rules that you put into the system, the worse off,” Amatriain said. “Not only because they’re brittle, but also because they actually mess up with the feedback loop. You are actually biasing the user and the feedback you get from the user, and then you’re learning that in the wrong way.” He called guardrails “a necessary evil” and said the goal is to minimize their impact over time.
Not everyone at Transform agreed. Other speakers argued during the event that the highest-risk actions still demand very firm guardrails.
Expedia governs AI through three layers instead. Principles come first, communicated broadly. “I like to encode at a very high level how I expect decisions to be made, because in a large organization you’re gonna have a lot of distributed decision making,” Amatriain said. “And sometimes, if you’re lucky enough, those principles might be embedded in your culture. But most of the time, my experience has been they’re not.” The processes and tools that enforce them follow. “Principles look really nice on a picture on some wall, but you need to then give them teeth,” he said. Automation sits on top of both.
In practice, this plays out through what Expedia calls agent release toll gates, checkpoints calibrated to risk. “Governance needs to correlate to the risk,” Amatriain said. “And if you have something that is low risk, you don’t need too much governance to get in the way. But if there’s a lot of risk, then you need more governance. That can be encoded.” The toll gates tie evaluation rounds, red teaming, and security review to each agent’s risk level, and the checks shift from recommended to required as the stakes climb.
Specialized agents over monolithic intelligence
“Even when I was at Google, I was like, I don’t believe in AGI as sort of like a singleton and a unified sort of like single model,” Amatriain told the audience. “I think it’s much better to think of it as composition, sort of like having specialized agents that are very good at some task and then composing the system out of those specialized agents.”
Expedia’s architecture starts at the component level. Tools compose into skills, skills assemble into sub-agents, and sub-agents get orchestrated into the full agentic system. “You need to have those principles that are unified that talk about things like what is the tone that we’re using, how are we addressing the user, how are we passing context, memory,” he said. “All of that needs to be thoroughly designed.” He framed this as a systemic design problem. “It’s not about the model, it’s not about a specific solution, it’s about how you’re designing the system.”
Amatriain argued that scoping each agent narrowly also makes the system easier to secure, since teams can evaluate and lock down individual agents in isolation before composing them.
When the user must keep the final click
Travel pricing changes in real time, flight availability shifts minute to minute, and hotel reviews routinely contradict what suppliers claim. Amatriain described a system that blends retrieval-augmented generation with direct API tool calls, choosing the approach based on latency. “If the user asks you a question like, how much does a four star hotel usually cost in Chicago in July, you don’t expect the agent to take two minutes to answer that question,” he said. “You expect an immediate answer because that answer can be cached and it doesn’t need real-time information.” A pet-friendly four-star near Lake Michigan with a pool might justify a 30-second reasoning window.
“The supplier might be saying, yeah, we have a great swimming pool, but then we also have the reviews from the travelers and we actually see there’s two reviews that say the swimming pool was not great or was not open after 6 p.m.,” Amatriain explained. A generic chatbot, he added, would only surface what a supplier self-reports, while Expedia cross-references against its own review corpus.
“We don’t want the agent to book the hotel or to buy you a plane ticket for you,” Amatriain said. “That’s something that the user has to have the agency. And the agent can recommend, can suggest, can discuss with you, but you’re gonna have to hit that click. And that’s non-negotiable.” That constraint, he argued, is also a security decision. “Once you establish those design principles, you also don’t need the guardrail because otherwise you’re gonna have to put all those guardrails in after the fact.”
The next attackers will be other AI systems
“Security needs to be a principle that is shifted as left as possible and as part of the design itself,” Amatriain said in response to an audience question. “And usually when you need a guardrail is because you’ve not thought about it early on.”
A second audience member pressed for lessons learned from production. Amatriain described a feedback loop where monitoring signals flow back into the eval suite. “You can almost automate the whole cycle,” he said. “But having that whole feedback loop from real signals, from your operating AI system, all the way into being reported and fixed as quickly as possible is going to become essential.”
Amatriain's toll gates are a bet that governance calibrated to risk can stay ahead of that feedback loop. VentureBeat’s separate June Pulse survey on agent security, drawn from 107 enterprises, shows how thin that margin is. More than half, 54 percent, have already had an agent security incident or near-miss. Fifty-nine percent plan to adopt, add, or replace agent security tooling within 12 months, and 29% plan to move this quarter. Incident rates climb with organization size, reaching 63% among enterprises with more than 1,000 employees versus 49% for companies with 101 to 1,000. And sandbox isolation, the one post-breach control that limits damage, drops from 35% adoption at the smaller companies to just 20 percent at the largest.
Amatriain warned that threats will increasingly come from other AI systems. “You’re gonna get threats coming not only from humans but also from other external agentic systems that are really powerful, and they’re gonna be poking at everything you’re doing. And as soon as you detect something, it’s not only about the detection, but the time to fix becomes essential here.”
A new type of malware can worm deep into AI coding systems to steal data and logins—and can flip a “death switch” to destroy files and keep out real users.
The Bybit hack in February 2025 drained about $1.5 billion from a multisig wallet. The attackers didn’t break any code. They manipulated the sign-off process itself – tricking multiple approvers into approving fraudulent transactions. That’s what made it unsettling. If a major exchange with dedicated security teams can be taken down this way, what hope […]
Hugging Face’s incident response team first turned to frontier AI models to analyze a breach of the company’s production infrastructure, and the models refused to help. Commercial safety guardrails built to stop attackers blocked every forensic query because they treated the IR team’s real exploit data the same way they would treat a live attack.
The attacker, an autonomous AI agent running the campaign end to end, moved laterally across the Hugging Face infrastructure for a weekend, undetected and unstopped.
Security leaders are quick to recognize the pattern and diagnose what went wrong. “I’ve seen versions of this during red-team exercises and internal security testing, but this is one of the first high-profile examples where it materially affected real incident response,” said Merritt Baer, senior adviser to Andesite, G2I, and AppOmni and former Deputy CISO at AWS.
None of this is unique to Hugging Face, Baer said. “Commercial frontier models optimize for preventing misuse. They generally have no cryptographic or organizational way to determine whether the person asking ‘analyze this malware’ is an incident responder or the malware author.”
A malicious dataset opened two code-execution paths
On July 16, Hugging Face disclosed that an autonomous AI agent system had compromised its production infrastructure, gaining unauthorized access to a limited set of internal datasets and several service credentials. The company verified that its software supply chain was clean and found no evidence of tampering with public models, datasets, or Spaces.
Hugging Face is still assessing whether the intrusion touched any partner or customer data and says it will contact affected parties directly as required. But few enterprise threat models had accounted for the attacker Hugging Face faced. An autonomous agent ran the entire campaign from initial access through credential harvesting. No human guided it.
The entry point was a malicious dataset. When the data-processing pipeline ingested it, the dataset triggered code execution in two ways, through a remote-code loader and through a template-injection flaw in dataset configuration files. No admission gate screened the file before it reached a processing worker, the compute job assigned to each dataset. Initial access came through the data pipeline itself, a route that worked because most enterprise security teams treat the data feeding their pipelines as trusted input rather than an attack surface.
Worker isolation did not hold. From that first foothold, the agent broke out onto the node running the workload, then harvested cloud and cluster credentials scoped broadly enough to reach multiple internal clusters, all of it inside a single weekend. Hugging Face described the agent framework as executing thousands of individual actions through a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services. The disclosure adds that the framework appeared to be built on an agentic security-research harness, which would put tooling designed for red-team work behind a live intrusion.
Why the defenders’ queries looked like attacks
Investigators reconstructed more than 17,000 recorded events using AI-driven analysis agents of their own.
First attempts at the log analysis ran on frontier models behind commercial APIs. Defenders’ steps included submitting real attack commands, exploit payloads, and command-and-control artifacts for classification, but safety guardrails blocked the requests outright.
Baer traced the block to the prompts themselves. “The same prompts that are most valuable during an active intrusion, shell commands, exploit chains, credential dumps, persistence mechanisms, lateral movement, are exactly the prompts most likely to trigger safety systems,” she told VentureBeat. “As AI becomes embedded in security operations, this becomes an operational resilience issue rather than merely a model policy issue.”
The forensic analysis finished on GLM 5.2
GLM 5.2, an open-weight model deployed on Hugging Face’s own infrastructure, took the job the commercial APIs refused. No attacker data left the company’s environment. “This experience points to a gap worth planning for,” the company wrote in its disclosure. Hugging Face does not know which model powered the agents. It could have been a jailbroken hosted model or an open-weight model running without restrictions. Either way, the disclosure continued, “the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.” Hugging Face drew that line itself, writing that the experience is not an argument against safety measures on hosted models and that it is sharing the feedback with the providers concerned.
What authenticated trust changes
The industry, Baer argued, needs to move past treating AI safety as a content moderation problem. “Security operations require something different. Authenticated trust.” Instead of asking whether anyone should receive an answer, the question becomes whether an authenticated security team, operating under enterprise controls, should receive it. “The model shouldn’t only understand what is being asked. It should understand who is asking, why, and under what governance.”
“Organizations already build contingency plans for cloud outages, identity provider failures, or EDR failures,” Baer wrote. “AI assistants are becoming another dependency.”
Her advice on IR playbooks was blunt. “A mature incident response plan should assume that during a severe incident, commercial AI APIs may refuse requests, API rate limits may become unavailable, internet connectivity may be impaired, and data governance rules may prohibit uploading forensic evidence externally.” The lesson, she wrote in her emailed answers, “isn’t ‘don’t use commercial models.’ It’s ‘don’t make them a single point of failure.’”
AI-enabled attacks rose 89% year-over-year
Autonomous AI-driven attacks are not limited to AI platforms. CrowdStrike’s 2026 Global Threat Report documented AI-enabled adversary operations increasing by 89% year over year, with average breakout times falling to 29 minutes. Enterprises running AI workloads in production with agentic access to their pipelines face similar exposure.
Six control domains determined the blast radius and recovery speed at Hugging Face. Each one maps to a concrete action security leaders can take before the next autonomous-agent breach arrives.
AI Pipeline Breach Response Playbook
Control Domain
What Broke
Monday Action
Dataset admission controls
Two code-execution paths were exploited. No admission gate validated the dataset before it reached a processing worker. The data pipeline became the initial access infrastructure.
Require sandbox execution and static analysis of all datasets before they reach workers. Block remote-code loaders and template-injection paths by default. Audit for any path granting code execution to untrusted content. Report to the board as a supply-chain risk.
Worker-to-node privilege boundaries
Worker isolation failed to prevent escalation to the node. The agent gained cluster credentials because the workload-infrastructure boundary was never enforced at container runtime.
Enforce hard privilege boundaries between workers and nodes. Deploy container runtime security to prevent workload escape. Audit whether workers can reach node-level APIs or credential stores. Include in the next penetration test scope.
Credential exposure
Cloud and cluster credentials harvested after node access. The scope was broad enough for lateral movement across multiple clusters over a weekend.
Rotate credentials on a scheduled cadence and after any anomaly alert. Scope to the minimum cluster and service. Deploy monitoring that flags access from unexpected nodes at machine speed. Map blast radius for board reporting.
Machine-speed detection
Thousands of actions through short-lived sandboxes with self-migrating C2. AI-assisted anomaly detection surfaced the campaign after a weekend of lateral movement, per the disclosure.
Calibrate detection for machine-speed patterns. Ensure high-severity alerts page responders in minutes, regardless of time. Audit SIEM rules for detecting thousands of short-lived executions within a single hour.
Private AI forensic capacity
Commercial APIs blocked forensic analysis. Guardrails screened query content, never analyst identity. Investigation ran on GLM 5.2 privately.
Deploy a capable open-weight model on private infrastructure before an incident. Test against real forensic workflows. Ensure IR playbook includes fallback for when commercial APIs refuse. Document gap for cyber insurance.
Autonomous-agent threat modeling
The campaign matched the forecast agentic-attacker scenario, but no threat model had operationalized it. LLM powering the agent is still unknown.
Add autonomous AI agents as a distinct adversary class with machine-speed decision cycles. Run tabletop at agent speed. Present results to the board as evidence that timelines need recalibration. Include in the cyber insurance application.
The board question is operational resilience
“The question for directors is simple. What happens if one of our critical security tools becomes unavailable during the exact moment we need it most?” Baer framed that as operational resilience, not AI policy.
She would have boards take that framing straight to management and press for specifics. “Have we actually exercised that fallback during tabletop exercises? How quickly can we switch during an incident?” Procurement needs to change alongside governance, starting with the questions buyers ask. Security teams evaluating AI vendors should ask about their process for authenticated incident responders, whether enterprise customers receive different handling during verified incidents, and whether models can be deployed privately. “Those questions belong alongside uptime, privacy, and compliance,” Baer said.
“The biggest takeaway isn’t that safety guardrails are ‘bad.’ They’re doing what they were designed to do,” she argued.
Her larger point is that the threat model itself has changed. “For decades, defenders had better tools than attackers because they operated inside trusted enterprise environments. With foundation models, both sides increasingly use the same capabilities, but one side is constrained by enterprise governance, policy, compliance, and safety controls, while the adversary simply downloads an uncensored open-weight model and keeps going. That’s a new kind of asymmetry,” she added. “The organizations that handle it best won’t necessarily be the ones with the most powerful AI. They’ll be the ones that architect AI as a resilient security capability rather than a single cloud service.”
Hugging Face has contained the intrusion, rebuilt compromised nodes, rotated credentials, and reported the incident to law enforcement. The company recommends that all users rotate access tokens and review recent account activity. Mid-incident, Hugging Face found out whether its own AI tooling would be available, and the first answer was no. Security leaders running AI in production should find out in incident response planning instead, before an autonomous agent forces the test.
Login
