It’s not every day that attackers can force a frontier AI model to cough up user passwords and other sensitive data without user confirmation. That’s exactly what researchers recently did to Microsoft 365 Copilot for enterprise. Even more unusual is the source they tapped to discover the critical vulnerability that made their exploit possible. Rather than employing reverse engineering or other traditional vulnerability-hunting methods, they asked Copilot. The LLM assistant readily complied.

Researchers at security firm Varonis knew they wanted to create an exploit that would exfiltrate user data when a user did nothing more than click on a link. Like most AI assistants today, Copilot steadfastly refused and made clear that sensitive prompts like that require explicit user consent in the form of a gesture, such as pressing a return key or other key. In response, the researchers peppered Copilot with questions about the guardrails that required user confirmation before the assistant could execute powerful commands.

Loose lips sink ships

The dialog was like a game of 20 questions. Each answer provided a new clue that divulged information about the complex safety mechanism. Why was auto-execution impossible, they asked. What URL structures and deep links were involved? What happens when a page is loaded with input already in the prompt field? Each answer provided a deeper view into the guardrail and its limits. Eventually, Copilot provided a stunning Microsoft trade secret—an undocumented prompt parameter that completely bypassed the requirement for user consent.

Read full article

Comments

© Photo Illustration by Thomas Fuller/SOPA Images/LightRocket via Getty Images

Gravis Robotics has raised $200 million in a Series A investment from SoftBank to scale its autonomous heavy machinery technology across construction sites globally. The company says growing demand for energy networks, data centers, housing, transit and climate-resilient infrastructure is placing increasing pressure on the construction industry. Labor shortages are adding to the challenge, while […]

Last Tuesday, Microsoft patched a vulnerability it rated as max critical in its M365 Copilot AI platform. On Monday, the researchers who discovered the vulnerability and reported it to Microsoft revealed how their proof-of-concept exploit could retrieve 2FA codes and other sensitive data from emails accessible to Copilot.

Microsoft and other LLM providers have been unable to prevent their products from complying with malicious requests to reveal data. The root cause: AI bots are unable to distinguish between instructions provided by users and those snuck into third-party content the models are summarizing, drafting responses to, or using to perform other actions on behalf of the user. With no way to secure this crucial boundary, Microsoft and its peers are left to erect complicated and ad hoc guardrails designed to rein in the consequences of this incurable gullibility.

Jumping over guardrails

One guardrail built into Copilot and most other LLMs prevents them from submitting web forms, sending emails, and taking similar actions that can be used to exfiltrate data from the user. To work around this, LLM hackers turned to markup language, which, among other things, allows users to add formatting elements such as headings, lists, and links to text without the need for HTML tags. Another workaround is to wrap sensitive data inside HTML tags such as <img> and <form>. In either case, a web request showing the data hits the attacker’s web server, where the secret information is captured in logs.

Read full article

Comments

© Photo Illustration by Thomas Fuller/SOPA Images/LightRocket via Getty Images