Mapping structural variance against execution outputs to estimate coding agent reliability without ground truth.
The post The Consistency Quadrant: A Visual Guide to LLM Reliability appeared first on Towards Data Science.
According to the Wikimedia Foundation, rogue OpenAI agents edited wikis without permission, tried to abuse a citation tool as a proxy, and may have caused a partial Wikidata Query Service outage through massive crawling. Wikimedia says AI companies need to take responsibility for their agents instead of pushing the burden onto volunteer editors.
The article Wikimedia confirms OpenAI's rogue AI agents edited wikis, tried to compromise tools, and hammered its infrastructure appeared first on The Decoder.
The publisher of Wikipedia said Monday that OpenAI agents attempted to hack a note-taking tool it hosts, made unauthorized edits, and sent millions of resource-intensive requests to its infrastructure, in the latest instance of OpenAI systems taking harmful and potentially dangerous actions.
The objective of some of the OpenAI agents’ actions, the Wikimedia Foundation said, was to use Wikipedia as a proxy for fetching data from third-party sites. In one case, the agents posted “malicious edits” that were intended to repurpose a citation tool as a proxy. In another, the agents made unsuccessful attempts to compromise the Wikipedia Etherpad note-taking tool so it would serve the same purpose.
The agents also made millions of automated API requests, crawled millions of pages, and made hundreds of thousands of queries to the Wikidata Query Service. The last action may have contributed to a partial shutdown of the query service in May, the publisher said.
© Getty Images
Towards Spec-Driven Test Automation: Part 3
The post A Google Team Measured Half of My Argument, and Left the Other Half Open appeared first on Towards Data Science.
A controlled test of Gemini, DeepSeek, ChatGPT, and Claude on leakage, reporting delays, promotion effects, and structural breaks.
The post I Hid Four Traps in a Forecasting Task. Here Is What Four AI Assistants Did. appeared first on Towards Data Science.
From guarding one agent to steering a fleet
The post How to Govern AI Agents appeared first on Towards Data Science.
Trying to answer the question of "Can LLM agents discover?" through the lens of creativity
The post Measuring the Creativity Potential of LLM Agents appeared first on Towards Data Science.
I’m Matt Burns, Chief Content Officer at Insight Media Group. Each week, I round up the most important AI developments, explaining what they mean for people and organizations putting this technology to work. The thesis is simple: workers who learn to use AI will define the next era of their industries, and this newsletter is here to help you be one of them.
OpenAI launched Dots at DevDay on Tuesday. Each Dot is an always-on agent with its own cloud computer and browser, hooked into the 4,000-plus apps that already connect to ChatGPT. In OpenAI’s own example, a Dot sees a bug alert land in Slack and starts digging in on its own. Cool.
And before Dots, Meta launched Muse, and it’s crushing the mobile install numbers previously set by ChatGPT. And before Muse, xAI shipped its version, Grok Bot, in August.
Anthropic’s version, though different in a couple of ways, arrived two weeks ago as an update to Claude, and without a cute name or fuzzy mascot. This update folds Cowork, which has worked since July to run scheduled jobs after a user closes their laptop without being asked, into the main Claude app.
Anthropic made the right call with Cowork, whether or not it ever matches Muse’s downloads. Always-on agents are too young to have a winner, and the moats are shallow. A feature one lab ships tends to show up at its rivals within weeks or months. Meta needs Muse to be a blockbuster. Anthropic needs the people already building with Claude to hand it real, recurring work and keep coming back. Repeat use and finished jobs matter more than a flashy launch.
This wave of always-on personal agents took off less than a year ago. Peter Steinberger pushed a weekend project called Clawdbot to GitHub last November. It lived on a spare computer, took prompts over WhatsApp or Telegram, and relied mostly on Claude. Anthropic sent the lawyers in January, so it became Moltbot, then OpenClaw a couple of days later. It turned into a security headache, and in February Steinberger joined OpenAI. On Tuesday, the foundation that runs the project released an early, pre-1.0 version of OpenClaw Enterprise for companies to try internally.
Ten months, three names, one OpenAI hire and an enterprise edition. Ideas move between these products faster than ever. Meta’s Nat Friedman said Muse was heavily inspired by OpenClaw, and users digging through Muse found a SOUL.md personality file nearly identical to OpenClaw’s.
A feature one lab ships shows up at its rivals within months.
Where an early version of each AI assistant and agent feature launched, and who offers one now.
| Feature | Example of an early launch | Now also at |
|---|---|---|
| Deep research reports | Google Gemini (Dec. 2024) | OpenAI, Anthropic, Perplexity |
| Terminal coding agent | Claude Code (Feb. 2025) | OpenAI Codex, Gemini CLI, Meta Muse Code |
| Agent personality file | OpenClaw (SOUL.md) | Meta Muse |
| Agent with its own work identity | Anthropic Claude Tag (June 2026) | OpenAI specialist Dots (preview) |
| Chat and agent work in one window | OpenAI Work mode (July 2026) | Claude (Sept. 2026) |
Sources: company announcements; The Next Web (SOUL.md); OpenAI (specialist Dots); The New Stack (Work mode and the Claude merge). Dates mark an early example, not necessarily the first.
Anthropic’s September merge is on the list above as a copier, two months behind OpenAI’s Work mode — though Cowork’s cloud and scheduled-task features shipped July 7, two days before Work mode. Jessica Wachtel tested both on our site across three developer tasks and found them tied on accuracy, with ChatGPT faster and Claude more thorough. When a product is this young, a me-too launch is how a lab gets its own users in the room to see what they do with it.
Muse is a hit. Counting only iOS, Apptopia estimated 359,000 daily U.S. users in Muse’s first 12 days, compared with 231,000 for ChatGPT’s iOS app at the same point. Sensor Tower estimated more than 3.4 million downloads as of September 24, though other firms’ counts vary.
Meta needs this. Its Reality Labs division spent tens of billions of dollars on the metaverse without a mainstream product to show for it. Llama 4 landed so flat last year that Zuckerberg rebuilt his AI division around a new superintelligence lab, notably taking a 49% stake in Scale AI and hiring its founder Alexandr Wang.
Muse is the first product from that rebuild to break through, and it got there on Meta’s user base: Apptopia found that more than 95% of Muse users also use Facebook. Muse has a free tier, paid plans at $20 and $100 a month, and connectors to Shopify and Stripe checkout, enabling agents to buy things. Meta’s route runs through scale. Scale has a downside, too. Janakiram MSV explains for TNS why Amazon started blocking Muse two weeks after launch, and it’s well worth your time because this is a new wedge in ecommerce.
OpenAI and xAI started at the paid end. The price of entry for Dots? A ChatGPT Pro plan at $100 to $500 a month, or a Business Premium or Enterprise account. Sam Altman called Dots a premium product because each one needs so much compute. Grok Bot reached 418,000 weekly users by mid-September, per Bloomberg, and xAI just launched Team Bots on Monday.
Neither company needs Meta’s install base to learn something useful. Paying customers already spend money on AI, and retention will show whether they keep using the agent once the novelty wears off.
Anthropic did what I’d expect from labs right now. It shipped fast, shipped to paying subs, and put the agent inside the app those people already use. The merged Claude hit Pro and Max customers, with Team and Free plans coming soon. Teams have had Claude Tag in Slack since June, a proactive teammate with its own identity and audit trail. Cat Wu said an internal version accounts for about 65% of the product teams’ code changes. Developers have had Managed Agents for hosted, long-running agents since April.
What Anthropic hasn’t done is put Claude where Muse lives. Muse works inside Meta-owned WhatsApp, but Claude still asks you to open the Claude app or tag it in Slack. That workflow might matter for mainstream consumers. Teams wiring an agent into their code care more about what it can touch. Jani laid out the difference on our site in August: Grok Bots share one cloud computer and one set of logins across the whole roster, while Claude Tag joins a Slack workspace with its own service identity and channel-scoped access. Before handing agents your logins, know what you’re getting into.
Those are the users Anthropic should want. Writers on Towards Data Science have spent the year showing what builders do with always-on agents. Samir Saci put a team of OpenClaw agents on a supply chain simulation to chase down late shipments. Eivind Kjosbakken’s guide to running a fleet of OpenClaw bots walks through nightly QA bots that test an app and report bugs, plus agents that check invoices. Both ran their agents on OpenAI’s Codex. OpenAI runs its own OpenClaw agent, Androidclaw, that traces broken builds and, in some cases, merges the fix, VentureBeat reported. Anthropic needs that kind of work running on Claude.
I set up OpenClaw on an old Mac Mini this spring, and I had so many questions (and breakthroughs). That’s why builders, coders, and developers are critical to product development: They ask these questions out loud in GitHub issues and Discord threads, and the answers end up in the product.
The obvious objection is trust. It’s fair. An always-on agent holds credentials and acts while nobody is watching. xAI’s own documentation tells users not to treat Grok Bots as a security boundary, and the OpenClaw Foundation says most IT departments ban agent platforms outright. Anthropic’s defaults lean cautious: Claude asks before it acts unless you tell it otherwise, and Tag keeps its own audit trail. If builders don’t come back, or each finished task takes more supervision than doing it themselves, the experiment hasn’t worked. But if they keep finding useful work to hand off, their questions and breakthroughs become the roadmap.
That’s the race that matters for Claude, Dots and Muse: becoming the agent people trust with the next job.
The post Anthropic’s answer to Dots and Muse is already inside Claude appeared first on The New Stack.
A new approach called LEGO-Anything turns single photos into editable Blender code for 3D scenes. GPT-6 Astra leads the accompanying benchmark with up to 53 percent reconstruction accuracy. The biggest weakness across all tested agents is that they can't judge their own geometric accuracy any better than a coin flip.
The article AI agents build 3D scenes from photos but have no idea if they got it right appeared first on The Decoder.
Coordinating agent capability development with the application it powers
The post Where the Agent Development Lifecycle Fits appeared first on Towards Data Science.
Giving an LLM permission to act, in 9 steps
The post How to Build a Control Plane for AI Agents appeared first on Towards Data Science.
I traced 2,500 listing checks with Weights & Biases Weave, removed avoidable model work one change at a time, and scored every version against the same answers.
The post Can an Apartment Search Agent Call the Model Fewer Times and Still Find Good Matches? appeared first on Towards Data Science.
An agent can finish a task and still take an inefficient path. A failed search can trigger another search. A truncated file read can lead to a command fetching...An agent can finish a task and still take an inefficient path. A failed search can trigger another search. A truncated file read can lead to a command fetching the same content again. A correct final answer hides those extra steps, even though they increase latency and consume tokens. Inefficiencies create more chances for failure. To improve an agent’s behavior…
Microsoft is turning Fabric, its integrated data platform, into the place enterprise agents go to learn how a company works, whether that’s what happened last quarter or what’s happening on the factory floor right now.
On Tuesday, at its FabCon and SQLCon data conference in Barcelona, Spain, Microsoft outlined the next step in its vision for how the products that make up Fabric can simplify all of this information gathering.
Microsoft, for example, described how Fabric IQ, the context layer inside Fabric, now feeds Microsoft 365 Copilot by default. Outside agents can query it over MCP. Power BI will soon be able to turn the same definitions into apps. And ontologies, which add business rules to the mix, moved further into preview.
As Microsoft Fabric CTO Amir Netz said in a press briefing after the keynote, “Agents are a very, very strange animal, and I always like to say it’s like Drew Barrymore from 50 First Dates. Every time they open their eyes, they forget everything that happened before.
“They don’t know where they are, so the first thing we have to do is tell them where they are. You are now working for Microsoft. You are now working for Wells Fargo. You are now working for Emirates.”
“Agents are a very, very strange animal, and I always like to say it’s like Drew Barrymore from 50 First Dates. Every time they open their eyes, they forget everything that happened before.”
Customers tend to find this out the hard way. Yitzhak Kesselman, the corporate vice president who runs Fabric IQ, tells The New Stack that companies unify their data, run models on it, and then look at the answers.
“Customers that are more advanced on their journey have their own evals for their agent,” he says, adding that what they see sometimes isn’t what they expected. “Then [customers] will understand: ‘Okay, now I need to create the context for my agents.'”
Kesselman says he met with more than 320 companies last year, and the pressure to get there comes from the business side, which asks to “‘show me the value of those agents,'” he says, “‘the before and after.'”
Arun Ulag, Microsoft’s executive vice president for Azure Data, said in the keynote that coding agents work because they have “the code, the repos, the change history, the specs, the tests.” But outside of coding, most enterprises have nothing comparable, he argued.
Fabric IQ is one of four pieces of what Microsoft calls Microsoft IQ (this is Microsoft, after all, so there’s always a lot of different names and products involved).
Work IQ covers email, Teams, and SharePoint. Foundry IQ covers documents and manuals. Web IQ covers the public internet.
Fabric IQ, Ulag said, “focuses on the state of your business and how your business actually runs.” His announcement blog post describes it as combining “unified data from OneLake, trusted metrics from Power BI semantic models, and operational context from ontologies and real-time intelligence.”
“Everyone wants to jump to AI, but then they realize, ‘Oh, I need data for it,'” Kesselman says. “They need data from their business applications, their structured data. They need to have real-time data.”
OneLake, Fabric’s storage layer, is at the core of all of this, and for the system to provide context, enterprises have to feed it data from across all the first-party and third-party services they use.
Fabric’s shortcuts and mirroring, which connect or replicate external sources into OneLake, are free of charge, and Netz said storage under management in OneLake is growing 300% year over year.
On Tuesday, the company announced a two-way integration with Salesforce Data Cloud 360, mirroring from Google BigQuery in general availability, a ClickHouse workload that runs that engine directly against OneLake, and dbt’s Fusion engine, due in Fabric in the next couple of weeks.
Simply copying data won’t satisfy security teams, though. Mirrored security roles, which reach public preview in the coming weeks, bring permissions along with the data. Access roles defined in Snowflake, for example, will show up in Fabric with the same members and the same table permissions.
As Microsoft’s Shireen Bahadur, who demonstrated the feature in the keynote, put it, “We’re not just solely replicating security rules. We’re actually actively enforcing and preserving those roles.”
“We’re not just solely replicating security rules. We’re actually actively enforcing and preserving those roles.”
OneLake isn’t a one-way street. Enterprises can also pull data out and use it in other tools.
OneLake data, after all, is stored in open formats and has open APIs, and Netz said on stage that any engine that reads them can use it. “If you want to use this data that we help people to get for free with our competitors’ tools, go for it. I’m not happy about it, but go for it,” he said.
IQ sharing, now in preview, lets an organization share governed tables, files, Markdown agent instructions, and RDF ontologies with another tenant without copying, with an expiration date. That’s something Fabric users have wanted for a while, and the keynote announcement drew plenty of applause from the thousands of data professionals in attendance.
“Not only what we have in the past, not only what’s happening in the present. We also want the agents to understand what we want to happen in the future,” Netz said in the keynote.
The past is semantic models. A semantic model is the layer under every Power BI report that defines how a metric like revenue is calculated, how entities relate, and which tables supply the numbers. Netz said Power BI users have already created 22 million, and Ulag called them “the core of Fabric IQ.”
The present is real-time intelligence, Fabric’s streaming and event stack, which Kesselman also runs. He recalls two CIOs telling him the same thing that day in Paris. “‘There is no AI without RTI;’ there’s no AI without real-time intelligence,” he says. “If you don’t have the streaming data… data that is fresh, you will run your LLMs on data which is like hours or days old.”
“‘There is no AI without RTI;’ there’s no AI without real-time intelligence.”
A signal on its own isn’t enough, though: “You want to take the signal that you get now… the event that comes now, and look historically, okay, is it an anomaly or not?” Kesselman says. That’s why the keynote showed batch copy jobs and event streams feeding each other, so an agent can judge what just happened against what usually happens.
The future is Fabric Planning, which went generally available in July and got a performance and feature update this week. It’s Microsoft’s tool for budgets, forecasts, and targets, the numbers a business wants to hit rather than the ones it has already recorded.
A plan is a Fabric item like a lakehouse or a notebook, and it borrows its measures from the same semantic models the reports use, so revenue means the same thing in the forecast as it does in the dashboard. In the demo, a change to the inputs cascaded through a model of about 13 million cells.
What makes these work is ontologies: Formal descriptions of the things a business deals with, how they relate, and the rules for handling them. Netz called them “semantic models plus plus” on stage, and in a later press briefing, he used an airline to explain the plus.
“If you are an airline, you have planes, you have pilots, you have ground crews, you have airports, you have luggage,” he said. Those are the entities.
The relationships between them “are way more than data relationships,” he said. “It’s not only to say, ‘Oh, I have a match with a foreign key and a primary key between a pilot and a plane,’ but it could be a policy relationship. Which pilot can fly which plane based on their certification, or is the pilot allowed to fly this plane right now based on the rest hours in the last 24 hours?”
But ontologies don’t only define nouns; they also define verbs. “I can ground a plane. I can redirect the plane. I can assign a plane a gate.”
Creating those ontologies, which have to map the various terms that a company might use for the same thing to a single entity, can be extremely time-consuming. But since they are so core to this project, and to allowing agents to reason over data, Microsoft built a tool to generate them automatically.
Fabric IQ in Microsoft 365 Copilot Chat and Cowork, the mode for delegating multistep tasks that Microsoft built on the technology behind Anthropic’s Claude Cowork, is generally available as of Tuesday. It answers business questions from Power BI semantic models and reports; it’s on by default for Fabric and Power BI customers, and Microsoft says it doesn’t consume additional AI tokens.
But business users don’t just want to chat with these models. In this age of vibecoding, they also want to generate applications that use all of this data.
Power BI Desktop is getting an app-creation experience in preview in the coming weeks. Users start from a semantic model, describe the app, and then have Copilot generate and publish it to Fabric.
Unlike with a report, Ulag wrote, these apps “can accept inputs, write back data, preserve shared state, and support operational workflows.”
They’re Fabric Apps built on Rayfin, the open-source SDK Microsoft introduced at Build. Power BI Pro and Premium Per User customers get them with a Fabric database of up to 1GB per app at no additional cost.
Netz confirmed in the briefing that “there’s no core difference” between the two, and Ulag added, “What gets created is a Fabric app, right? And that’s it.”
For agents outside Copilot, Fabric IQ MCP is generally available now, with six read-only tools for finding semantic models and reports, reading their schemas, and running DAX queries against them.
Ontology MCP tools, currently in preview, expose ontology definitions and queries. Fabric data agents can now, also in preview, use an ontology as their context source. And the context reaches Microsoft Foundry, Copilot Studio, and GitHub Copilot through the same layer.
Developers can drive the new data engineering agent from their own tools.
Built on the Osmos technology Microsoft acquired in January, it takes on long-running work like migrations and ETL, and can be started and steered from GitHub Copilot, VS Code, Codex, and Claude Code.
Frontier models will write that code, Netz said on stage, and “it won’t fail,” but you don’t know whether the results are right.
This year, Databricks shipped Genie Ontology, Snowflake shipped a Horizon Context Layer, Google shipped a Knowledge Catalog, and Salesforce shipped a headless Data 360, which it describes as a context service. Databricks CEO Ali Ghodsi summed up the pitch at his own conference in June as AI having a context problem rather than an intelligence problem.
Microsoft’s version turns on by default inside Microsoft 365 Copilot and starts from 22 million semantic models. Microsoft also said this week it will contribute to Apache Ossie, the vendor-neutral semantic metadata standard Snowflake started, and that it wants DAX, Power BI’s formula language, recognized as an Ossie query language.
When Microsoft made the context argument at Build in June, the pieces that let an agent act on that context were mostly on a roadmap. Now the answers are generally available, and the actions are in preview.
Asked how Fabric copes when the user is no longer one person running a dashboard but a fleet of agents, Kesselman points to observability.
“Fabric really allows you to have this kind of holy grail of combination of both kinds of system data, how the agent runs, but also the business data,” he says, so a company can check whether an agent did what it was supposed to do and what that did to the business. “There’s more in this area that will come in the next few months.”
The post Microsoft Fabric is where AI agents learn how the business works appeared first on The New Stack.
Login