Impressive benchmark results should make it easy for engineers to select the right open model for the outcome they are tasked with developing. However, that is not the case.
Benchmarks indicate an open model’s optimal performance, but in production, a model must work repeatedly. The challenge is that in production, other factors matter, such as the information provided to the model and its accuracy. So benchmarks can’t tell an engineer whether the AI they are developing will complete the task accurately, or even at all. This is especially important for long-horizon AI tasks.
Ryan Hanrui Wang, SVP of Applied AI at AI cloud provider Nebius, tells The New Stack it is important to measure an open model by the industry benchmarks in conjunction with internal benchmarks and harnesses to ensure you get the inference outcomes the organization requires, something he says the Amsterdam, Netherlands-based company always does in partnership with its clients.
Wang joined Nebius when it acquired Eigen AI, the inference-optimization startup he co-founded and led as CEO, in a roughly $643 million deal announced in May; Nebius sells inference through its Token Factory platform.
“Customers often have their own benchmarks, so we can use their agent workflow and its traces to see the level of success we can work with. We always work closely with them to define the benchmarks.”
Wang goes on to add: “The public benchmark may not cover the multi-domain tasks that an engineer wants inference to do in the actual production environment.” He says this can misalign agent tasks with the true cost of inference calls. “The workflow is not tested by the benchmark design. There needs to be in-context learning.”
Engineers and organizations should therefore not spend too much time selecting the open model; they should look at the entire inference stack. Model selection should depend on whether the engineer needs a lower-cost, flexible open-model stack or a frontier model for inference.
For those who need lower cost, Wang says Nebius evaluates open models such as Kimi and DeepSeek, as well as lighter “flash” models, to see whether they can meet inference accuracy needs and whether fine-tuning is required.
Those who want superior accuracy and quality usually narrow it down to two or three large open models and then fine-tune them to improve accuracy. Wang says open model selection is only the first step; after selection, they need to ensure long-context tasks and cached inputs deliver the expected outcome.
In post-training, Wang says the organization uses its data to improve accuracy and task fit, while the serving layer, quantization, batching, caching, and kernel optimization drive inference speed. In addition, engineers should develop the cache system for production traffic, ensuring the hit rate is maximized for the workload so repeated inputs hit the cache rather than being recomputed.
“The other important thing is throughput, the measure of how many tokens are generated in a certain time span, as they are a major trade-off against interactivity,” Wang adds. Engineers need to balance better utilization and cheaper tokens with acceptable interactivity– per-user speed.
Getting concurrency right ensures the organization uses the open model effectively; it is a fine balancing act for engineers. If compute isn’t fully utilized, token cost is high. But high utilization with increased concurrency can reduce end-user token-per-second speed.
Engineers also need to watch tail latency, not just averages. “P95 and P99 are important metrics for the per-user token speed and time to first token (TTFT); the time they need to wait needs to be below five seconds for some of our customers,” Wang says. A system can post a great average response time and still be unusable if one request in twenty stalls.
Engineers need to be careful when considering benchmark results and accuracy. Benchmarks may test reasoning on a mathematical problem, which isn’t very useful if the engineer is developing an AI agent for customer support, where high emotional intelligence (EQ) matters more. “So there is a misalignment between the different benchmarks and actual production,” Wang says.
On cost per successful task, Wang says engineers must remember that cheap tokens are useless if the model is inaccurate.
Engineers need to consider the end-to-end process the AI will perform. This lets engineers select the right-sized open model for the AI task. Wang gives the example of a specialized assistant agent where Nebius gathered 6,000 traces to refine a small model and improved accuracy by 11% while executing 22% fewer tool calls.
He raises this point because it counters the tendency to think bigger is better, but accuracy gains from increasing model size diminish. “There is typically a tipping point for this kind of task,” Wang says. For a task like customer support, he says, 10 billion parameters could be enough, while 100 billion or even a trillion parameters yields only minimal accuracy gains. “What we want to do is find the sweet spot.”
Engineers can use large models as a reference point, then tune the reasoning with their organizational data to find an open model with the right accuracy and size for the task.
“We typically do a sweep of several models, using the largest model as a reference to know where the sweet spot for model size is,” Wang says. “For a complicated task, we can dissect the complex task into smaller subtasks, working with the customer, which connects workflows and can then use smaller models.”
Wang describes this process as a division of labor, referencing the famous economist Adam Smith.
This approach shows the course of AI engineering maturity. Like every sector in technological and engineering history, the initial gold rush targets bigger-is-better. As the engineering matures, it focuses on outcomes and engineering excellence, and therefore on working with partners who understand how to achieve them.
Sometimes that is a bigger model; other times a small niche development is best.
The post Assess the entire inference stack — not just open model benchmarks appeared first on The New Stack.
Meta launches Muse Spark Open, a coding-optimized 816B mixture-of-experts (MoE) model with 42B active parameters and a 1M token context window. Trained on...Meta launches Muse Spark Open, a coding-optimized 816B mixture-of-experts (MoE) model with 42B active parameters and a 1M token context window. Trained on NVIDIA GB300 NVL72 Blackwell Ultra specifically for agentic coding workflows, it delivers higher first attempt accuracy, reliable tool calling, and deep codebase understanding across repositories. The 1M token context window holds an entire…
This week, OpenAI opened its Decisions API to public beta and added image inputs to make fast decisions from visual content.
First announced last week as a likely response to TypeSafe’s Jev, OpenAI is now billing its decision model at $0.10 per 1M input tokens. There are no output-token charges, nor charges for cache reads or writes.
Running on GPT-6 Luna, the public beta version gives all developers the ability to turn text or images into three types of structured outputs: predicates (i.e., how likely something is to be true), choices (i.e., which predefined option is the best fit), and scores (i.e., where an input falls on a numeric range).
But OpenAI’s Decisions API may be but one fish in a sea of decision models. Perplexity, Cloudflare, and Amazon also dropped their own versions this month, opting for open-weight models that developers can download and run on their own infrastructure.
In a new video, OpenAI shows how image inputs can help applications take quick action based on visual understanding.
It offers the example of a video game with a car moving through traffic. Using frames of the road and obstacles, Decisions API can choose whether the car should switch lanes or keep driving.
OpenAI claims its API can make those decisions in a fraction of the time a reasoning model would take to do the same.
Zooming out, the AI company hints at what this visual understanding could enable if developers pair the API with basic computer-use tasks, like taking screenshots and quickly choosing the next action.
The same video also shows OpenAI received a preview unit of Hugging Face’s upcoming programmable Microduck robot, an open-source bipedal developer robot.
By integrating the robot with GPT-Live, OpenAI’s real-time voice model, and the Decisions API, OpenAI says its decision model can analyze the robot’s camera frames to make fast decisions about where it should look.
In the given example, the Decisions API responds to a request to “follow the apple” by first identifying the fruit and then directing the robot to turn toward it. OpenAI adds that the GPT-Live and Decisions API pairing can go further and respond to more ambiguous commands, like “follow the fruit.”
OpenAI also promises its new API can bring more dimension to voice conversations.
It shows GPT-Live and Decisions API paired with an animated character; the former handles the voice and the latter steps in to choose which expression the character should use.
TypeSafe kicked up quite a storm when it released Jev last month. But its time in the spotlight seems to be coming to a quick end. In addition to OpenAI’s foray into decision models, Perplexity, Cloudflare, and Amazon have all stepped onto the scene — but they’re letting developers download the model weights and run them themselves.
Perplexity, Cloudflare, and Amazon have all stepped onto the scene — but they’re letting developers download the model weights and run them themselves.
Released on Hugging Face this month, Perplexity’s pplx-decider-v1-27b is an open-weight decision model fine-tuned from Qwen3.8-27B, released under Apache 2.0. Per its model card, it supports a similar decision pattern to TypeSafe and OpenAI: selecting among predefined choices and returning yes/no probabilities.
Unlike OpenAI, Perplexity showed its benchmark results, reporting its decision model beats Jev overall across 11 benchmarks at 85.71% to TypeSafe’s 84.51%. But that overall score comes from a mixed bag; Jev takes the cake on six out of 11 benchmarks, including WinoGrande, BBH, and TruthfulQA binary.
Amazon’s local answer to Jev, Strands Decider 2B, is also a downloadable model — this time, built on Qwen3.5-2B. Amazon says it ranks second among public models with roughly 2 billion parameters on JevBench’s public set.
Like OpenAI’s and Perplexity’s models, Clef has a vision encoder to accept images as well as text. Jev and Strands Decider are currently text only.
Cloudflare, meanwhile, has given developers two decision models, Clef and Clef-flash, both available as Apache 2.0 open weights, as well as through its own hosted Workers AI service. Like OpenAI’s and Perplexity’s models, Clef has a vision encoder to accept images as well as text. Jev and Strands Decider are currently text-only.
More consequential than image or text inputs is OpenAI’s decision to keep its model behind a hosted API, even as many contemporaries are opening up their versions so developers can download and run them themselves. As decision models become the new norm, where and how they can be deployed may become the deciding factor in which model developers let decide.
The post Decision models are suddenly everywhere. OpenAI’s is now public. appeared first on The New Stack.
Microsoft is turning Fabric, its integrated data platform, into the place enterprise agents go to learn how a company works, whether that’s what happened last quarter or what’s happening on the factory floor right now.
On Tuesday, at its FabCon and SQLCon data conference in Barcelona, Spain, Microsoft outlined the next step in its vision for how the products that make up Fabric can simplify all of this information gathering.
Microsoft, for example, described how Fabric IQ, the context layer inside Fabric, now feeds Microsoft 365 Copilot by default. Outside agents can query it over MCP. Power BI will soon be able to turn the same definitions into apps. And ontologies, which add business rules to the mix, moved further into preview.
As Microsoft Fabric CTO Amir Netz said in a press briefing after the keynote, “Agents are a very, very strange animal, and I always like to say it’s like Drew Barrymore from 50 First Dates. Every time they open their eyes, they forget everything that happened before.
“They don’t know where they are, so the first thing we have to do is tell them where they are. You are now working for Microsoft. You are now working for Wells Fargo. You are now working for Emirates.”
“Agents are a very, very strange animal, and I always like to say it’s like Drew Barrymore from 50 First Dates. Every time they open their eyes, they forget everything that happened before.”
Customers tend to find this out the hard way. Yitzhak Kesselman, the corporate vice president who runs Fabric IQ, tells The New Stack that companies unify their data, run models on it, and then look at the answers.
“Customers that are more advanced on their journey have their own evals for their agent,” he says, adding that what they see sometimes isn’t what they expected. “Then [customers] will understand: ‘Okay, now I need to create the context for my agents.'”
Kesselman says he met with more than 320 companies last year, and the pressure to get there comes from the business side, which asks to “‘show me the value of those agents,'” he says, “‘the before and after.'”
Arun Ulag, Microsoft’s executive vice president for Azure Data, said in the keynote that coding agents work because they have “the code, the repos, the change history, the specs, the tests.” But outside of coding, most enterprises have nothing comparable, he argued.
Fabric IQ is one of four pieces of what Microsoft calls Microsoft IQ (this is Microsoft, after all, so there’s always a lot of different names and products involved).
Work IQ covers email, Teams, and SharePoint. Foundry IQ covers documents and manuals. Web IQ covers the public internet.
Fabric IQ, Ulag said, “focuses on the state of your business and how your business actually runs.” His announcement blog post describes it as combining “unified data from OneLake, trusted metrics from Power BI semantic models, and operational context from ontologies and real-time intelligence.”
“Everyone wants to jump to AI, but then they realize, ‘Oh, I need data for it,'” Kesselman says. “They need data from their business applications, their structured data. They need to have real-time data.”
OneLake, Fabric’s storage layer, is at the core of all of this, and for the system to provide context, enterprises have to feed it data from across all the first-party and third-party services they use.
Fabric’s shortcuts and mirroring, which connect or replicate external sources into OneLake, are free of charge, and Netz said storage under management in OneLake is growing 300% year over year.
On Tuesday, the company announced a two-way integration with Salesforce Data Cloud 360, mirroring from Google BigQuery in general availability, a ClickHouse workload that runs that engine directly against OneLake, and dbt’s Fusion engine, due in Fabric in the next couple of weeks.
Simply copying data won’t satisfy security teams, though. Mirrored security roles, which reach public preview in the coming weeks, bring permissions along with the data. Access roles defined in Snowflake, for example, will show up in Fabric with the same members and the same table permissions.
As Microsoft’s Shireen Bahadur, who demonstrated the feature in the keynote, put it, “We’re not just solely replicating security rules. We’re actually actively enforcing and preserving those roles.”
“We’re not just solely replicating security rules. We’re actually actively enforcing and preserving those roles.”
OneLake isn’t a one-way street. Enterprises can also pull data out and use it in other tools.
OneLake data, after all, is stored in open formats and has open APIs, and Netz said on stage that any engine that reads them can use it. “If you want to use this data that we help people to get for free with our competitors’ tools, go for it. I’m not happy about it, but go for it,” he said.
IQ sharing, now in preview, lets an organization share governed tables, files, Markdown agent instructions, and RDF ontologies with another tenant without copying, with an expiration date. That’s something Fabric users have wanted for a while, and the keynote announcement drew plenty of applause from the thousands of data professionals in attendance.
“Not only what we have in the past, not only what’s happening in the present. We also want the agents to understand what we want to happen in the future,” Netz said in the keynote.
The past is semantic models. A semantic model is the layer under every Power BI report that defines how a metric like revenue is calculated, how entities relate, and which tables supply the numbers. Netz said Power BI users have already created 22 million, and Ulag called them “the core of Fabric IQ.”
The present is real-time intelligence, Fabric’s streaming and event stack, which Kesselman also runs. He recalls two CIOs telling him the same thing that day in Paris. “‘There is no AI without RTI;’ there’s no AI without real-time intelligence,” he says. “If you don’t have the streaming data… data that is fresh, you will run your LLMs on data which is like hours or days old.”
“‘There is no AI without RTI;’ there’s no AI without real-time intelligence.”
A signal on its own isn’t enough, though: “You want to take the signal that you get now… the event that comes now, and look historically, okay, is it an anomaly or not?” Kesselman says. That’s why the keynote showed batch copy jobs and event streams feeding each other, so an agent can judge what just happened against what usually happens.
The future is Fabric Planning, which went generally available in July and got a performance and feature update this week. It’s Microsoft’s tool for budgets, forecasts, and targets, the numbers a business wants to hit rather than the ones it has already recorded.
A plan is a Fabric item like a lakehouse or a notebook, and it borrows its measures from the same semantic models the reports use, so revenue means the same thing in the forecast as it does in the dashboard. In the demo, a change to the inputs cascaded through a model of about 13 million cells.
What makes these work is ontologies: Formal descriptions of the things a business deals with, how they relate, and the rules for handling them. Netz called them “semantic models plus plus” on stage, and in a later press briefing, he used an airline to explain the plus.
“If you are an airline, you have planes, you have pilots, you have ground crews, you have airports, you have luggage,” he said. Those are the entities.
The relationships between them “are way more than data relationships,” he said. “It’s not only to say, ‘Oh, I have a match with a foreign key and a primary key between a pilot and a plane,’ but it could be a policy relationship. Which pilot can fly which plane based on their certification, or is the pilot allowed to fly this plane right now based on the rest hours in the last 24 hours?”
But ontologies don’t only define nouns; they also define verbs. “I can ground a plane. I can redirect the plane. I can assign a plane a gate.”
Creating those ontologies, which have to map the various terms that a company might use for the same thing to a single entity, can be extremely time-consuming. But since they are so core to this project, and to allowing agents to reason over data, Microsoft built a tool to generate them automatically.
Fabric IQ in Microsoft 365 Copilot Chat and Cowork, the mode for delegating multistep tasks that Microsoft built on the technology behind Anthropic’s Claude Cowork, is generally available as of Tuesday. It answers business questions from Power BI semantic models and reports; it’s on by default for Fabric and Power BI customers, and Microsoft says it doesn’t consume additional AI tokens.
But business users don’t just want to chat with these models. In this age of vibecoding, they also want to generate applications that use all of this data.
Power BI Desktop is getting an app-creation experience in preview in the coming weeks. Users start from a semantic model, describe the app, and then have Copilot generate and publish it to Fabric.
Unlike with a report, Ulag wrote, these apps “can accept inputs, write back data, preserve shared state, and support operational workflows.”
They’re Fabric Apps built on Rayfin, the open-source SDK Microsoft introduced at Build. Power BI Pro and Premium Per User customers get them with a Fabric database of up to 1GB per app at no additional cost.
Netz confirmed in the briefing that “there’s no core difference” between the two, and Ulag added, “What gets created is a Fabric app, right? And that’s it.”
For agents outside Copilot, Fabric IQ MCP is generally available now, with six read-only tools for finding semantic models and reports, reading their schemas, and running DAX queries against them.
Ontology MCP tools, currently in preview, expose ontology definitions and queries. Fabric data agents can now, also in preview, use an ontology as their context source. And the context reaches Microsoft Foundry, Copilot Studio, and GitHub Copilot through the same layer.
Developers can drive the new data engineering agent from their own tools.
Built on the Osmos technology Microsoft acquired in January, it takes on long-running work like migrations and ETL, and can be started and steered from GitHub Copilot, VS Code, Codex, and Claude Code.
Frontier models will write that code, Netz said on stage, and “it won’t fail,” but you don’t know whether the results are right.
This year, Databricks shipped Genie Ontology, Snowflake shipped a Horizon Context Layer, Google shipped a Knowledge Catalog, and Salesforce shipped a headless Data 360, which it describes as a context service. Databricks CEO Ali Ghodsi summed up the pitch at his own conference in June as AI having a context problem rather than an intelligence problem.
Microsoft’s version turns on by default inside Microsoft 365 Copilot and starts from 22 million semantic models. Microsoft also said this week it will contribute to Apache Ossie, the vendor-neutral semantic metadata standard Snowflake started, and that it wants DAX, Power BI’s formula language, recognized as an Ossie query language.
When Microsoft made the context argument at Build in June, the pieces that let an agent act on that context were mostly on a roadmap. Now the answers are generally available, and the actions are in preview.
Asked how Fabric copes when the user is no longer one person running a dashboard but a fleet of agents, Kesselman points to observability.
“Fabric really allows you to have this kind of holy grail of combination of both kinds of system data, how the agent runs, but also the business data,” he says, so a company can check whether an agent did what it was supposed to do and what that did to the business. “There’s more in this area that will come in the next few months.”
The post Microsoft Fabric is where AI agents learn how the business works appeared first on The New Stack.
If you rely too heavily on one AI provider, switching can mean rebuilding workloads and moving data while you face untangling integrations. A new vendor-neutral group aims to help organizations keep their options open. That’s the goal of the Eclipse Foundation, which announced the formation of the Sovereign AI Foundation on Wednesday.
The new body will serve as a vendor-neutral initiative to help member organizations manage AI platform lock-in, monitor cyber risk, and maintain control of their data. The group has been active on Eclipse’s project site since July.
An initial community of 17 organizations comes together to advance AI sovereignty through open-source collaboration.
The foundation’s mission is to help organizations “navigate AI dependencies” (as in dependence on any single AI service, not control over where application dependencies get tied to a particular model’s DNA… although, by extension, that too), where dependence can limit a software engineering team’s ability to control where data is processed, choose which technologies to use, or change direction as costs, requirements, and provider terms evolve.
Executive director at the Eclipse Foundation, Mike Milinkovich, tells The New Stack that the risk of not adopting an open approach to AI sovereignty control depends on how deeply AI is embedded in an organization and how difficult the underlying systems are to change.
“The deeper the dependency, the greater the potential risk,” Milinkovich says. “But however embedded the underlying systems are, the costs show up in predictable places: higher prices that make switching difficult, and the challenge of rebuilding integrations, moving data and workloads, documenting systems after the fact, compliance delays, and operational disruption.”
He underlines his point and says part of the Sovereign AI Foundation’s work will be to “compare real-world experiences” and build stronger evidence around those risks.
“The deeper the dependency, the greater the potential risk. But however embedded the underlying systems are, the costs show up in predictable places.”
The Sovereign AI Foundation will work to give members an open forum to understand AI dependencies, compare experiences, and develop practical guidance for decisions about data, models, infrastructure, and operations.
Acknowledging what he calls the pre-IPO hype cycle around the promotion of frontier models, Milinkovich calls for a concerted focus on where the real value in AI lies: enterprise and industrial applications that deliver measurable business value, without vendor lock-in.
“AI will not deliver on its potential if it is based entirely on a dependence on a relatively small number of providers for critical AI models, platforms, and infrastructure,” Milinkovich insists. “As AI becomes part of the systems organizations rely on every day, that dependence can leave them with too little visibility into where their data goes, which models use it, or how easily they can switch when costs, terms, or requirements change.”
He asks us to understand how far the Sovereign AI Foundation will go to find practical ways to “preserve choice and control,” with a vision of delivering open-source and open-weight solutions that allow freedom of action.
As admirable and philanthropic as this sounds — and coming from a Brussels-based foundation whose founding members are mostly European — is there any America-first idealism here, given that open-weight Chinese models are also openly available?
“This is not about China or any single provider. The same questions apply to AI technologies from the United States, Europe, Asia, or anywhere else,” confirms Milinkovich. “Closed models can limit visibility, while access to model weights alone does not provide the transparency, control, or the independent governance that organizations may need. The real question for our members is whether they can understand, adapt, and, if necessary, replace the systems they depend on.”
As noted above, this initiative launches with 17 participating organizations spanning technology, industry, research, and the open source community: France’s CEA LIST, Thales and Kentyou, Germany’s EclipseSource, Open Elements, TypeFox, Vector Informatik, and Bosch, Italy’s Engineering Ingegneria Informatica and Eurotech, Sweden’s Ericsson, India’s Infosys, Belgium’s KU Leuven, Japan’s Renesas Electronics, Malaysia’s The IO Foundation… and then there’s North Carolina’s Red Hat, and the University of York in the UK.
The Sovereign AI Foundation focuses on five areas:
There’s a lot to unpack here in an organization only just out of the blocks; it begs the question whether the Sovereign AI Foundation will focus on data control challenges and efficiency, cyber safety, or platform lock-in challenges.
“It’s all three, and they reinforce one another,” Milinkovich says. “It is an efficiency issue if moving to a better or less expensive model requires rebuilding the workload. It is a security issue if organizations lack the visibility needed to assess risk. And it is a lock-in and resilience issue if critical operations become so tied to one provider that changing direction is costly or disruptive.”
“The need here is clearest wherever AI is used in systems with long lifecycles, demanding safety and security requirements, or sensitive data.”
He explains that “the specific needs vary”, so a manufacturer may need AI to keep operating when connectivity is limited, while a public agency may need to explain how an AI-supported decision was made.
“The need here is clearest wherever AI is used in systems with long lifecycles, demanding safety and security requirements, or sensitive data. That includes automotive, manufacturing, telecommunications, aerospace and defense, energy, and the public sector,” adds Milinkovich.
He confirms that the foundation’s initial participants do “reflect that mix”, as companies including Bosch, Vector, Renesas, Ericsson, and Thales put sovereignty high on the agenda. But he qualifies that dependence on critical AI technologies is a global concern, “so we expect interest well beyond Europe” in the weeks and months ahead.
Milinkovich offers a selection of still-forming use cases that he thinks are already solid.
“Across these examples, the common approach is modularity: keeping models, infrastructure, and applications separate enough that one can change without rebuilding everything. Our job is to capture what works and turn it into guidance others can use,” affirmed Milinkovich.
The Sovereign AI Foundation team says that alongside the 17 founding members, additional organizations are in the process of joining, and it invites eligible Eclipse Foundation member organizations to participate.
The post Eclipse wants companies to be free to switch AI providers. Today, doing so can mean a costly rebuild. appeared first on The New Stack.
Parallel work, model-family isolation, reversible changes, and GPU-backed validation shaped an open source project designed around coding agents NVIDIA TensorRT...Parallel work, model-family isolation, reversible changes, and GPU-backed validation shaped an open source project designed around coding agents NVIDIA TensorRT Model Connect is an open source collection of AI model reference implementations in C++, built on top of NVIDIA TensorRT. It began with a practical question: could the performance of the NVIDIA inference stack be made accessible to…
Fireworks Research launched Ember-1 on September 23 as a research preview. Built on Moonshot’s open-weight model, Kimi K3, it claims it matches Kimi K3’s quality using roughly 40% fewer tokens. Fireworks says it “learned to cut unnecessary reasoning while keeping the thinking that matters.” Reasoning tokens bill as output, so a model that thinks less should cost less.
There’s a catch, though. On OpenRouter, Ember costs $3 per million input tokens and $15 per million output tokens. Fireworks charges the same for Kimi K3, but other providers sell Kimi for as little as $1 per million input tokens and $9 per million output tokens.
I ran both models through OpenRouter on Fireworks, so both were billed at $3 and $15, and I also calculated what Kimi would have cost at the lower price.
It doesn’t matter how much cheaper a model is if it isn’t accurate. I wanted to know whether Ember’s shorter thinking holds up. If it cuts reasoning it actually needs, accuracy should slip first on the hardest problems. So I built tests that get progressively harder and ran each one five times, the same consistency check I’ve been adding to my recent testing.
I called both models through OpenRouter with identical prompts and their default reasoning settings. I routed both to Fireworks to keep the speed comparison fair. Each test ran five times per model, and I logged reasoning tokens separately from the rest of the output.
I confirmed every answer key with two independent methods before either model saw it. I’ve included the prompts at the end of this article for anyone who wants to replicate these tests on their system.
Both models solved all three puzzles on all five runs.
Ember averaged 13,630 reasoning tokens, 3 minutes 46 seconds, and $0.27 per run. Kimi averaged 16,679 reasoning tokens, 12 minutes 26 seconds, and $0.34. That’s 18% fewer reasoning tokens for Ember. Though not included in the marketing claims, Ember was much faster than Kimi. Kimi’s slowest run took nearly 20 minutes.
Both models found the 17-hour schedule every time, and every schedule passed my checker.
Ember averaged 6,543 reasoning tokens, 1 minute 29 seconds, and $0.10. Kimi averaged 7,792 reasoning tokens, 4 minutes 46 seconds, and $0.13. That’s 16% fewer reasoning tokens and significantly faster speed.
This test produced the only miss. Kimi answered all five questions correctly on every run. Ember got them all right four times. On the fifth, it made a small arithmetic slip on the first question, 0.94619 instead of 0.94629, and that error carried into two other answers.
It also saved the most. It averaged 6,242 reasoning tokens, 1 minute 47 seconds, and $0.13. Kimi averaged 9,682 reasoning tokens, 6 minutes 48 seconds, and $0.19. This was the first time Ember came close to its marketing claim of 40% fewer tokens. Once again, Ember was significantly faster.
| Test (5 runs each) | Ember-1 | Kimi K3 |
| Logic puzzles | 5/5 perfect, 3:46, 13,630 reasoning / 17,766 total out, $0.27 | 5/5 perfect, 12:26, 16,679 reasoning / 22,553 total out, $0.34 |
| Deploy scheduling | 5/5 perfect, 1:29, 6,543 reasoning / 6,822 total out, $0.10 | 5/5 perfect, 4:46, 7,792 reasoning / 8,365 total out, $0.13 |
| Probability | 4/5 perfect, 1:47, 6,242 reasoning / 8,365 total out, $0.13 | 5/5 perfect, 6:48, 9,682 reasoning / 12,381 total out, $0.19 |
| Perfect runs | 14 of 15 | 15 of 15 |
| Total cost, both on Fireworks ($3/$15) | $2.48 | $3.26 |
| Total cost, Kimi at cheapest price ($1/$9) | $2.48 (no cheaper provider) | $1.96 |
Kimi K3 had 15 perfect runs out of 15, and Ember-1 had 14. Ember-1’s one miss was an arithmetic slip on the probability test; it doesn’t seem too significant to me since it was one out of 15.
Something I noticed during my testing that wasn’t in the marketing claims was that Ember-1 finished each set of tests 3.4 times faster than Kimi K3. As for the reduced reasoning token usage, Ember-1 used 23% fewer reasoning tokens. This helped keep costs down. In total, it cost $2.48 compared to Kimi K3’s $3.26 on Fireworks, which makes it 24% cheaper.
One caveat is that Kimi K3’s cost depends on the provider. At its lowest listed price of $1 per million input tokens and $9 per million output tokens, the same runs would have cost $1.96, less than Ember-1.
Ember-1 is about as accurate as Kimi K3 and uses far fewer reasoning tokens. The lower costs on Ember-1 on this test aren’t as important because a user who wants to use Kimi K3 can route to a cheaper provider on OpenRouter, which brings the price down below Ember-1’s.
One thing I know about Kimi K3 is that it’s slow. Cheap and slow. My speed numbers come from Fireworks’ standard endpoint, though, and I didn’t test the cheaper providers. If you have time and want to pay less, Kimi K3 wins. If you want almost identical results to Kimi K3 at a much faster rate, use Ember-1.
The prompts
Each prompt ends with a fixed answer format so grading can be automatic.
Logic puzzles
Solve all three logic puzzles below. Each has exactly one solution.
PUZZLE SMALL: 4 engineers (Ava, Bo, Cleo, Dev) each own exactly one server. Each server has a rack position (1, 2, 3, 4, numbered left to right), an operating system (Debian, Alpine, Fedora, Ubuntu), a role (cache, queue, proxy, db). No two servers share any of these values.
Clues:
If the server in rack 3 runs Fedora, then the Debian server is in rack 3.
The server in rack 1 is Dev’s.
Exactly one of these is true: the server in rack 4 runs Ubuntu, or the Alpine server is not Bo’s.
Cleo is the proxy.
The db server is Dev’s.
Ava runs Alpine.
The queue server is not Ava’s.
Bo is in rack 3.
Bo and the Ubuntu server are in neighboring racks.
PUZZLE MEDIUM: 5 engineers (Ava, Bo, Cleo, Dev, Eun) each own exactly one server. Each server has a rack position (1, 2, 3, 4, 5, numbered left to right), an operating system (Debian, Alpine, Fedora, Ubuntu, Arch), a role (cache, queue, proxy, db, build), a replication data center (Oslo, Lima, Pune, Accra, Perth). No two servers share any of these values.
Clues:
Cleo is the db.
Exactly one of these is true: the cache server runs Arch, or the server replicated to Perth is the proxy.
The server in rack 5 runs Fedora.
Exactly one of these is true: the server in rack 2 is not the proxy, or Cleo runs Debian.
Exactly one of these is true: the server replicated to Pune is the db, or Dev is in rack 3.
The server in rack 1 is the db.
The proxy server is in rack 5.
The server in rack 4 runs Ubuntu.
The cache server and the server replicated to Pune are in neighboring racks.
Ava is the cache.
The server replicated to Accra is the queue.
The server replicated to Lima is not in rack 3.
Eun runs Alpine.
The queue server is not in rack 2.
The Fedora server is not Dev’s.
The server in rack 2 is not replicated to Lima.
PUZZLE LARGE: 7 engineers (Ava, Bo, Cleo, Dev, Eun, Finn, Gia) each own exactly one server. Each server has a rack position (1, 2, 3, 4, 5, 6, 7, numbered left to right), an operating system (Debian, Alpine, Fedora, Ubuntu, Arch, Rocky, NixOS), a role (cache, queue, proxy, db, build, metrics, auth), a replication data center (Oslo, Lima, Pune, Accra, Perth, Quito, Riga). No two servers share any of these values.
Clues:
The build server is Cleo’s.
The proxy server does not run Fedora.
The server replicated to Quito is Finn’s.
The server in rack 6 is replicated to Lima.
The Alpine server is exactly 4 racks to the right of the auth server.
The server replicated to Oslo is Dev’s.
The Debian server is replicated to Quito.
The server replicated to Accra is not the build.
The Rocky server is exactly 1 rack to the right of Gia.
Exactly one of these is true: the Ubuntu server and the server replicated to Pune are in neighboring racks, or the server replicated to Oslo runs Fedora.
The server replicated to Oslo does not run Fedora.
Ava runs Rocky.
Exactly one of these is true: the proxy server is somewhere to the left of the server replicated to Riga, or the Debian server is in rack 7.
The auth server and Bo are in neighboring racks.
The metrics server is Gia’s.
The Fedora server is somewhere to the left of the server replicated to Pune.
Exactly one of these is true: the server replicated to Accra is in rack 2, or Finn is the queue.
If the Debian server is Finn’s, then the server in rack 7 does not run Rocky.
The NixOS server is not replicated to Perth.
The server in rack 7 is replicated to Accra.
The server replicated to Pune is the db.
At the end of your response, give each solution as a block, one line per engineer, in the order the engineers are listed in that puzzle:
SMALL:
<name> | <rack> | <os> | <role>
MEDIUM:
<name> | <rack> | <os> | <role> | <dc>
LARGE:
<name> | <rack> | <os> | <role> | <dc>
Deploy scheduling
You are planning a production rollout of 12 services. Time is measured in whole hours from hour 0.
Services (owning team, deploy duration in hours):
auth: team A, 3 hours
billing: team B, 4 hours
catalog: team C, 2 hours
search: team C, 3 hours
cart: team B, 2 hours
checkout: team B, 3 hours
payments: team A, 4 hours
notify: team C, 2 hours
ledger: team A, 2 hours
gateway: team A, 3 hours
reports: team B, 3 hours
inventory: team C, 4 hours
Rules:
Dependencies. A service may start deploying only after every service it depends on has finished deploying:
gateway depends on auth
payments depends on auth
search depends on catalog
cart depends on catalog
cart depends on inventory
checkout depends on cart
checkout depends on payments
ledger depends on billing
ledger depends on payments
notify depends on checkout
reports depends on ledger
gateway depends on search
notify depends on gateway
Each team can deploy only one of its services at a time.
Different teams can deploy at the same time.
Blackout windows. No deploy may be in progress at any time during hours 9 to 11 or hours 17 to 19. A deploy may end exactly at hour 9 or 17, and may start exactly at hour 11 or 19. A deploy cannot pause and resume.
Each deploy runs from its start hour for its full duration without interruption.
What is the earliest hour by which all 12 services can be finished? Give a schedule that achieves it.
At the end of your response, give your answer in exactly this format:
MAKESPAN: <hour>
<service>: <start hour>
(one line per service, all 12 services)
Probability
A client calls a payment API and retries on failure.
The server is in one of two states on each attempt, Healthy or Degraded.
On the first attempt, the server is Healthy with probability 4/5 and Degraded with probability 1/5.
Between consecutive attempts, the state changes like this: from Healthy, it stays Healthy with probability 3/4 and becomes Degraded with probability 1/4. From Degraded, it stays Degraded with probability 2/3 and becomes Healthy with probability 1/3.
An attempt succeeds with probability 9/10 if the server is Healthy and 2/5 if it is Degraded, independently of everything else given the state.
The client makes at most 4 attempts and stops as soon as one succeeds.
Circuit breaker: if two consecutive attempts both hit a Degraded server and both fail, the client stops immediately and makes no more attempts.
Answer these questions with exact fractions in lowest terms:
Q1. What is the probability that the request eventually succeeds?
Q2. What is the expected number of attempts the client makes?
Q3. Given that the request succeeds, what is the probability that it succeeded on exactly the second attempt?
Q4. What is the probability that the circuit breaker ends the request early, before the client has used all 4 attempts?
Q5. Given that the request succeeds, what is the probability that the first attempt failed?
At the end of your response, give exactly five lines in this format:
Q1: <fraction>
Q2: <fraction>
Q3: <fraction>
Q4: <fraction>
Q5: <fraction>
The post Ember-1 vs. Kimi K3: Nearly identical results at 3.4 times the speed appeared first on The New Stack.
Radiology AI has made remarkable strides in detecting abnormalities across chest X-rays, pathology slides, and 2D scans. Yet one of the most clinically rich and...Radiology AI has made remarkable strides in detecting abnormalities across chest X-rays, pathology slides, and 2D scans. Yet one of the most clinically rich and data-dense modalities—the 3D computed tomography (CT) scan—remains largely underserved by modern vision language models (VLMs). Frontier general-purpose models perform poorly on volumetric imaging, and most open medical AI models lack the…
Anthropic wants developers to use Claude and its family of tools to handle complete coding tasks. The company introduced Claude Opus 5.5 on Tuesday to span more of the software application development lifecycle: from design specification creation through debugging to code generation and testing.
As the first release in a new family of Claude 5.5 models, Anthropic says that Claude Opus 5.5 performs at the level of Claude Fable 5.1 on “most work” tasks and costs around 40% less to run than Opus 5.
GitHub chief product officer Mario Rodriguez is quoted in Anthropic’s launch announcement on exactly where software engineers sit today with frontier models for code automation. He said that developers want agents.
“In our testing [of Claude Opus 5.5] across GitHub Copilot CLI and VS Code, Claude Opus 5.5 used among the fewest tokens and steps we measured. In VS Code, it solved more terminal tasks than Opus 5 in less than half the steps. More than making individual tasks more efficient, it’s making developers’ bigger projects more achievable,” says Rodriguez.
Anthropic explains that the Claude 5.5 family’s expanded full-lifecycle capabilities were developed under an established set of practices.
These practice elements include extensive alignment testing (model evaluation processes put in place to make sure actions and outputs closely match human values and intended goals), pre-release evaluation by outside organizations, and safeguards for high-risk areas such as cybersecurity and biology.
The organization says that on its most comprehensive alignment test, Opus 5.5 is the strongest-performing model tested to date, with particular improvements in several behaviors that contributed to recent cybersecurity incidents (e.g., biased reasoning, attempting to escape a sandbox, and others).
Independent SRE & AI reliability architect Akash Thakur tells The New Stack that Anthropic’s work getting its model to complete whole coding tasks is impressive, but “getting it to know when it hasn’t” is the harder problem developers need to think about.
“Models working at the level of Claude Opus 5.5 are genuinely good at breaking a project into small, finishable pieces — and that’s the real unlock, because momentum on software comes from finishing things, not starting them,” Thakur says. “…But ‘completed’ and ‘correct’ aren’t the same thing. The task that looks done and completed is exactly the one that often costs the team later, so the win isn’t removing the human — it’s moving them from writing the code to verifying it.”
“Models working at the level of Claude Opus 5.5 are genuinely good at breaking a project into small, finishable pieces …but ‘completed’ and ‘correct’ aren’t the same thing.”
Suggesting that we are now witnessing a “higher bar for more capable models”, Anthropic said that models that could fully automate AI research itself should meet a higher safety standard. The company noted that as AI becomes more capable, public policy should play a larger role in making sure these systems are safe. Its recent work with Accenture is offered as an example of how Anthropic is building the infrastructure to support this.
Getting more specific, Anthropic has claimed Opus 5.5 is “particularly good” at long, sprawling jobs like codebase-wide migrations and audits. An early tester said it audited and fixed a 200,000-line codebase in under three hours, while Opus 5 took over 20 hours and used 2.5x as many tokens.
“In an internal test, we asked Opus 5.5 and Fable 5.1 to translate HAProxy, widely used software that balances web traffic loads across servers, from C into Rust. Both rewrites passed nearly all of HAProxy’s own regression tests, but Opus 5.5 finished in 9.5 hours compared to 12 for Fable 5.1, and cost 51% less,” said Anthropic in a press statement.
Field CTO for the EMEA region at Coder, Eric Paulsen, tells The New Stack that he is happy to hear Claude Opus 5.5 is becoming more holistically capable, but he’s “not surprised” because AI is “eating the software delivery chain end to end” today.
“Despite the efficiencies showcased in Opus 5.5, the moment an agent can reliably finish real work, the question stops being whether the model is good enough and becomes where you’re letting it run,” Paulsen says.
“Kicking off a Claude Code session on a laptop that can be compromised, stolen, or simply run out of compute is not an engineering environment; capable agents need dedicated, governed infrastructure with the same guardrails, secrets handling, and observability a developer would demand of any other production workload. As we’ve seen here with Anthropic’s own work, firms need to underline testing and safeguard procedures for launches of this kind,” he added.
Anthropic assures users that external testing has been conducted and that the model was tested before release by Frontier Design and METR. When established safeguards for Opus 5.5 intervene, requests “fall back to another model transparently,” meaning developers may not see which model actually handled a given call.
In cybersecurity, users can identify and fix bugs in their code, but most cybersecurity tasks will be re-routed to Opus 4.8. Requests flagged by biology and frontier LLM development classifiers will be re-routed to Opus 5. Vetted organizations can apply to Anthropic’s Life Sciences Verification Program to use Opus 5.5 for biology research, and the company says it will expand its Cyber Verification Program in the coming weeks.
HasData co-founder, Sergey Ermakovich, tells The New Stack that his work as a web scraping specialist for data pipelines and AI means he sees zen-like, one-brick-at-a-time logic in what Anthropic has done.
“The biggest change — and it’s a trend that will have driven Anthropic’s design and development aspirations for Claude Opus 5.5 — is that the terminal is no longer just a place where a developer pastes generated code,” Ermakovich says. “The terminal now becomes part of the model’s workspace.”
Because a model can run commands, inspect failures, modify files, and verify the result, Ermakovich suggests it can “close the loop” instead of handing unfinished work back to an engineer. “That makes small complete tasks much more valuable. Fix one failing test, update one dependency, migrate one endpoint, verify it, then move to the next task,” he adds.
“The terminal is no longer just a place where a developer pastes generated code — the terminal now becomes part of the model’s workspace.”
Commenting on the development of this model as part of Anthropic’s approved corporate messaging, John Ruelas, staff software engineer at Ramp, said that verbose, hard-to-follow output has been his “biggest frustration” with frontier models, but “Claude Opus 5.5 fixes it” for him.
He noted that it writes like a good colleague and follows his company’s writing rules. A design spec came out usable with very minimal edits, and when it rewrote one of his prompts, he preferred its version to his own. When it optimized the team’s test suite, he could follow its reasoning easily and “shipped the change with confidence.”
Vice president of AI strategy at Abbyy, Maxime Vermeir, tells The New Stack that the more lifecycle-wide Claude Opus 5.5 features on offer show that “we’re done with the autocomplete era” for basic code automation tools.
“Anthropic’s elevation here reflects the fact that it used to be thought of as marvelous if AI could complete a developer’s next line of code, but today the expectation is that you hand it a whole Jira ticket and that it gets done,” Vermeir says. “But despite the power on show with Claude Opus 5.5, the question remains as to how much these newer models will actually understand what ‘done’ means, as often it seems they have a desire to keep burning tokens by offering you yet another thing it didn’t quite do right.”
Opus 5.5 also “communicates more naturally” than prior models, with early testers finding its writing clearer and easier to follow, making it a better work partner over long sessions.
Costed out lower than Opus 5, Opus 5.5 is priced at $4 per million input tokens and $20 per million output tokens (20% less than Opus 5), and Anthropic has cut cache read prices by 60% for token-billed usage. Opus 5.5 also needs fewer tokens for higher-quality work and generates output more than 30% faster than Opus 5.
The post Claude Opus 5.5 wants to finish your coding tasks, not just start them appeared first on The New Stack.
AI agents spend a ridiculous amount of compute generating text nobody actually needs. The decisions an agent makes along the way don’t require a written answer and yet, agents still send them to generative models, wait for an answer while burning through tokens and then parse that output back. The overhead is already drawing scrutiny — OpenAI’s own researchers recently disclosed spending $7,000 a day running agent workloads.
Kev, a new family of open decision models built on Qwen 3.5, takes a different approach and skips the generation entirely.
Developer Jared Palmer released a new generation of Kev on Sunday, with 0.8 billion, 4 billion, and 9 billion parameter models built on Qwen 3.5. Kev is prefill-only, processing the state, questions, and candidates in a single forward pass before reading the decisions from a pointer head without an autoregressive decoding loop.
Kev, a new family of open decision models built on Qwen 3.5, takes a different approach and skips the generation entirely.
Kev supports three decision types: Noul for yes/no, Choice for selecting among candidates, and Score for ordered levels, mirroring TypeSafe’s System One API. Developers provide the state and questions, and the pointer head returns probabilities across the available candidates.
For a tool-routing decision, the output could look like this:
search: 0.82
database: 0.13
calculator: 0.05
Kev can still choose the wrong tool, but because it scores only the candidates it’s given, it can’t introduce an option that isn’t on the list.
Routing, safety checks, escalation, and ranking can then move to the decision layer, leaving larger reasoning models to handle the open-ended work.
Kev can still choose the wrong tool, but because it scores only the candidates it’s given, it can’t introduce an option that isn’t on the list.
Multiple decisions can also be made against the same state in a single forward pass, with a block-causal attention mask isolating the questions while the pointer head scores each set of candidates independently.
Palmer’s documentation shows the 4B model processing three questions in 277 milliseconds in bf16 on an M5, although without a controlled comparison against Qwen generating equivalent answers on the same hardware, the result doesn’t establish how much faster the approach is in practice.
The ability to evaluate several decisions against the same context could become more useful as agent loops grow more complex, but skipping generation doesn’t make the resulting decisions inherently better.
The largest model, Kev-9B, reached 83.7% accuracy on the project’s locked out-of-domain test, according to Palmer’s model card. That’s a developer-reported benchmark, and Palmer documents some limitations alongside it.
The probabilities Kev returns don’t always reflect how confident developers should be in the result. Palmer found that temperature calibration can drift on unseen source distributions, a problem for agents that use probability thresholds to decide whether to execute an action or escalate it, since even a high-probability choice can still be wrong.
Fine-tuning also changes some of the capabilities inherited from the underlying model. Palmer’s evaluations show declines on general-knowledge and arithmetic tests, particularly among the smaller models. That’s consistent with Kev’s more specialized role alongside a general-purpose model, although its performance in dynamic agent environments will also depend on how well it handles tools, choices, and labels it never encountered during training — and debugging agent failures often points to infrastructure rather than the model itself.
The approach predates Kev. TypeSafe introduced Jev earlier this month as part of its System One platform, using the same Noul, Choice and Score primitives, and Kev implements its /v1/systemone request and response format so applications built against the API can point to a local Kev server instead.
The biggest difference is that Palmer released Kev under Apache 2.0 with the model weights, training code, and evaluation tooling, giving developers the option to run and train it on their own infrastructure. Jev’s weights and training data aren’t public, however, which makes direct performance comparisons difficult because differences between the models can’t be isolated to architecture, size, or training.
For applications that make only a handful of bounded decisions, constrained decoding on a model that’s already running may be simpler than adding another model to the stack. Agent loops can make those decisions constantly, however, moving through routing, ranking, safety checks, tool selection, and escalation before generating much user-facing text. It’s a pattern showing up across model architectures — stripping out unnecessary computation when the task doesn’t require it.
When those steps only require a choice or probability, Kev can handle the decision directly while leaving open-ended reasoning and final responses to the larger generative model.
When those steps only require a choice or probability, Kev can handle the decision directly while leaving open-ended reasoning and final responses to the larger generative model.
The post Your AI agent is burning tokens on choices that don’t need words appeared first on The New Stack.
The 1.58 in a 1.58-bit language model sounds like a hard limit, but Intel researchers pushed a ternary model below it by changing how its weights are stored rather than changing the model itself.
Their new BITCOS format compressed one checkpoint to 1.485 bits per weight and improved decoding throughput by as much as 18% on CPUs and 27% on GPUs.
The key is that the familiar 1.58-bit figure assumes a model uses its three possible weight values equally, while real ternary models contain far more zeros than that calculation accounts for. BITCOS stores the location and sign of each nonzero weight separately, allowing zeros to take up less space without retraining the model or altering its output — the equivalent of packing the same contents into a smaller box.
Ternary models use only three weight values — -1, 0, and +1 — and 1.58 bits is the theoretical minimum needed to represent three equally likely options. That number is cleaner than the reality of storing the weights, where the standard approach fits five ternary values into an eight-bit byte for an average of 1.6 bits each. Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.
When Intel’s researchers measured the distribution of weights across 29 checkpoints from seven ternary model families, they found that zeros accounted for between 29.7% and 51.5% of the weights. In 26 of those checkpoints, there were enough zeros for BITCOS to beat five-trit packing.
The sparsest was a ternary version of Qwen3-1.7B produced with CAT-Q post-training quantization, where 51.48% of the weights were zero, and BITCOS brought the storage cost down to 1.485 bits per weight.
Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.
BITCOS stands for “BITmap and COmpacted Signs” and divides a model’s weights into two streams. The first assigns one bit to every weight to record whether it is zero or nonzero, while the second assigns a sign bit only to nonzero weights.
A positive or negative weight therefore consumes two bits, but a zero needs only the presence bit because it has no sign to record.
If z is the proportion of zero weights, BITCOS uses 2 − z bits per weight, dropping from 1.6 bits at 40% zeros to 1.485 bits at 51.5%. Because it changes only the storage format, unpacking restores the original -1, 0 and +1 values without affecting accuracy.
BITCOS becomes smaller than five-trit packing once more than 37.5% of a model’s weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.
BITCOS becomes smaller than five-trit packing once more than 37.5% of a model’s weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.
Built for token-by-token decoding with small batch sizes, the format reduces the weight data moving through memory. Intel developed separate unpacking kernels for AVX-512 and AVX2 CPUs as well as Xe2 GPUs, joining other efforts to fit compressed models into faster inference pipelines for AI agents.
On AVX-512 hardware, the kernel uses the presence bitmap as a mask and pdep to scatter the compacted sign bits across the nonzero weight positions. Because Xe2 GPUs lack an equivalent instruction, Intel implemented the same operation with a 2KB lookup table.
Compared with the 2-bit kernels, BITCOS ran 10% to 18% faster on the 64-core Xeon server and 2% to 15% faster on the 24-core Core Ultra 9. Performance improved by 9% to 22% on the integrated Arc 140V and by 2% to 27% on the discrete Arc Pro B70. These results measure decoding after the model has loaded, separate from efforts to cut GPU inference cold starts from minutes to seconds.
On the eight-core Lunar Lake CPU, Intel’s fixed 2-bit kernel beat BITCOS on every model because the system had enough bandwidth to make unpacking the bottleneck. BITCOS remained faster on the GPUs, although decoding overhead limited the gains. Computer scientist and AI infrastructure author Chip Huyen has made the same point about inference more generally, arguing that the right optimization depends on whether compute, memory or bandwidth is holding back the workload.
On the eight-core Lunar Lake CPU, Intel’s fixed 2-bit kernel beat BITCOS on every model because the system had enough bandwidth to make unpacking the bottleneck.
The paper has not been peer-reviewed; all five test systems used Intel hardware, and the end-to-end benchmarks covered seven models at batch size one. Intel has yet to test the format on Nvidia, AMD, or Arm hardware.
The post Intel squeezed a 1.58-bit LLM down to 1.485 bits without changing a single weight appeared first on The New Stack.
Mixture of experts (MoE) has become one of the defining architectural trends in large-scale AI model training. DeepSeek, Qwen, and Mixtral are examples of MoE...Mixture of experts (MoE) has become one of the defining architectural trends in large-scale AI model training. DeepSeek, Qwen, and Mixtral are examples of MoE models that match or exceed the performance of dense model counterparts at a fraction of the training compute. MoE models provide efficient training through conditional computation. Instead of one dense feed-forward network (FFN) shared…
Everyone knows the open-weight model pitch by now: companies can download the weights, customize them, run them on infrastructure of their choosing, and retain far greater control over where their data is processed — often at a much lower cost than using proprietary models.
Moreover, open-weight models are now thought to trail the leading frontier models by only around four to five months. Nvidia, the world’s most valuable company, is betting heavily on that future. In early September, it agreed to acquire Hugging Face — the sprawling “GitHub for AI” that hosts more than three million models — for $12.9 billion, while pledging to keep the platform open to different models, clouds and computing providers. And on Thursday, Nvidia detailed how Nvidia is using its own open-weight Nemotron model to manage its vast global supply chain in partnership with Palantir.
That power also comes with serious security questions. OpenAI president Greg Brockman recently warned that increasingly capable open-weight models — pointing specifically to China’s GLM-5.3 — could “significantly accelerate the threat landscape” as models with advanced cyber capabilities become freely downloadable and modifiable.
But for businesses accessing those models through third-party services, there is another concern closer to home: where their own data goes when they use those models, particularly when the model originated in China.
Hugging Face data from February showed models from Chinese developers accounted for 41% of downloads in the preceding 12 months, ahead of the US at 36.5%. Over on OpenRouter, meanwhile, open-weight models now account for around 60% of tokens consumed by US-originating requests, with the company noting that Chinese models constitute the majority.
And that’s why OpenRouter is now giving companies a way to put a geographic fence around that traffic. The AI model marketplace has officially launched US in-region routing into general availability for business and enterprise customers, promising that requests sent through its US endpoint are decrypted, processed and served entirely inside the country — or rejected if that can’t be done.
The feature itself had been quietly available in some form before now, with OpenRouter updating its documentation in early August to say US in-region routing was available to enterprise customers by request. It’s also worth noting that this is in addition to European in-region routing, which it says has been available since October 2025.
Started in early 2023 by former OpenSea CTO Alex Atallah, OpenRouter serves as an interface to the crowded AI model market, with developers able to switch between hundreds of models from myriad providers via a single API. Payments giant Stripe recently announced plans to acquire the company in a reported $8 billion deal, while a slew of other companies including Cursor, Ramp, and Meta, are also building their own model routers.
The reason why model routers are such hot property right now is largely down to economics. Developers have traditionally hard-coded applications to send everything to the same model, while a model router can instead make that choice request by request, sending easier jobs to cheaper models while reserving the pricier frontier systems for the work that actually needs them.
That intermediary role is also what makes OpenRouter’s new residency controls possible: it already decides which provider serves each request, and can now restrict that choice to provider endpoints operating in the US.
In a blog post announcing the new feature on Wednesday, Cailee Moberg, who works on OpenRouter’s product team, notes that while US-developed models from Nvidia and Thinking Machines are contributing to the broader open-weight model boom, Chinese models dominate usage and raise tough questions for companies concerned about their data.
“Models from Chinese labs are still most of the [open-weight model] volume, and procurement approval for those models can be difficult.”
“Models from Chinese labs are still most of the [open-weight model] volume, and procurement approval for those models can be difficult,” Moberg writes.
In its 2026 State of AI in the Enterprise report, Deloitte concluded that sovereign AI was on the rise, noting that 77% of companies “now factor country of origin into their vendor selection,” while nearly 60% construct their AI stacks “primarily with local vendors.”
And this at least partly explains why OpenRouter is now offering in-region routing for US customers. Moberg points to DeepSeek V4 Pro, Kimi K3 and GLM 5.2 as specific examples. All three are available through US In-Region Routing because Baseten, Fireworks and Azure serve them from US data centers. Companies could already keep these models inside the US by self-hosting them or using a US provider directly; OpenRouter’s new routing gives its own customers that residency guarantee without having to manage those deployments themselves.
OpenRouter maintains a live list of models eligible for US in-region routing, ranging from proprietary frontier models from OpenAI and Anthropic to open-weight models from the major Chinese labs.
“In-Region Routing allows teams with data residency requirements to get the price and performance gains from Chinese open-weight models,” Moberg continues. “When a US or EU provider hosts a model, requests go to that provider and the lab is not involved.”
“In-Region Routing allows teams with data residency requirements to get the price and performance gains from Chinese open-weight models.”
The technical change happens at the routing layer. With OpenRouter’s standard global endpoint, a request can be served by an eligible provider operating in any region, so even using a model from a US company does not guarantee that the request itself is processed in the US. With us.openrouter.ai, the request is decrypted on OpenRouter infrastructure inside the US and the pool of providers is filtered to endpoints OpenRouter has approved as operating there.
If no compliant US provider can serve the requested model, OpenRouter returns a 404 error. Companies can also enforce the regional restriction through OpenRouter’s Guardrails at the workspace, team or API-key level, while tools that would send prompt data outside the US are disabled on the regional endpoint.
So while none of this ultimately changes where the DeepSeek, Kimi or GLM models are developed, in-region routing alters which copies of those models its US customers can be routed to, and where their prompts are handled along the way.
The post Chinese AI models dominate OpenRouter’s US token consumption. It can now guarantee that traffic stays entirely in the US. appeared first on The New Stack.
Enterprise AI company Cohere announced North Small Translate last week, a mixture-of-experts (MOE) open-weight machine translation model that works across 50 languages.
Developers can download the weights for noncommercial use under CC BY-NC 4.0. Cohere offers commercially licensed deployment through Model Vault, which is a Cohere-managed inference environment. Cohere positions the model as part of its sovereign AI strategy, aimed at organizations that want greater control over where their models run and how their data is handled.
North Small Translate builds on Cohere’s multilingual and translation lineage, which includes its Tiny Aya and Command A Translate model families. The company claims North Small Translate outperforms “similarly sized open-weight models” under 1T parameters, as well as API-based translation models in various dimensions of machine translation on average.
Cohere co-founder Nick Frosst tells The New Stack that the model’s efficiency draws from the fact that it is non-reasoning, i.e., it relies on learned statistical patterns without a step-by-step logic process, which means it uses fewer tokens.
“We spent nine years scaling an architecture invented to fix translation, and machine translation is still broken for most of the world’s languages,” Frosst says. “General-purpose models get you most of the way and then stop. The next phase of enterprise AI in this space is smaller, more specialized, and runs inside your own walls.”
“…machine translation is still broken for most of the world’s languages.”
In Cohere’s reported evaluation using WMT26 benchmarks, the company states that North Small Translate leads with a WMT26 All Languages benchmark score of 83.60, compared with 81.56 for Qwen 3.5 397B A17B, 76.50 for GLM 5.2 FP8, 81.37 for DeepL NextGen, 79.46 for Gemma 4 31B (on), and 68.20 for Google Translate.
With its mixture-of-experts architecture and 218 billion total parameters, with 25 billion active. Cohere points to North Small Translate’s smaller compute & memory footprint than other models. Some model-to-model comparisons in this space aren’t fully substantiable, since not every vendor discloses parameter counts.
“Machine translation allows documents to be translated from one language to another automatically. With current solutions, long documents start to fall apart,” Frosst says. “Google Translate scores 21.3 on our long-context test, Gemma 4 31B 19.4; we score 48.9. That’s [for example] a safety manual that reads fine on page one… and has drifted by page ten. The other risk is where the text goes. Once you push HR policies or regulated documents through a third-party API, that data has left your building, and necessarily that means your control over it is diminished.”
“The risk [in machine translation] is where the text goes. Once you push HR policies or regulated documents through a third-party API, that data has left your building and necessarily that means your control over it is diminished.”
Explaining why the model offers “stronger translation performance” across complex enterprise translation tasks, Frosst says the model can support work spanning “a high volume” of sensitive documents.
As well as its 50 languages (32 ‘high-resource’ languages + 18 others), the Cohere team explains that the model also supports translation-workflow-focused capabilities, such as structured translations (i.e., Markdown or JSON documents), instruction following (i.e., recommended tone & format), and terminology guides (i.e., providing specific vocabulary to use in the translation), all as part of the model.
“North Small Translate works with a multi-pass workflow,” explains Frosst. “The model translates, reviews its own output, finds errors, and fixes them – and this is the same loop we used in training. We ship both because standard is one pass and built for volume, while the agentic [version] spends more tokens for 84.36 against 83.60 on WMT26. That difference ends up being worth it when the document is a contract or a safety procedure, for instance, but in other cases you’d rather optimize for efficiency.”
“The model translates, reviews its own output, finds errors and fixes them.”
This model uses the same architecture as prior Cohere models but improves performance through post-training advances, including reinforcement learning and new datasets, specifically for machine translation tasks.
Frosst concludes that, across the translation model marketplace, generative machine translation models offer the highest quality and steerability (i.e., suggesting tone, formatting, etc.) but typically cost much more than Neural Machine Translation (NMT) models commonly used in commercial use cases.
North Small Translate was developed in partnership with RWS, an AI solutions company pioneering in language technology and services. Collaboration with RWS, specifically with its Language Weaver research and science teams along with its language experts, helped shape the model’s real-world translation performance throughout development.
As noted above, developers can access the weights free of charge for non-commercial use in three quantizations. There is also a Hugging Face Space and an API for those who lack the required hardware.
The post “Machine translation is still broken for most of the world’s languages”: Cohere builds non-reasoning for a reason appeared first on The New Stack.
This week, Cohere released North Small Translate 1.0 under a CC BY-NC 4.0 license: the weights are there to download, evaluate and study, but not to run in production without a commercial agreement.
It’s an interesting choice from the Canadian foundation model company, which has built its pitch around AI sovereignty for regulated industries and describes this release as part of a mission “to make sovereign AI a technological reality.” Sovereignty there means control over where the model runs and who sees the data. A commercial license keeps that promise intact. It stops short of independence from Cohere. Enterprises keep their data and their infrastructure. They don’t get to fork the model, build a product on it, or keep running it if the terms change at renewal.
North Small Translate is an open-weights mixture-of-experts model built for machine translation across over 50 languages and locale variants. It has 218 billion total parameters, with 25 billion active parameters and a 16,000-token context window.
Not all users have the same access to those weights.
Per Cohere, the model is designed to give researchers, developers, and enterprises “flexible ways to evaluate and deploy machine translation while retaining control over their data and infrastructure.”
That’s an appealing description for organizations keen on pursuing sovereign AI. But the open-weight release comes with an important caveat: Not all users get the same rights to take advantage of those weights.
North Small Translate is available today on Cohere’s free tier through the Chat V2 API. For those who intend to use the model weights for non-commercial use, the FP8 weights are available on Hugging Face under the CC BY-NC 4.0 license.
But if enterprises want to put them into production, then a different set of terms applies. They’ll have to purchase a commercial license and deploy North Small Translate through Model Vault, Cohere’s fully managed inference platform.
Other AI companies are starting to attach more conditions to their open-weight models, too.
Last month, Chinese AI lab Z.ai released the weights for its flagship GLM-5.3 model on Hugging Face. But like the Canadian AI company, it also changed its licensing terms depending on who is deploying the model — a departure from its previous approach. While GLM-5.2 shipped under the permissive MIT license, GLM-5.3 adds new requirements for certain commercial users.
Cohere, for its part, has been similarly mum about why it made North Small Translate’s open weights noncommercial.
These requirements apply only to companies with aggregate revenue over $10 billion over 12 consecutive months. Additionally, if these companies want to host GLM-5.3 or its derivative works for commercial purposes, they have to first pass the Chinese lab’s security review.
Z.ai didn’t explicitly spell out why it decided to make such an about-face for GLM-5.3, which is especially puzzling given that its predecessor shipped under MIT without any commercial stipulations. Cohere, for its part, has been similarly mum about why it made North Small Translate’s open weights non-commercial.
The Canadian company’s decision to make North Small Translate available as open weights but gate commercial use is a head-scratcher, given its history of selling sovereign AI to enterprises.
In fact, in June, it pitched North Mini Code, its first coding model, as a response to developers demanding the same sovereignty guarantees that regulated industries have long required.
Unlike North Small Translate, though, this open-weight model was released under an Apache 2.0 license from the get-go — without any comparable restrictions for commercial users.
Clearly, Cohere is going in a different direction with its latest open-weight release, emerging as another example of AI companies putting tighter terms around increasingly capable open-weight models.
The post Cohere’s new translation model is open weights — but not for commercial use appeared first on The New Stack.
Login