The 1.58 in a 1.58-bit language model sounds like a hard limit, but Intel researchers pushed a ternary model below it by changing how its weights are stored rather than changing the model itself.

Their new BITCOS format compressed one checkpoint to 1.485 bits per weight and improved decoding throughput by as much as 18% on CPUs and 27% on GPUs.

The key is that the familiar 1.58-bit figure assumes a model uses its three possible weight values equally, while real ternary models contain far more zeros than that calculation accounts for. BITCOS stores the location and sign of each nonzero weight separately, allowing zeros to take up less space without retraining the model or altering its output — the equivalent of packing the same contents into a smaller box.

Where the 1.58-bit figure comes from

Ternary models use only three weight values — -1, 0, and +1 — and 1.58 bits is the theoretical minimum needed to represent three equally likely options. That number is cleaner than the reality of storing the weights, where the standard approach fits five ternary values into an eight-bit byte for an average of 1.6 bits each. Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.

When Intel’s researchers measured the distribution of weights across 29 checkpoints from seven ternary model families, they found that zeros accounted for between 29.7% and 51.5% of the weights. In 26 of those checkpoints, there were enough zeros for BITCOS to beat five-trit packing.

The sparsest was a ternary version of Qwen3-1.7B produced with CAT-Q post-training quantization, where 51.48% of the weights were zero, and BITCOS brought the storage cost down to 1.485 bits per weight.

Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.

How zeros save space

BITCOS stands for “BITmap and COmpacted Signs” and divides a model’s weights into two streams. The first assigns one bit to every weight to record whether it is zero or nonzero, while the second assigns a sign bit only to nonzero weights.

A positive or negative weight therefore consumes two bits, but a zero needs only the presence bit because it has no sign to record.

If z is the proportion of zero weights, BITCOS uses 2 − z bits per weight, dropping from 1.6 bits at 40% zeros to 1.485 bits at 51.5%. Because it changes only the storage format, unpacking restores the original -1, 0 and +1 values without affecting accuracy.

BITCOS becomes smaller than five-trit packing once more than 37.5% of a model’s weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.

BITCOS becomes smaller than five-trit packing once more than 37.5% of a model’s weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.

Making smaller weights run faster

Built for token-by-token decoding with small batch sizes, the format reduces the weight data moving through memory. Intel developed separate unpacking kernels for AVX-512 and AVX2 CPUs as well as Xe2 GPUs, joining other efforts to fit compressed models into faster inference pipelines for AI agents.

On AVX-512 hardware, the kernel uses the presence bitmap as a mask and pdep to scatter the compacted sign bits across the nonzero weight positions. Because Xe2 GPUs lack an equivalent instruction, Intel implemented the same operation with a 2KB lookup table.

Benchmarks across five systems

Compared with the 2-bit kernels, BITCOS ran 10% to 18% faster on the 64-core Xeon server and 2% to 15% faster on the 24-core Core Ultra 9. Performance improved by 9% to 22% on the integrated Arc 140V and by 2% to 27% on the discrete Arc Pro B70. These results measure decoding after the model has loaded, separate from efforts to cut GPU inference cold starts from minutes to seconds.

The smaller format did not win everywhere

On the eight-core Lunar Lake CPU, Intel’s fixed 2-bit kernel beat BITCOS on every model because the system had enough bandwidth to make unpacking the bottleneck. BITCOS remained faster on the GPUs, although decoding overhead limited the gains. Computer scientist and AI infrastructure author Chip Huyen has made the same point about inference more generally, arguing that the right optimization depends on whether compute, memory or bandwidth is holding back the workload.

On the eight-core Lunar Lake CPU, Intel’s fixed 2-bit kernel beat BITCOS on every model because the system had enough bandwidth to make unpacking the bottleneck.

Format limits and open questions

The paper has not been peer-reviewed; all five test systems used Intel hardware, and the end-to-end benchmarks covered seven models at batch size one. Intel has yet to test the format on Nvidia, AMD, or Arm hardware.

The post Intel squeezed a 1.58-bit LLM down to 1.485 bits without changing a single weight appeared first on The New Stack.

Mixture of experts (MoE) has become one of the defining architectural trends in large-scale AI model training. DeepSeek, Qwen, and Mixtral are examples of MoE...

Mixture of experts (MoE) has become one of the defining architectural trends in large-scale AI model training. DeepSeek, Qwen, and Mixtral are examples of MoE models that match or exceed the performance of dense model counterparts at a fraction of the training compute. MoE models provide efficient training through conditional computation. Instead of one dense feed-forward network (FFN) shared…

Source

Everyone knows the open-weight model pitch by now: companies can download the weights, customize them, run them on infrastructure of their choosing, and retain far greater control over where their data is processed — often at a much lower cost than using proprietary models.

Moreover, open-weight models are now thought to trail the leading frontier models by only around four to five months. Nvidia, the world’s most valuable company, is betting heavily on that future. In early September, it agreed to acquire Hugging Face — the sprawling “GitHub for AI” that hosts more than three million models — for $12.9 billion, while pledging to keep the platform open to different models, clouds and computing providers. And on Thursday, Nvidia detailed how Nvidia is using its own open-weight Nemotron model to manage its vast global supply chain in partnership with Palantir.

That power also comes with serious security questions. OpenAI president Greg Brockman recently warned that increasingly capable open-weight models — pointing specifically to China’s GLM-5.3 — could “significantly accelerate the threat landscape” as models with advanced cyber capabilities become freely downloadable and modifiable.

But for businesses accessing those models through third-party services, there is another concern closer to home: where their own data goes when they use those models, particularly when the model originated in China.

China and the open-weight factor

Hugging Face data from February showed models from Chinese developers accounted for 41% of downloads in the preceding 12 months, ahead of the US at 36.5%. Over on OpenRouter, meanwhile, open-weight models now account for around 60% of tokens consumed by US-originating requests, with the company noting that Chinese models constitute the majority.

And that’s why OpenRouter is now giving companies a way to put a geographic fence around that traffic. The AI model marketplace has officially launched US in-region routing into general availability for business and enterprise customers, promising that requests sent through its US endpoint are decrypted, processed and served entirely inside the country — or rejected if that can’t be done.

The feature itself had been quietly available in some form before now, with OpenRouter updating its documentation in early August to say US in-region routing was available to enterprise customers by request. It’s also worth noting that this is in addition to European in-region routing, which it says has been available since October 2025.

Started in early 2023 by former OpenSea CTO Alex Atallah, OpenRouter serves as an interface to the crowded AI model market, with developers able to switch between hundreds of models from myriad providers via a single API. Payments giant Stripe recently announced plans to acquire the company in a reported $8 billion deal, while a slew of other companies including Cursor, Ramp, and Meta, are also building their own model routers.

The reason why model routers are such hot property right now is largely down to economics. Developers have traditionally hard-coded applications to send everything to the same model, while a model router can instead make that choice request by request, sending easier jobs to cheaper models while reserving the pricier frontier systems for the work that actually needs them.

That intermediary role is also what makes OpenRouter’s new residency controls possible: it already decides which provider serves each request, and can now restrict that choice to provider endpoints operating in the US.

Keeping Chinese models inside the US

In a blog post announcing the new feature on Wednesday, Cailee Moberg, who works on OpenRouter’s product team, notes that while US-developed models from Nvidia and Thinking Machines are contributing to the broader open-weight model boom, Chinese models dominate usage and raise tough questions for companies concerned about their data.

“Models from Chinese labs are still most of the [open-weight model] volume, and procurement approval for those models can be difficult.”

“Models from Chinese labs are still most of the [open-weight model] volume, and procurement approval for those models can be difficult,” Moberg writes.

In its 2026 State of AI in the Enterprise report, Deloitte concluded that sovereign AI was on the rise, noting that 77% of companies “now factor country of origin into their vendor selection,” while nearly 60% construct their AI stacks “primarily with local vendors.”

And this at least partly explains why OpenRouter is now offering in-region routing for US customers. Moberg points to DeepSeek V4 Pro, Kimi K3 and GLM 5.2 as specific examples. All three are available through US In-Region Routing because Baseten, Fireworks and Azure serve them from US data centers. Companies could already keep these models inside the US by self-hosting them or using a US provider directly; OpenRouter’s new routing gives its own customers that residency guarantee without having to manage those deployments themselves.

OpenRouter maintains a live list of models eligible for US in-region routing, ranging from proprietary frontier models from OpenAI and Anthropic to open-weight models from the major Chinese labs.

“In-Region Routing allows teams with data residency requirements to get the price and performance gains from Chinese open-weight models,” Moberg continues. “When a US or EU provider hosts a model, requests go to that provider and the lab is not involved.”

“In-Region Routing allows teams with data residency requirements to get the price and performance gains from Chinese open-weight models.”

The technical change happens at the routing layer. With OpenRouter’s standard global endpoint, a request can be served by an eligible provider operating in any region, so even using a model from a US company does not guarantee that the request itself is processed in the US. With us.openrouter.ai, the request is decrypted on OpenRouter infrastructure inside the US and the pool of providers is filtered to endpoints OpenRouter has approved as operating there.

If no compliant US provider can serve the requested model, OpenRouter returns a 404 error. Companies can also enforce the regional restriction through OpenRouter’s Guardrails at the workspace, team or API-key level, while tools that would send prompt data outside the US are disabled on the regional endpoint.

So while none of this ultimately changes where the DeepSeek, Kimi or GLM models are developed, in-region routing alters which copies of those models its US customers can be routed to, and where their prompts are handled along the way.

The post Chinese AI models dominate OpenRouter’s US token consumption. It can now guarantee that traffic stays entirely in the US. appeared first on The New Stack.

Enterprise AI company Cohere announced North Small Translate last week, a mixture-of-experts (MOE) open-weight machine translation model that works across 50 languages.

Developers can download the weights for noncommercial use under CC BY-NC 4.0. Cohere offers commercially licensed deployment through Model Vault, which is a Cohere-managed inference environment. Cohere positions the model as part of its sovereign AI strategy, aimed at organizations that want greater control over where their models run and how their data is handled.

North Small Translate builds on Cohere’s multilingual and translation lineage, which includes its Tiny Aya and Command A Translate model families. The company claims North Small Translate outperforms “similarly sized open-weight models” under 1T parameters, as well as API-based translation models in various dimensions of machine translation on average. 

Cohere co-founder Nick Frosst tells The New Stack that the model’s efficiency draws from the fact that it is non-reasoning, i.e., it relies on learned statistical patterns without a step-by-step logic process, which means it uses fewer tokens.

Machine translation is still broken for most of the world’s languages

“We spent nine years scaling an architecture invented to fix translation, and machine translation is still broken for most of the world’s languages,” Frosst says. “General-purpose models get you most of the way and then stop. The next phase of enterprise AI in this space is smaller, more specialized, and runs inside your own walls.”

“…machine translation is still broken for most of the world’s languages.”

In Cohere’s reported evaluation using WMT26 benchmarks, the company states that North Small Translate leads with a WMT26 All Languages benchmark score of 83.60, compared with 81.56 for Qwen 3.5 397B A17B, 76.50 for GLM 5.2 FP8, 81.37 for DeepL NextGen, 79.46 for Gemma 4 31B (on), and 68.20 for Google Translate. 

With its mixture-of-experts architecture and 218 billion total parameters, with 25 billion active. Cohere points to North Small Translate’s smaller compute & memory footprint than other models. Some model-to-model comparisons in this space aren’t fully substantiable, since not every vendor discloses parameter counts.

With current solutions, long documents start to fall apart

“Machine translation allows documents to be translated from one language to another automatically. With current solutions, long documents start to fall apart,” Frosst says. “Google Translate scores 21.3 on our long-context test, Gemma 4 31B 19.4; we score 48.9. That’s [for example] a safety manual that reads fine on page one… and has drifted by page ten. The other risk is where the text goes. Once you push HR policies or regulated documents through a third-party API, that data has left your building, and necessarily that means your control over it is diminished.”

“The risk [in machine translation] is where the text goes. Once you push HR policies or regulated documents through a third-party API, that data has left your building and necessarily that means your control over it is diminished.”

Explaining why the model offers “stronger translation performance” across complex enterprise translation tasks, Frosst says the model can support work spanning “a high volume” of sensitive documents. 

As well as its 50 languages (32 ‘high-resource’ languages + 18 others), the Cohere team explains that the model also supports translation-workflow-focused capabilities, such as structured translations (i.e., Markdown or JSON documents), instruction following (i.e., recommended tone & format), and terminology guides (i.e., providing specific vocabulary to use in the translation), all as part of the model.

“North Small Translate works with a multi-pass workflow,” explains Frosst. “The model translates, reviews its own output, finds errors, and fixes them – and this is the same loop we used in training. We ship both because standard is one pass and built for volume, while the agentic [version] spends more tokens for 84.36 against 83.60 on WMT26. That difference ends up being worth it when the document is a contract or a safety procedure, for instance, but in other cases you’d rather optimize for efficiency.”

“The model translates, reviews its own output, finds errors and fixes them.”

Model ‘steerability’ drives suggesting language tone and formatting

This model uses the same architecture as prior Cohere models but improves performance through post-training advances, including reinforcement learning and new datasets, specifically for machine translation tasks.

Frosst concludes that, across the translation model marketplace, generative machine translation models offer the highest quality and steerability (i.e., suggesting tone, formatting, etc.) but typically cost much more than Neural Machine Translation (NMT) models commonly used in commercial use cases. 

North Small Translate was developed in partnership with RWS, an AI solutions company pioneering in language technology and services. Collaboration with RWS, specifically with its Language Weaver research and science teams along with its language experts, helped shape the model’s real-world translation performance throughout development. 

As noted above, developers can access the weights free of charge for non-commercial use in three quantizations. There is also a Hugging Face Space and an API for those who lack the required hardware. 

The post “Machine translation is still broken for most of the world’s languages”: Cohere builds non-reasoning for a reason appeared first on The New Stack.

This week, Cohere released North Small Translate 1.0 under a CC BY-NC 4.0 license: the weights are there to download, evaluate and study, but not to run in production without a commercial agreement.

It’s an interesting choice from the Canadian foundation model company, which has built its pitch around AI sovereignty for regulated industries and describes this release as part of a mission “to make sovereign AI a technological reality.” Sovereignty there means control over where the model runs and who sees the data. A commercial license keeps that promise intact. It stops short of independence from Cohere. Enterprises keep their data and their infrastructure. They don’t get to fork the model, build a product on it, or keep running it if the terms change at renewal.

Open weights, except for commercial production

North Small Translate is an open-weights mixture-of-experts model built for machine translation across over 50 languages and locale variants. It has 218 billion total parameters, with 25 billion active parameters and a 16,000-token context window.

Not all users have the same access to those weights.

Per Cohere, the model is designed to give researchers, developers, and enterprises “flexible ways to evaluate and deploy machine translation while retaining control over their data and infrastructure.”

That’s an appealing description for organizations keen on pursuing sovereign AI. But the open-weight release comes with an important caveat: Not all users get the same rights to take advantage of those weights.

North Small Translate is available today on Cohere’s free tier through the Chat V2 API. For those who intend to use the model weights for non-commercial use, the FP8 weights are available on Hugging Face under the CC BY-NC 4.0 license.

But if enterprises want to put them into production, then a different set of terms applies. They’ll have to purchase a commercial license and deploy North Small Translate through Model Vault, Cohere’s fully managed inference platform.

Cohere’s not the only one drawing a line around open-weight use

Other AI companies are starting to attach more conditions to their open-weight models, too.

Last month, Chinese AI lab Z.ai released the weights for its flagship GLM-5.3 model on Hugging Face. But like the Canadian AI company, it also changed its licensing terms depending on who is deploying the model — a departure from its previous approach. While GLM-5.2 shipped under the permissive MIT license, GLM-5.3 adds new requirements for certain commercial users.

Cohere, for its part, has been similarly mum about why it made North Small Translate’s open weights noncommercial.

These requirements apply only to companies with aggregate revenue over $10 billion over 12 consecutive months. Additionally, if these companies want to host GLM-5.3 or its derivative works for commercial purposes, they have to first pass the Chinese lab’s security review.

Z.ai didn’t explicitly spell out why it decided to make such an about-face for GLM-5.3, which is especially puzzling given that its predecessor shipped under MIT without any commercial stipulations. Cohere, for its part, has been similarly mum about why it made North Small Translate’s open weights non-commercial.

Sovereign deployment, with restrictions

The Canadian company’s decision to make North Small Translate available as open weights but gate commercial use is a head-scratcher, given its history of selling sovereign AI to enterprises.

In fact, in June, it pitched North Mini Code, its first coding model, as a response to developers demanding the same sovereignty guarantees that regulated industries have long required.

Unlike North Small Translate, though, this open-weight model was released under an Apache 2.0 license from the get-go — without any comparable restrictions for commercial users.

Clearly, Cohere is going in a different direction with its latest open-weight release, emerging as another example of AI companies putting tighter terms around increasingly capable open-weight models.

The post Cohere’s new translation model is open weights — but not for commercial use appeared first on The New Stack.

The multi-modal model could help astronauts navigate craters, investigate ancient lava, and search for ice, as the US plans for a long-term lunar presence.

Nvidia and Palantir announced Thursday that they’re working together to bring “sovereign AI to critical supply chains,” kicking off initially with Nvidia’s own sprawling supply chain.

The news builds on a partnership that kicked off last October, when the duo said they would combine Nvidia’s AI computing and models with Palantir’s software to help companies use AI to make complex operational decisions. Then in June, they expanded that effort into sovereign AI, allowing organizations to run and customize Nvidia’s AI models inside tightly controlled environments while keeping sensitive data and model weights under their own control.

Now, they’re applying that technology inside Nvidia itself, where they say a smaller, fine-tuned model is already outperforming a far larger one.

A proving ground for sovereign AI

The companies have fine-tuned Nvidia’s 30-billion-parameter Nemotron 3.5 Lightning model on decisions made by Nvidia’s supply-chain operations team. Palantir’s Foundry and Artificial Intelligence Platform (AIP) bring together the data behind those decisions, while its Ontology acts as a live map connecting components, factories, capacity and production commitments. Nvidia’s cuOpt software, meanwhile, works out how to distribute scarce parts, with Nemotron weighing the wider context and recommending what planners should do.

They then plan to “extend the learnings from Nvidia’s deployment” to companies in other sectors, including manufacturing, energy, healthcare, automotive and aerospace. Palantir’s own customers will be able to build versions tailored to their own supply chains by training Nemotron on their proprietary data using Foundry and AIP, then run the resulting system on-premises or through cloud and colocation providers.

So, in effect, Nvidia and Palantir are putting the sovereign AI partnership they outlined in June into practice inside Nvidia, while using that deployment as a proving ground for an architecture other companies can adapt to their own use cases.

Nvidia as a test case

As the world’s most valuable public company at a $5.4 trillion market cap, there’s good reason for Nvidia to start close to home. Its supply chain spans millions of parts, thousands of suppliers, and a global network of manufacturing partners, with the company saying a single Vera Rubin rack alone contains some 1.3 million parts. Those components have to arrive in the right place at the right time: if one part is missing, assembly can stall while everything else that arrived sits waiting.

“Supply chains are the operating system of the physical economy, and AI factories are among the most complex systems ever built.

Jensen Huang

And that complexity is what Nvidia founder and CEO Jensen Huang says makes supply chains a natural target for the technology. From chips and memory to manufacturing, networking, power and cooling, he argues that building modern AI systems increasingly depends on coordinating an enormous web of companies and components.

“Supply chains are the operating system of the physical economy, and AI factories are among the most complex systems ever built,” Huang says in a statement.

Palantir co-founder and CEO Alex Karp goes further, arguing that Nvidia’s operations provide an unusually demanding environment in which to put the companies’ approach to the test.

“Nvidia has arguably the most valuable, intricate, and complex supply chain in the world.”

“Nvidia has arguably the most valuable, intricate, and complex supply chain in the world,” Karp adds in a separate statement.

The sovereignty selling point

Nvidia has long been positioning itself at the center of the open-model debate. In July, Huang even used his first-ever post on X to promote an industry letter lobbying Washington to support frontier open-weight models, arguing that they give companies and countries more control over their AI infrastructure.

Then in early September, Nvidia swooped in with a $12.9 billion deal for Hugging Face, the so-called “GitHub for AI models.” Amid concerns that ownership by the world’s dominant AI chipmaker could undermine Hugging Face’s neutrality, Huang pledged that it would remain open, continue hosting models from across the industry and support hardware beyond Nvidia’s own.

Nemotron is central to Nvidia’s own open-model push. The name dates back to 2023, when Nvidia released its first Nemotron-3 8B models for enterprises to customize and fine-tune. Those early models were downloadable through Hugging Face and Nvidia’s NGC catalog, although access was gated and governed by Nvidia’s own community license. So they were customizable, and their weights were available, but the much broader “open model” positioning Nvidia uses today came later.

The current Nemotron 3 series arrived back in December, initially spanning Nano, Super and Ultra models aimed at different agentic AI jobs. Nvidia now publishes weights and, for many of the models, training data and recipes so developers can customize themselves. Nemotron 3.5 Lightning, released in August, is the 30B model Nvidia and Palantir have fine-tuned for this supply-chain deployment.

That openness is also at the heart of the whole sovereignty pitch: companies can adapt Nemotron using proprietary data while keeping that data, the model weights, and inference inside their own environment.

Specialization over size

Nvidia’s own deployment gives outsiders a result to chew on. It says the fine-tuned 30B Lightning scored 86.7% accuracy on its supply-allocation task, versus 55.5% for the 550B Nemotron 3 Ultra—a model roughly 18 times its size.

In a technical blog post published on Thursday alongside the main announcement, Nvidia solutions architects Nell Barber, Rana Haber, and Aastha Jhunjhunwala note that the result shows how far specialization can go. On a tightly defined allocation task, the 30B model outperformed a general-purpose model more than an order of magnitude larger.

“This doesn’t mean the smaller model is more capable overall. Its gains are concentrated in the domain it was post-trained on.”

“This doesn’t mean the smaller model is more capable overall,” they add. “Its gains are concentrated in the domain it was post-trained on. Future production risk forecasting remained difficult despite fine-tuning. Specialization improved the decision task but failed to solve every prediction problem attached to it.

For companies considering Nvidia’s blueprint, the more interesting takeaway may be this: a smaller open model, trained on business specifics, can sometimes be more useful than reaching for the biggest model available.

The post Nvidia and Palantir fine-tune a 30B Nemotron model for Nvidia’s supply chain. It beats a model 18 times its size. appeared first on The New Stack.

Update: As of 6:30 p.m. Eastern on Friday, September 4, GPT-6 Astra was available on ChatGPT for all paying users with plans that include access to it.

Thibault Sottiaux, a leading member of the technical staff at OpenAI, posted on his X account, “OK nevermind, the team and Astra did a good job and our systems are more scalable than we anticipated. Astra is now rolled out to all Plus and Business users too. Hope you have a blast and let us know how it goes!.”

Earlier in the day Friday, OpenAI CEO Sam Altman posted on X, “GPT-6 Astra is now available to all Pro, Enterprise, and Business Premium users in Work/Codex, and is available in the API. We will start rollout to Plus and Business users next. Thank you for the patience.”

Users who pay for the $8/month ChatGPT Go plan do not and will not have access to GPT-6 Astra or GPT-5.6, according to the company’s pricing tier documentation.

Sottiaux posted on Thursday evening after OpenAI announced the debut of Astra but before it released it to users: “We are starting to release GPT-6 Astra, and we are doing it as carefully and quickly as possible.”

OK nevermind, the team and Astra did a good job and our systems are more scalable than we anticipated.

Astra is now rolled out to all Plus and Business users too. Hope you have a blast and let us know how it goes!

— Tibo (@thsottiaux) September 4, 2026

---

Our original story, published at 12:24 p.m. Eastern on Friday, continues below:

OpenAI launched GPT-6 Astra on Thursday, but many developers hoping to give it a test drive are still waiting for access.

Hours after the announcement, OpenAI CEO Sam Altman apologized for what he called a “messy rollout,” acknowledging that broad access to Astra had not yet begun for either API customers or ChatGPT subscribers.

“First, sorry for the messy rollout,”

“First, sorry for the messy rollout,” Altman wrote in a post on X. He added that OpenAI expected to begin the broader rollout “in the near future,” starting with ChatGPT Pro subscribers.

first, sorry for the messy rollout.

second, when we screw up, we try to make it right.

third, we should be able to begin broad rollout to API customers and chatgpt subscribers in the near future. as usual we will start with pro subscribers. https://t.co/nKOhW18CDK

— Sam Altman (@sama) September 4, 2026

For developers, that creates an unusual bind. OpenAI has already published Astra’s API documentation and pricing, including its 1.05 million-token context window, support for up to 128,000 output tokens, and standard API rates of $10 per million input tokens and $50 per million output tokens. But the endpoint itself is still rolling out, and OpenAI has yet to explain specifically what went wrong with the rollout. So while Astra’s benchmark results were impressive, developers and independent reviewers who want to evaluate those claims against their own workloads are still left waiting.

Why developers are waiting

OpenAI’s engineering lead for Codex, Thibault Sottiaux, offered a little more detail in an X post of his own about what is happening behind the scenes. He said the rollout will take “a few days” to complete and that OpenAI is bringing new systems and additional compute online as it expands access.

“We are starting to release GPT-6 Astra, and we are doing it as carefully and quickly as possible,” Sottiaux wrote. He added that “many novel systems will operate at scale for the first time” during the rollout and that OpenAI is “bringing a lot of compute up.”

“We are starting to release GPT-6 Astra, and we are doing it as carefully and quickly as possible,”

While this does not establish that compute capacity caused the delay, it does offer some insight into what OpenAI is dealing with as it expands access. The company’s original announcement said Astra would initially be available to a limited group of organizations, with ChatGPT Plus, Pro, Business, and Enterprise users, the OpenAI API, Microsoft Azure, and AWS Bedrock expected to follow “over the coming days.”

Developers who have secured early API access are already encountering surprises beyond the rollout itself. As The New Stack reported this week, Astra’s API introduces a new class of safety-triggered interruptions that look like timeouts but aren’t — which could make a real difference for developers building production workflows around the model.

Banked resets explained briefly

While users wait, OpenAI is trying to make up for at least part of the delay with something it calls a “banked reset.”

Sottiaux said paid ChatGPT subscribers will receive one banked reset for every day they remain without Astra access, beginning September 3, adding that the “team is moving mountains to give access as fast as we can.”

“Team is moving mountains to give access as fast as we can.”

OpenAI has used banked resets before with ChatGPT Work and Codex, giving users a way to replenish their usage after hitting a limit. They aren’t additional API credits or a permanent increase in usage limits but resets users can save until they need them.

What’s different with Astra is how OpenAI is handing them out: one for every day a paying ChatGPT subscriber remains without access. So far, OpenAI hasn’t announced anything similar for developers waiting to use Astra through the API.

The gesture fits a broader pattern of OpenAI experimenting with how it charges for AI: the company recently announced an outcome-based pricing model that would bill only when the model produces a correct result; another sign that its pricing strategy is still very much in flux.

Compute capacity complicates rollout

OpenAI says Astra will roll out to Plus, Pro, Business and Enterprise users, along with the OpenAI API, Microsoft Azure and AWS Bedrock, “over the coming days,” with Pro subscribers first. Sottiaux said the rollout should take a few days to complete.

For developers, the biggest unanswered question is when broad API access will arrive and whether it will come with tighter usage limits. Astra’s persistent-agent capabilities also make the wait more significant for developers.

OpenAI did not immediately respond to The New Stack’s questions about the rollout problems, banked resets, or API timeline.

The post “Sorry for the messy rollout”: OpenAI launches GPT-6 Astra to most paying users a day after its unveiling appeared first on The New Stack.

Nvidia has confirmed that it’s agreed to acquire Hugging Face in a mammoth $12.9 billion deal that will bring one of the AI industry’s most important open-model platforms under the auspices of the world’s dominant AI chipmaker — and, with a market cap of well over $5 trillion, the most valuable company on Earth.

When reports first emerged in August that Nvidia was lining up a gargantuan bid for what has often been described as the “GitHub of AI Models,” concerns quickly surfaced over what the deal could mean for the platform’s openness and hardware neutrality. As The New Stack reported at the time, Hugging Face’s value lies partly in giving developers a neutral place to find and deploy open models across Nvidia, AMD, Intel and cloud accelerators — raising the question of what happens if that platform is owned by just one of them.

Keeping Hugging Face open

That, ultimately, is why Nvidia founder, president and CEO Jensen Huang is going to great lengths to allay fears that Hugging Face could become a vehicle for steering developers toward Nvidia’s own hardware and software stack.

“Nvidia compute will not be required to build on or deploy through Hugging Face.”

In the official announcement on Thursday, Huang makes a series of explicit promises around neutrality, noting that Hugging Face would “remain an open platform for the entire AI ecosystem,” continue to support multiple clouds and accelerators, while stressing that “Nvidia compute will not be required to build on or deploy through Hugging Face.”

“Hugging Face will continue to support open source and open weight models from across the ecosystem, from every model builder,” Huang writes. “It will continue to support multi-cloud and multi-accelerator development and deployment, so builders can use the hardware and infrastructure that best fit their work.”

Indeed, it’s clear Nvidia had clocked the concerns around the impending acquisition. The word “open” appears no fewer than 19 times in Huang’s relatively short announcement, underlining just how central that reassurance is to Nvidia’s pitch for the deal.

Huang also points to a recent open letter on open weights that he co-signed alongside executives and researchers from across the AI industry, including those from Hugging Face. The letter argued that open-weight models are critical to broadening access to AI, strengthening competition and giving developers more control over how models are deployed and adapted.

For what it’s worth, Nvidia has been pushing hard into open-weight AI itself, including through its Nemotron models and a broader effort to make frontier-class models easier to run locally. Hugging Face co-founder and CEO Clément Delangue went so far as to call Nvidia the “King of American open-source AI” a few months ago, pointing to its growing collection of public models, datasets and Spaces on Hugging Face.

In the wake of the announcement on Thursday, Delangue doubled down on that position in a fresh post announcing the deal, arguing that Hugging Face has reached the point where keeping open-source AI competitive would require substantially more resources.

“It needs more compute, more support, more collaboration and more visibility.” – Hugging Face CEO Clément Delangue

“10 years after starting Hugging Face, open-source AI is at an inflection point,” Delangue writes on LinkedIn. “Thanks to the community, we’ve shown that it can be a complement, and even an alternative, to closed-source APIs. But for it to happen at larger scale, it needs more compute, more support, more collaboration and more visibility.”

He adds that Nvidia has committed to backing Hugging Face while keeping the platform open, independent and compute-agnostic, with the founders and existing team staying on.

“Together, we think we can make open source the default way to build AI,” Delangue writes, setting out a goal of helping 100 million AI builders “own their intelligence rather than rent it.”

GitHub as a historical precedent

However, there is some historical precedent for taking such assurances with a degree of caution. When Microsoft bought GitHub for $7.5 billion in 2018, it similarly promised that the platform would remain independent and open. GitHub largely retained that openness, though Microsoft’s later use of public GitHub code to help train the proprietary, paid Copilot service sparked a backlash among factions of the open-source community.

In truth, that obvious comparison may actually understate Nvidia’s challenge. In recent analysis for Forbes, technology analyst Janakiram MSV argues that neutrality at Hugging Face has a hardware dimension that GitHub never had to contend with. Hugging Face maintains integrations spanning AWS Trainium and Inferentia, Google TPUs, Intel Gaudi, AMD Instinct and other accelerators. Under Nvidia ownership, continued support for those rival chips becomes a real test of just how neutral the platform will remain in the long run.

Not a done deal yet

As for the acquisition itself, well, it’s not over the line quite yet. In a filing with the US Securities and Exchange Commission (SEC), Nvidia notes that it expects the deal to close in the first half of 2027, subject to customary closing conditions and regulatory approvals. Given Nvidia’s lofty position in AI infrastructure and Hugging Face’s role as a major distribution point for open models, those approvals are unlikely to be a mere formality. Nvidia’s filing also flags the possibility that future regulation around open-source AI could affect Hugging Face’s operations or increase compliance costs.

“Hugging Face [will] continue to permit model makers, developers, and users to upload and download models and datasets of their choosing and to support other silicon vendors.”

Notably, Nvidia also uses its SEC filing to reaffirm its neutrality commitments, stating that under the commitment, “Hugging Face would continue to permit model makers, developers, and users to upload and download models and datasets of their choosing and to support other silicon vendors.”

Of the roughly $12.9 billion headline price, about $11.9 billion will go to Hugging Face stockholders, with up to $1 billion earmarked for equity-based retention awards for employees joining Nvidia.

And while the headline price is usually rounded to $12.9 billion, the actual figure is an oddly specific $12,930,300,000. That’s no accident either, as Hugging Face co-founder Thomas Wolf alludes to in a LinkedIn post. For those still in the dark, 129303 is the decimal Unicode value for the 🤗 emoji, while #129303 is a green color code nodding to Nvidia’s branding.

A neat little Easter egg buried inside what can only be described as one of the biggest AI deals of the year.

The post “Hugging Face will remain an open platform”: Nvidia strikes $12.9B deal for the ‘GitHub of AI’ appeared first on The New Stack.

A 438-billion-parameter reasoning model isn’t an obvious choice when speed is a priority. Multiverse Computing is betting that compression can make it one. 

On Wednesday, the Spanish company launched Quasar 438B, its first large-scale model and a system designed specifically for coding and enterprise agents. Quasar achieves a score of 43 on Artificial Analysis’ Intelligence Index and 69.3 on Terminal-Bench v2.1, while Artificial Analysis currently records its output speed at approximately 183 tokens per second. Multiverse is positioning Quasar as the European model with the highest score on the Intelligence Index, outperforming Mistral Medium 3.5 (30) and NVIDIA Nemotron 3 Ultra (38).

Multiverse is betting that a 438B-parameter model can be fast and inexpensive enough for agents that repeatedly reason, call tools, and check the results.

That puts Quasar in an interesting middle ground. It doesn’t match the strongest models on coding performance, but Multiverse is betting that a 438B-parameter model can be fast and inexpensive enough for agents that repeatedly reason, call tools, and check the results.

The model features a context window of one million tokens, is available in English and Spanish, and can be accessed via the Multiverse CompactifAI API.

Compression claims, missing details 

Multiverse built CompactifAI to shrink large AI models so they need less memory and compute to run. The company says it can reduce model size by 80% to 95% with only a small loss in accuracy, but it hasn’t disclosed how much Quasar was compressed or which model it started with.

In July, Multiverse announced a $570 million Series C to expand its library of compressed models and commercialize the technology. Quasar is the biggest test of that approach so far.

Multiverse hasn’t said what hardware is required to run Quasar or how much the compression reduces its memory and compute needs. That matters for agents, which may repeatedly call the model and other tools before finishing a task.

Multiverse hasn’t said what hardware it takes to run Quasar or how much the compression cuts its memory and compute needs.

Coding benchmarks show tradeoffs 

In Multiverse’s comparison, Quasar’s Terminal-Bench v2.1 score of 69.3 places it ahead of Mistral Medium 3.5 but still well behind the best frontier systems, while Claude Opus 5 achieves the highest score on that benchmark at 89.1.

Multiverse is pitching Quasar for software engineering, technical copilots, research, and workflow automation. Its 1-million-token context window gives agents room to work with large codebases and hold onto information as a task progresses, although processing more context also requires more compute. That can be especially important in coding, where code that passes every test can still trip up the next AI agent if it loses track of what came before.

Agent latency beyond throughput 

Artificial Analysis found that Quasar starts responding in about 1.1 seconds and can produce a 500-token response, including reasoning, in around 15.3 seconds. Those numbers are fast, but an agent also has to wait for tools, process growing context, and make repeated model calls over the course of a task. The agent tooling layer itself is still catching up to what these models need.

Those numbers are fast, but an agent also has to wait for tools, process growing context, and make repeated model calls over the course of a task.

Proprietary model, open questions 

Quasar is proprietary and only available through Multiverse’s API, so developers can’t inspect the weights or run it on their own hardware. For now, that also makes it difficult to know whether the speed Multiverse is reporting will carry over to everyday agent use.

Quasar also arrives as European AI companies are building more of their own model and compute infrastructure instead of relying on U.S. hyperscalers. Multiverse is taking a different route, using compression to make a 400B-plus model cheaper and faster to run. The next step is to see how that holds up against benchmarks.

The post Multiverse says its 438B model is fast enough for AI agents. The benchmarks tell a more complicated story. appeared first on The New Stack.

This post is the third in a series on AI model co-design. It explores how to accelerate LLM inference while maintaining accuracy using speculative decoding and...

This post is the third in a series on AI model co-design. It explores how to accelerate LLM inference while maintaining accuracy using speculative decoding and offers five guidelines for selecting draft length and draft mechanism across the Pareto frontier. For a discussion of how model design choices impact both throughput and interactivity without sacrificing accuracy, see AI Model Co…

Source

Alibaba this week unveiled Qwen3.8-Flash, an open-weight, multimodal Mixture-of-Experts (MoE) model. 

Hot on the heels of Qwen 3.8 Max, which arrived at the start of the month, this 125-billion-parameter AI model is offered as a prelude to Qwen 4. It is positioned as both a performance and value-for-money play. As such, it is claimed to have “superior capabilities in coding and office tasks” and an optimal balance among capability, latency, and cost.

“In this release, we are opening the weights of Qwen3.8-Flash-Next, a multimodal MoE model that also serves as an early preview of the architecture used in Qwen4,” confirmed Alibaba in its release blog.

How and why does Alibaba offer early access precursor models?

By releasing the architectural changes in Qwen3.8-Flash early, the organization hopes the community will examine the mechanics, constructs, and components within and start road-testing them before the full Qwen4 model family is built on top of them.

By stating that Qwen3.8-Flash paves the way for Qwen4, Alibaba is showcasing (and, importantly, openly sharing) design forms that it will carry into its subsequent models, so that developers can start building (or at least planning) their next codebases early. 

Specifically then, Alibaba has stated that Qwen3.8-Flash “plays the same role” that Qwen3-Next played for Qwen3.5 i.e. by which the company means that Qwen3.8-Flash introduced developers to the company’s hybrid Gated DeltaNet + Gated Attention design (computational components that enable AI models to balance long-context efficiency with contextual focus and attention during inference and training), which was then used across the Qwen3.5, Qwen3.6, Qwen3.7 and Qwen3.8 series.

“Qwen3.8-Flash-Next upgrades the model systematically along four aspects – attention, residual, embedding and optimization –  improving model capability while further optimizing computational efficiency, model capacity and training stability,” stated Alibaba.

Benchmark scores against rival models

When benchmarked on agentic coding, long-horizon agent tasks and multimodal intelligence, Qwen3.8-Flash appears to perform respectably against models including DeepSeek-V4-Flash and Claude-Opus-4.6 across SWE-bench Pro (agentic coding), CoWorkBench (long-horizon office work), Toolathlon Verified (real-world tool use), MathVision (visual maths problem-solving), AndroidWorld (agentic mobile use) and ERQA (embodied intelligence).

“Qwen3.8-Flash-Next upgrades the model systematically along four aspects – attention, residual, embedding and optimization – improving model capability while further optimizing computational efficiency, model capacity and training stability.”

Tested on agentic coding using SWE-bench Pro, Qwen3.8-Flash-Next scores 62.5, compared to 61.7 on Qwen3.8-27B, 55 on Qwen3.7-Plus, 56.0 on DeepSeek-V4-Flash-0731, and 53.4 on Claude-Opus-4.6 (Max).

Importantly, Qwen3.8-Flash requires only what Alibaba details as “around one-ninth of the training resources,” while delivering superior performance. The company says that this means Qwen3.8-Flash “significantly reduces” both training and inference costs compared with Qwen3.7-Plus, a model three times its size. 

What’s the difference between Qwen3.8-Flash-Next and Qwen3.8-Flash?

For the sake of nomenclature, Qwen3.8-Flash-Next is the open-weight research-frontier model available to developers on both the Hugging Face AI developer hub and Alibaba’s ModelScope community portal. Built on the same underlying architecture, Qwen3.8-Flash is the production version of the model, offered via the QwenCloud API with 1 million tokens by default and official built-in tools. 

Qwen3.8-Flash-Next features a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token. It natively supports 262,144 tokens of context and can be extended to 1,000,000 tokens with YaRN.

What architectural updates have happened?

As noted, Qwen3.8-Flash introduces architectural extensions across attention mechanisms, residual connections, embeddings, and optimization. Its hybrid attention architecture combines Gated DeltaNet (GDN), which compresses historical information, with Qwen Sparse Attention (QSA), a novel design that uses a lightweight compressed indexer to select relevant context, substantially reducing attention costs for long sequences.

Alibaba has explained that the Gated Residual (GR) mechanism expands data pathways between layers while strengthening cross-layer information flow and training stability. The N-gram Embedding technique scales model capacity with minimal additional computation, while the Muon Optimizer enhances the efficiency of large-scale model training.

“I’d love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either…it’s just not there (yet) for local professional software development.”

What do developers think of Qwen3.8-Flash?

In terms of developer reaction, it’s been a mixed bag so far. 

Mechatronics engineer Alok posts on X, saying he thinks the Video RAM barrier (i.e., the need for physical, high-speed GPU-based memory for LLM memory management in the face of model quantization that aims to enable better long-context inference) is now officially dead. 

“I just ran Qwen3.8-Flash-Next (MoE) 125B A6B with a  250,000 context window on a single 24GB RTX 4090 – 21 tokens/sec decode. 364 t/s prefill – no mtp. No dflash. No KV cache quantization! We are running datacenter models on consumer hardware,” enthused Alok.

Multi-disciplined software developer Embedding Shapes is less happy.

They post on Hacker News using some colorful language to describe how models keep [insert expletive]-ing up very basic things before saying, “I’d love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either. But I too got hopeful reading the sentiment on the Internet about Qwen 3.8, but it’s just not there (yet) for local professional software development.”

Pricing and access

Qwen3.8-Flash can be accessed via API Model Studio and Qwen Cloud, Alibaba’s AI-native cloud platform. Pricing per 1 million tokens is US$0.16 for input and US$0.47 (or 3 RMB for developers inside China) for output.

The model is also available on QwenWork, Alibaba’s workplace AI agent platform, where it runs a redesigned “standard mode” that cuts token consumption per task by 75% and “roughly doubles generation speed” compared with the current mode.

Alibaba has claimed that this brings flagship-level capabilities within reach of everyday workloads and that developers can run this model on hardware that they likely already own.

When is Alibaba’s Qwen4 scheduled for launch?

Alibaba has not confirmed a firm launch date for Qwen4. Still, a casual web search on the topic yields a range of commentators and market watchers who broadly agree it will arrive before the end of the year, possibly as early as September.

Conjecture in this space explores whether the next model family could be optimized for complex 3D coding and design tasks in advanced spatial modeling. Others think that the Mixture-of-Experts (MoE) architecture will be extended and that deeper native multimodal processing capabilities might be featured.

Alibaba was contacted for broader comment on this story but declined to engage.

Qwen, in traditional Chinese, 通義千問 (pronounced Tōngyì qiān wèn), translates to “a thousand questions on general meaning” in English. Use that in your local pub quiz this weekend.

The post Alibaba just released Qwen3.8-Flash: “An early preview of the architecture in Qwen4” appeared first on The New Stack.

Growing datasets and public benchmarks are making it harder to tell whether a model is being tested on something it hasn’t seen before.

On Thursday, Google DeepMind showed off what the company calls the first double-blind evaluation of a proprietary frontier-class AI model. The setup keeps Gemini’s model weights hidden from the evaluators while the test questions are hidden from Google.

The pilot tested Gemini 2.5 Flash Lite against private benchmarks from MLCommons and the Singapore AI Safety Institute, but rather than using the results to tout a new Gemini score, Google is focusing on how the tests were run, with neither side having access to the other’s data.

Benchmark leakage inflates scores

Google’s technical report cites earlier research that found signs of benchmark leakage in about half of the 31 models tested, and another study published this year that found contamination can inflate scores, particularly for larger models. This dynamic has already raised questions about where certain post-training coding gains actually come from.

Keeping benchmarks private would seem like the obvious answer, except closed models make that difficult. With a closed model, evaluators usually have to run their questions through the provider’s API, which means the company can see a test that was supposed to stay private, whereas running it independently would require the provider to hand over its model weights. Google’s setup is meant to give both sides another option.

Keeping benchmarks private would seem like the obvious answer, except closed models make that difficult.

How the enclave works

The pilot is using Google Cloud Confidential Space together with an NVIDIA H100 Confidential GPU and Intel TDX host memory encryption.

Google DeepMind provides Gemini with its weights and inference code, while the evaluator offers its benchmark prompts and evaluation code; these are then transmitted over encrypted connections into the enclave, where the evaluation takes place without either party gaining access to the other’s protected assets.

The model weights are stored in hardware-encrypted GPU memory, and the evaluation prompts are kept in encrypted host memory as well. After the test is complete, the evaluator is given the permitted results, and the temporary environment can then be destroyed.

Before either side sends over its private data, remote attestation verifies that the enclave is running the software they agreed on.

Code controls beyond encryption

The enclave protects what’s stored in memory, but the code running inside it can still create problems. If an application has unrestricted network access, for example, it could send sensitive information somewhere else.

OpenMined’s PySyft handles that part of the process by allowing Google and the evaluator to approve the code in advance and to block sensitive parts of the evaluation from making external connections.

That process takes some work, although compute doesn’t appear to be the problem. The paper puts the overhead at less than 5% and instead points to legal agreements and code reviews between organizations as some of the bigger hurdles to running these evaluations.

The researchers eventually want to make attestation much less hands-on, comparing the goal to the HTTPS lock icon in a browser, where the security checks happen without the user having to deal with the underlying hashes and keys.

Google still has a hand in verification because, although the Confidential Space guest OS is open source and its build process has been externally validated, individual builds rely on private signing keys and can’t be independently reproduced. At the same time, Google’s own services are used to sign and verify the attestation report.

The paper puts the overhead at less than 5% and points instead to the legal agreements and code reviews between organizations as some of the bigger hurdles to running these evaluations.

Scaling past one GPU

So while the system reduces how much the two sides need to trust each other, it doesn’t remove trust entirely from the equation. Some of it simply moves elsewhere, including to the hardware itself. There is still some trust involved in the hardware itself, including the assumption that the cloud provider and the hardware maker aren’t working together to circumvent the protections.

Then there’s the benchmark itself. MLCommons points out that it still needs to be carefully managed, no matter how well the questions are protected.

The pilot ran Gemini 2.5 Flash Lite on a single H100 80GB Confidential GPU. Still, the researchers are already looking at clusters of H100 and B200 GPUs connected via encrypted links to eventually evaluate models that are too large to fit on a single GPU.

If this approach catches on, developers could have another way to look beyond the benchmark score, with some proof that the company behind the model didn’t see the test before the results came out.

If this approach catches on, a benchmark score could offer some proof that the company behind the model didn’t get to see the test first.

The post Google found a way to test Gemini without seeing the questions appeared first on The New Stack.

NVIDIA Groq 3 LPX is the interactive AI inference accelerator for the NVIDIA Vera Rubin platform. At the core of the platform is NVIDIA Vera Rubin NVL72, the...

NVIDIA Groq 3 LPX is the interactive AI inference accelerator for the NVIDIA Vera Rubin platform. At the core of the platform is NVIDIA Vera Rubin NVL72, the most versatile machine ever built, delivering high throughput and interactivity across the widest range of AI workloads—from small to large models, both open and closed. Groq 3 LPX, when paired with Vera Rubin NVL72, extends the platform’s…

Source

I’m Matt Burns, Chief Content Officer at Insight Media Group. Each week, I round up the most important AI developments, explaining what they mean for people and organizations putting this technology to work. The thesis is simple: workers who learn to use AI will define the next era of their industries, and this newsletter is here to help you be one of them.

---

Model triage is becoming one of the most important skills for the AI-native developer. I’ve argued all summer that the people getting the most out of frontier models are the ones disciplined enough not to run the best model by default. On Wednesday, Stripe and Ramp validated that idea 70 minutes apart: Stripe bought OpenRouter, and Ramp released its internal router.

Bloomberg puts the OpenRouter price tag above $7 billion, and Axios says it’s more than $8 billion in cash and stock. Stripe has not released the terms, so the details remain fuzzy.

While the acquisition made headlines, the architecture is the story. For the last couple of years, picking a model was something written into an application, a string in a config file, and swapping models took some work. A router changes that workflow. Stripe bought the layer, and Ramp built it. Both are betting their existing relationships give them a unique wedge to own this critical layer in the new AI stack.

Picking a model is becoming a runtime decision

OpenRouter is an endpoint serving more than 400 models from over 80 providers, processing more than 10 trillion tokens a day. Andrej Karpathy calls it the transfer switch for AI. Ramp’s Router does the same job on a smaller catalog and claims roughly 40% lower cost for the same output.

Both are betting the model name in a codebase is a liability. They’re mostly right. Back in June, I pointed to Mitchell Hashimoto, who found a standard coding task that cost about $1.50 on GPT-5.5 and roughly $9 on Claude Fable, with both producing equally acceptable results. A router automates that triage, making decisions on every request rather than only on those a developer explicitly configures.

This is becoming a large problem and a large opportunity. Our own Amanda Caswell reported this week that Anthropic’s /claude-api skill was burning about 200,000 tokens before answering a single question, and that loading its reference docs on demand instead of up front cut that to roughly 25,000. Hafiz Hassan wrote for us last week about why AI pipelines cost 10x more than the demo, and every culprit on his list is an engineering decision: system prompts resent every turn, whole conversation histories appended, oversized RAG chunks, raw JSON dumped into context. It’s a great practical guide, and none of the items Hassan identifies are procurement problems.

The token bill is generated by your code, which is why the tools to control it are arriving there as well.

Stripe and Ramp want the same layer for opposite reasons

Stripe is attacking the problem from the bottom up, through developers. The company’s investor letter, leaked Wednesday by Eric Newcomer, makes the argument directly: “Up until now, every developer has needed a straightforward and reliable way to manage their revenue pipeline, and serving this need gave rise to Stripe. Going forward, however, every developer will also need a straightforward and reliable way to manage their intelligence pipeline.”

Stripe wants to control AI spending through the long tail of developers. Ramp wants to control it through its existing relationship with finance.

Stripe has built this product before. Its payments business hides dozens of local payment methods behind a single API, routing each transaction to the payment method most likely to convert. The AI version is the same idea applied to models instead of payment networks.

Ramp is attacking it from the top down, through finance. The company bought the router.com domain and says its customers already buy quadrillions of tokens a month through Ramp. Founder Veeral Patel’s launch post pitches the service simply: “Monitor and control your AI bill across every provider.” Adam Wazzan sums up Ramp’s strategy better than I can: “when a CFO ships a product for CTOs.”

Stripe wants to control AI spending through the long tail of developers. Ramp wants to control it through its existing relationship with finance. Both are chasing what is rapidly becoming one of the largest line items in corporate technology budgets: tokens.

On X, Kabir Goel pushes back on Stripe’s framing. Routing tokens is a way to spend less, while Stripe’s other products are designed to help businesses make more. 

“Stripe is just not where teams go to understand how much they’re spending,” he writes. “That’s pretty squarely Ramp territory.” 

He has a point about where teams look today. Whether that’s still true three years from now is exactly what Stripe just spent billions betting against.

The router worth pointing at is the one with no model to sell

Whichever router you point at decides which model writes your code, and not every router is disinterested. Our own Paul Sawers flagged the problem in July when he covered the first wave of Cursor’s, Ramp’s, and Meta’s routers. Cursor backs Grok and Composer. Meta is building Muse Spark. Both have reasons to send work to their own models, and Paul quoted developer Elvis Saravia asking whether routing logic ought to be open source rather than a vendor’s private judgment call.

Stripe and Ramp do not sell models. OpenRouter CEO Alex Atallah says as much in Stripe’s own announcement: Developers “need a neutral layer to orchestrate and manage them all.” Investor Gavin Baker frames the opportunity the same way, arguing that Stripe can become the neutral infrastructure layer for AI, just as it became the neutral infrastructure layer for payments. 

Neutrality isn’t free, though. Stripe takes a percentage of token spend, and Ramp wants your spending relationship, so “free through 2026” is a customer acquisition strategy with an expiration date.

The obvious objection is that vendor motives are the wrong thing to worry about, and routing quality is what really matters. That’s fair, and Towards Data Science published one of the best practical examples I’ve read. Pratik Rupareliya describes a routing layer that cut a support agent’s inference bill by 40% but also broke the product. A classifier sent “simple” queries to a cheaper model, but some of those “simple” queries were actually fraud investigations. 

The cheaper model answered them confidently and incorrectly. Customers stopped using the agent, churn rose above baseline in month four, and retention costs were four to five times higher than the savings. It took three months to surface and another month to identify the cause. His fix was per-tier quality monitoring combined with an uncertainty-routed cascade, which ultimately settled at 35% savings without sacrificing quality. It’s an excellent article to read before diving into model routing.

So instrument the routing. Log what the router picks on every request and break out your quality metrics by the model that served them. Ramp Router reportedly records the model, provider, tier, tokens, latency, cost, and fallback attempts for every call. Stripe OpenRouter rankings have been a public version of that telemetry for years. You can get similar visibility with either approach.

Right now, the model is becoming an implementation detail. The competition is shifting to the layer that decides which model gets the job. Stripe and Ramp are betting that developers won’t care what sits behind the endpoint, so long as the bill is lower and the results are good enough. 

The post Forget the model wars, Stripe and Ramp just started the router wars appeared first on The New Stack.

Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data...

Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data needed to adapt these models may be distributed across institutions or organizations that cannot centralize their raw records. Federated learning provides a way to coordinate training across these data-local sites. For VLMs…

Source

Teams customize their models to hit their targets for latency, speed, memory, and compute. With the open NVIDIA Nemotron family of models, developers can find...

Teams customize their models to hit their targets for latency, speed, memory, and compute. With the open NVIDIA Nemotron family of models, developers can find the right-sized model for their needs. The new Nemotron 3.5 Lightning NVFP4 checkpoint, for example, preserves accuracy while unlocking up to 4x faster throughput. It’s compressed down to 22 GB from the 66 GB full precision checkpoint…

Source

Anthropic announced it will embed invisible watermarks into text generated by new Claude models, including output produced through its API, coding tools and cloud partners. For developers, the mark offers another way to trace where AI-generated text or code might have come from, but it is not strong enough to prove its origin.

Laying out the plan in a support document, Anthropic said Claude models launched in the EU on or after Aug. 2, 2026, will include machine-readable marking from release. The company is working to add support to older models as well.

The marks will apply worldwide across supported Claude products, including the Claude API, Claude Code, Claude Cowork and Claude Tag. Text generated through AWS, Google Cloud or Microsoft Foundry will also carry the watermark when those platforms use a supported model. Because the mark is added at the model level, it follows the output into applications built on top of Claude — the same applications that are already reshaping how enterprises deploy AI infrastructure — although Anthropic cautions that some platforms and features may not support every type of mark.

The change follows the Aug. 2 start of Article 50’s transparency requirements under the EU AI Act, which require providers of generative AI systems to make synthetic output detectable in a machine-readable format. Anthropic signed the accompanying Code of Practice as a provider of both generative AI models and systems. OpenAI, Google, Meta, Microsoft and Mistral are among the other model providers that have committed to the code.

Yet, Anthropic is handling text and files differently. Text gets a watermark hidden in the words themselves, while supported files such as SVGs, PNGs and JPGs receive a digital signature using the C2PA standard. The file metadata can show that Claude processed an asset and whether the metadata has been altered.

“Because the watermark is part of the text, it will travel with the text when it’s copied and pasted elsewhere, and may persist through some editing,” Anthropic said.

“Because the watermark is part of the text, it will travel with the text when it’s copied and pasted elsewhere, and may persist through some editing.”

How token-level watermarking works

Anthropic has not explained how its text watermark works or said whether Claude uses KGW, a semantic version or another method. The company has also not shared any figures showing whether watermarking affects latency or adds to inference costs — a gap that matters for teams already wrestling with the hidden costs of agentic AI workflows.

Alex Cui, CTO and co-founder of AI detection company GPTZero, wrote in a technical explainer on X that watermarking systems fast enough to run on a streaming frontier model often follow the same general approach. One such technique, known as the KGW method, changes the probabilities the model uses when selecting its next token.

A language model normally calculates a probability for every token that could appear next. In a simplified watermarking system, a secret key and the preceding tokens are used to generate a hash, which divides the candidate tokens into two groups, often described as green and red. The model then slightly increases the probability of selecting one of the green tokens.

A detector with the same key can use the preceding text to reconstruct which tokens would have been favored at each position. A passage containing an unusually high number of those choices may carry the watermark.

Cui wrote that more advanced approaches can derive the watermark from the meaning of nearby text rather than an exact sequence of tokens, which may help the signal survive some paraphrasing because replacing a word does not always change the surrounding context.

“Their watermark needs to work token-by-token because they are streaming their text to users,” Cui wrote. “Many watermark methods plan sentences or paragraphs at a time, or change the text after it’s entirely written, in order to make their watermark robust to paraphrasers.”

Anthropic has not confirmed that Claude uses any of these methods, but streaming limits the techniques available because the model must add the signal while generating its response rather than rewriting a completed passage afterward.

Code resists invisible marking

Code presents a different problem because the model has fewer valid choices. Words can often be swapped or sentences rewritten without changing their meaning, but seemingly minor changes can break working code. That challenge intensifies as the AI coding era matures and more production code flows through model-assisted pipelines.

“There are some texts, like code, that cannot be arbitrarily changed; otherwise the code will break,” Cui wrote. “In those cases, the watermark needs to selectively change words in parts of the text that can tolerate synonyms,” such as variable names.

Code may also be difficult to track through a normal development workflow. Anthropic has not published tests showing how well its watermark survives those changes, so teams do not yet know whether a Claude-generated patch will remain detectable after passing through a pull request.

“In my testing, the watermarks don’t survive intense paraphrasing, especially if you combine word choice and syntax attacks.”

Pipelines silently erase watermarks

The same issue comes up when applications change Claude’s output before showing it to a user or committing it to a repository. Summarizing it with another model, translating it, splitting it into smaller sections, turning it into structured data or mixing it with database content could all make the watermark harder to detect.

Anthropic acknowledges this limitation. Editing, paraphrasing, translating or combining the response with other text may weaken or remove the watermark, while short excerpts may not contain enough of the signal to detect.

Cui wrote that determined users can attack a watermark by changing both the vocabulary and the structure of a passage.

“In my testing, the watermarks don’t survive intense paraphrasing, especially if you combine word choice and syntax attacks,” he wrote. Cui added that free paraphrasing tools he tested were able to bypass Google DeepMind’s SynthID text watermark.

Research supports those concerns. The “Watermarks in the Sand” paper found that, under defined assumptions, attackers can remove watermarks without severely damaging the quality of the content. The absence of a watermark does not show that Claude had no role in creating the content. The response may have come from an older model, may be too short to carry a detectable signal or may have been changed somewhere in an application pipeline. It may also have passed through a platform or feature that does not support that type of mark.

Finding a watermark does not prove authorship either. Claude may have proofread, translated or reformatted material written by a person. Anthropic says a detected mark means only that the content “may have been processed by Claude,” not that Claude created the underlying work.

“If Anthropic releases the watermark detector publicly, I think they defeat their own watermark. People find reliable watermark-removal strategies by testing against Anthropic.”

Detection creates new risks

Anthropic plans to give users and third parties a way to detect its marks, but it has not said whether that will take the form of a local tool, a detection API or access limited to selected organizations. A public detector would be easier for developers to add to their applications, but it would also allow someone trying to remove a watermark to keep editing and checking the text until the signal disappears.

“If Anthropic releases the watermark detector publicly, I think they defeat their own watermark,” Cui wrote. “People find reliable watermark-removal strategies by testing against Anthropic.”

The secret keys behind the watermark create another challenge. A leaked key could make it easier to remove the mark or imitate it in text that Claude did not produce. The scenario echoes what happened when provenance attestations were turned into camouflage — a trust signal that was supposed to increase confidence instead became an attack surface.

“To avoid a large blast damage from this, you need to have a couple secret keys in rotation,” Cui wrote.

Key rotation would require detectors to recognize marks created with both current and retired keys, including those embedded in content generated months or years earlier. Anthropic has not explained how it plans to handle that history.

Watermarks aren’t a substitute for real provenance

For developers, Claude’s watermark is best treated as another clue, not a replacement for audit logs or provenance tracking. Applications that need to show where an artifact came from can record the model ID, prompt version, response time and a hash of the original output, then log any changes made before it reaches a user or is committed to a repository.

The watermarking announcement arrives as Anthropic navigates deeper questions about what its models do in the wild. Recent incidents have exposed gaps between lab safety evaluations and real-world containment, and the company has publicly backed calls for the most powerful AI labs to slow down. Watermarking fits into that posture — a transparency mechanism rather than a safety guarantee — but its practical value depends on technical details Anthropic has not yet shared.

Anthropic tells customers to determine how Article 50 applies to their own products and says more technical documentation is coming. But until Anthropic shares those details, teams do not know how they will detect the marks, how key rotation will work or how well the watermark will survive common changes to code and application output.

The post Anthropic’s watermark survives copy-paste, but not the real dev workflow appeared first on The New Stack.

At the start of 2025, investors wondered whether Google could keep pace with OpenAI. By December, Alphabet was completing its best year on the stock market since 2009, helped by growing confidence in Gemini and Google’s broader AI strategy.

Much of that work came out of DeepMind, the British AI lab Google acquired in 2014 for about £400 million ($659 million in 2014). Now, Google is changing its leadership, while four of its best-known engineers are leaving to start an automated research lab.

DeepMind’s leadership reshuffles

Google announced Wednesday that DeepMind founder Demis Hassabis will step away from the lab’s day-to-day operations to become chair of Google DeepMind and chief scientist of Alphabet. Koray Kavukcuoglu, DeepMind’s chief technology officer and Google’s chief AI architect, will take control of Gemini model development, frontier AI research, the Gemini app and its developer teams.

At the same time, Jeff Dean, Sanjay Ghemawat, Oriol Vinyals and Quoc Le are also leaving Google to start Discovery Loop. As a public-benefit corporation, the company wants to use AI to automate scientific and engineering research. Google will remain involved as a founding investor and cloud provider. Just one person will now oversee everything from research to products.

In a January interview for CNBC’s podcast The Tech Download, Hassabis described DeepMind as the “engine room” of Google’s AI efforts. Hassabis said DeepMind develops Google’s core AI technology before it is distributed across the company’s products.

To get new AI into products faster, Google had to do more than update its models. DeepMind spent years reworking Google’s infrastructure so its AI work could move out the door quicker. That infrastructure push has extended to silicon, too — Google recently bet its inference future on a chip built for one model, a sign of how tightly the company is coupling hardware to its Gemini roadmap.

Hassabis said Google never struggled to invent new technology. After all, its researchers came up with the transformer architecture, which became the backbone of large language models. The problem was turning that research into products fast enough.

And moving faster is exactly what Google did. Hassabis pointed to Gemini 2.5, which landed in March 2025, as a turning point. Gemini 3 came out in November and put Google back in the mix with OpenAI and Anthropic — both of which have been making aggressive moves of their own to capture developer share.

By January, Hassabis said he and Google CEO Sundar Pichai were speaking almost every day, sometimes adjusting product plans and research roadmaps daily.

Kavukcuoglu inherits the Gemini roadmap

Kavukcuoglu has been at DeepMind for 13 years, started the deep learning team, and worked on projects like WaveNet and DQN. Now, model research, the Gemini app, and developer products are all on his plate. The flagship version of Gemini 4 remains unreleased after a planned June launch, according to Reuters. Hassabis confirmed the model’s name in his note to employees, saying Google was making progress on Gemini 4.

Discovery Loop’s founding engineers

Dean joined Google in 1999 and helped create Google Brain before becoming a technical co-lead for Gemini. Working with Ghemawat, he developed systems including MapReduce, Bigtable, and Spanner. Dean was one of the primary designers of TensorFlow, while Ghemawat worked on the infrastructure behind Google Search and several generations of its distributed computing systems.

Vinyals and Le made foundational contributions to deep learning and Gemini. Along with former OpenAI chief scientist Ilya Sutskever, they co-authored the influential 2014 paper that introduced sequence-to-sequence learning with neural networks. Their new company plans to use that experience to work on machine learning itself, at least to start.

“We are building AI solutions that can automatically solve important problems in machine learning, science, and engineering,” Discovery Loop says on its website.

“We are building AI solutions that can automatically solve important problems in machine learning, science, and engineering.”

The idea that AI can automate scientific discovery is no longer speculative. OpenAI’s Astra recently proved 10 long-standing math and science theorems for about $2,000 in token costs — a data point that suggests Discovery Loop is entering a space where early results are already landing.

Discovery Loop is built around the idea that AI can automate the entire experimental cycle. At Y Combinator‘s Startup School in July, Dean said the system could run the entire process, from proposing and carrying out an experiment to evaluating the results and deciding what to test next.

Ghemawat told Wired that Google’s systems were designed to support products such as Search, advertising and large consumer applications. Discovery Loop wants to build specialized infrastructure around research instead.

“We wanted to build something differently than how things are built at Google right now,” he said.

“We wanted to build something differently than how things are built at Google right now.”

Google keeps a stake

Rather than cut ties with the departing engineers, Google is investing in Discovery Loop and has signed a cloud partnership to provide the startup with computing capacity. That arrangement gives Discovery Loop access to the infrastructure needed to run large numbers of experiments without first building its own data centers. It gives Google a stake in anything the new company discovers and another major AI workload for Google Cloud.

Rather than cut ties with the departing engineers, Google is investing in Discovery Loop and has signed a cloud partnership to provide the startup with computing capacity.

Google has already lost Gemini co-leader and transformer co-author Noam Shazeer to OpenAI and AlphaFold researcher John Jumper to Anthropic. Worth noting, Alphabet shares fell more than 5% following Wednesday’s announcement.

Hassabis, meanwhile, will focus more of his attention on long-term AGI strategy and Isomorphic Labs, the drug-discovery company spun out of DeepMind.

In his message to employees, he said AGI now feels “close at hand” and that he wants more time to influence what happens next — a sentiment that carries extra weight as some of the most powerful AI labs face growing pressure to slow down.

​

The post Google’s four AI departures: “We wanted to build something differently” appeared first on The New Stack.

DeepSeek has launched DeepSeek-V4-Flash-0731, delivering a significant boost in agent performance without changing the model’s core architecture.

Following an announcement last week, the company made the update available as a public beta through DeepSeek’s API, and the open weights were published on Hugging Face under the MIT license later the same day.

Although the model itself hasn’t changed, DeepSeek says additional post-training is responsible for the performance gains, showing that meaningful improvements don’t always require a larger model.

🚀 DeepSeek-V4-Flash Official API is now LIVE in public beta!

🔷 We’ve massively upgraded its Agent capabilities—benchmark scores are now far surpassing the V4-Pro-Preview. Check out the massive performance leap below! 👇
🔷 The official V4-Flash now natively supports the… pic.twitter.com/NUzOyxza2f

— DeepSeek (@deepseek_ai) July 31, 2026

DeepSeek’s decision to release the production-ready weights under a permissive license gives organizations much more control over how they deploy and customize the model.

Although the model itself hasn’t changed, DeepSeek says additional post-training is responsible for the performance gains, showing that meaningful improvements don’t always require a larger model.

Same architecture, better results

DeepSeek says V4-Flash-0731 uses the same architecture as the preview release, with 284 billion total parameters and 13 billion activated parameters per token.

This is much smaller than V4-Pro, which has 1.6 trillion total parameters and 49 billion activated parameters. For companies running agents at scale, the activated-parameter gap translates directly into inference cost — though model price alone doesn’t tell the full story.

Even though it is still the smaller model, DeepSeek says the updated Flash version now beats the earlier V4-Pro preview on several agent-focused benchmarks.

Even though it is still the smaller model, DeepSeek says the updated Flash version now beats the earlier V4-Pro preview on several agent-focused benchmarks.

The company reported 82.7 on Terminal-Bench 2.1, 54.4 on DeepSWE, and 70.3 on Toolathlon-Verified.

Benchmark claims under scrutiny

Early independent testing by Artificial Analysis found a lower Terminal-Bench 2.1 score of 79%, which suggests that DeepSeek’s reported numbers may not always match independent results.

DeepSeek also shared results from several internal tests, though they have not yet been independently verified. If those results hold up, they add to growing evidence that companies can get more performance out of existing models through post-training instead of simply making them larger.

Open weights, full control

The MIT license means organizations aren’t limited to using DeepSeek through its hosted API. The release adds to a trend towards open-weight models closing the gap with proprietary alternatives. That flexibility is paired with support for tools many developers already use. V4-Flash now supports the Responses API for building AI agents and multi-step workflows, and DeepSeek has published instructions for integrating the model into Codex-based development workflows.

Familiar APIs, lower switching costs

For teams already using OpenAI-style APIs, that lowers the barrier to trying another model because they can evaluate it without making major changes to their existing setup.

The V4 technical report also covers inference improvements, like speculative decoding with DeepSeek’s DSpark framework, which are designed to make serving more efficient. When combined with self-hosted deployments, these features give infrastructure teams more ways to adjust performance for their own production needs.

This release reflects that companies are now finding new ways to improve model effectiveness without making models larger, and they’re competing on how those models are delivered. While many AI vendors focus on hosted APIs, DeepSeek continues to publish downloadable weights that organizations can run on their own infrastructure. Support for familiar API formats also makes it easier for teams to test open-weight models without revamping present workflows.

Companies are finding new ways to improve model effectiveness without making models larger, and they’re increasingly competing on how those models are delivered.

The post DeepSeek’s smaller model just outperformed its own flagship appeared first on The New Stack.