The race to make AI inference cheaper is pushing chip design beyond general-purpose accelerators. We’re now moving toward silicon that is tailored to individual models. And Google’s reported “Frozen v2” project suggests the Gemini AI model is part of the movement.

First reported by The Information, the unannounced chip would reportedly hardwire parts of Gemini’s architecture while leaving its weights updatable. That compromise could give Google much of the efficiency of model-specific silicon without making the hardware obsolete every time Gemini changes.

A spokesperson for Google tells The New Stack, “Our teams are constantly researching and experimenting with new innovations to deliver maximum performance and efficiency for our users and customers.”

A spokesperson for Google tells The New Stack, “Our teams are constantly researching and experimenting with new innovations to deliver maximum performance and efficiency for our users and customers. While not every project moves into production, this rigorous exploration is central to our full stack approach. By co-designing our hardware and software from the ground up, we ensure our systems are integrated and highly optimized for real-world workloads.”

According to the reporting, Google hopes the chip will help relieve the AI compute crunch that’s made it harder for cloud providers to keep up with demand, while also making Gemini much cheaper and more efficient to serve. Internal projections reportedly estimate the design could deliver six to ten times more tokens per watt than Google’s current generation of AI chips.

If the project moves forward, it would represent a different approach to AI infrastructure because Google would be building hardware specifically for Gemini. For developers, that’s an early indication that future AI systems may be designed with much tighter integration between the model and the hardware beneath it. 

Specialized silicon replaces flexibility

Currently, the vast majority of AI inferences run on Nvidia GPUs or Google’s own Tensor Processing Units (TPUs). Because these chips are intended to accommodate a wide variety of AI models, they carry a high degree of processing load.

There’s precedent for this kind of shift. Bitcoin mining followed a similar path, starting with CPUs, then GPUs, before ultimately settling on ASICs. AI inference may be headed in the same direction. Training still benefits from the flexibility of GPUs, but once a model reaches production, the priority shifts to serving as many requests as possible while using less power. 

Training still benefits from the flexibility of GPUs, but once a model reaches production, the priority shifts to serving as many requests as possible while using less power.

Competitors hardwire their own

Google isn’t the only company looking beyond general-purpose GPUs for AI inference. As inference becomes a larger share of AI workloads, more companies are experimenting with specialized hardware designed to improve performance while using less power. Even Nvidia, whose GPUs dominate the AI market, has invested heavily in inference, striking a reported $20 billion deal with Groq last year to license the startup’s technology.

Among the more ambitious efforts is Canadian startup Taalas. The company has demonstrated a chip with an entire 8-billion-parameter Llama model embedded directly into the silicon. By keeping the model on the chip instead of constantly moving data back and forth from external memory, Taalas says it can dramatically speed up inference, claiming throughput of roughly 17,000 tokens per second. Meanwhile, other companies are optimizing the spatial distance between memory and compute to avoid hardware lock-in.

d-Matrix’s new Corsair platform uses an SRAM-based in-memory compute architecture rather than relying on standard high-bandwidth memory (HBM) packaging. Similarly, SambaNova is deploying custom dataflow technology with a three-tier memory architecture to maximize tokens per watt for advanced workflows.

Google’s Frozen v2 sits uniquely in this landscape — borrowing the hyper-efficiency of hardwired architectures like Taalas, but retaining just enough flexibility to remain viable across multiple product cycles.

Freezing architecture, not weights

The “Frozen” name reportedly comes from the idea of permanently etching part of Gemini’s design into the chip itself. According to The Information, Google’s engineers have spent years trying to find the right balance between efficiency and flexibility.

An earlier concept, led by Google DeepMind Chief Scientist Jeff Dean, reportedly would have embedded Gemini’s model weights directly into the silicon. That idea was eventually abandoned because it would have tied the hardware to a single version of the model, dramatically limiting its useful life as Gemini continued to evolve.

Frozen v2 reportedly takes a different approach. Rather than locking in the weights, the chip would hardwire parts of Gemini’s underlying design while still allowing the weights to be updated over time. That would let Google continue improving the model without needing to replace the hardware every time Gemini receives a new version.

Rather than locking in the weights, the chip would hardwire parts of Gemini’s underlying design while still allowing the weights to be updated over time.

Cheaper inference reaches developers

By moving some of Gemini’s execution into the chip itself, Google could reduce some of the overhead that comes with running models on more general-purpose hardware. That could translate into lower latency, especially for applications that depend on near real-time responses. 

Google’s reported goal of delivering six to ten times more tokens per watt is ultimately about efficiency, and those savings could eventually make their way to enterprise teams building on Gemini, through lower API costs or more available capacity. While Google hasn’t said how it would pass those gains along, reducing the cost of inference has become a priority across the industry.

The post Google just bet its inference future on a chip built for one model appeared first on The New Stack.

Imagine you just finished a two-hour meeting. You were wearing a small AI work companion that promised to capture the conversation, structure it, and let you ask questions about it later. Two hours after the meeting, you open the app and ask for the transcript. You wait. The spinner turns. The whole reason you bought the device was so you’d never have to remember a meeting again, and now the one thing it promised to do well — recall what was said — is the thing that’s making you wait.

This is the failure mode that fascinates me about AI hardware products, because it’s not a model problem. The transcription was perfect. The summarization was good. The thing that frayed was the part nobody markets: getting the right bytes off disk and onto the screen at the moment the user asks. That’s a data problem. And for a product whose entire promise is memory, a data problem is a product problem. 

“For a product whose entire promise is memory, a data problem is a product problem.”

I want to walk through a real version of this because the team that hit it, Plaud, makes the most popular AI notetaker on the market, and the architecture that broke is the architecture almost every team in this category starts with. It feels reasonable at launch. It becomes a liability at scale. And the reasons why are worth understanding before you ship, not after.

First, what was built correctly

It would be easy to tell this story as “they made a mistake.” They didn’t. The original architecture was a sensible set of decisions, and I want to be precise about that before I dissect what went wrong, because the lesson is in the gap between “reasonable” and “right at scale,” not in anyone’s competence.

Plaud’s product generates two very different kinds of data per recording. There’s structured metadata: who recorded it, when, how long, what tags, what state the processing pipeline is in. And there’s the unstructured payload: the audio file and its transcript, which can run tens of megabytes per session. The team did the textbook thing. They put the structured metadata in MySQL, where relational queries and transactions are cheap, and they put the large objects in S3, where storage is cheap and effectively infinite.

If you’ve built systems, you’ve made this exact call. Object storage for blobs, a relational database for everything you need to query. It’s in every architecture diagram. It’s the default. And for a long stretch of the product’s life, it worked fine.

The problem is not that the decision was wrong. The problem is that it contained a hidden assumption: that the metadata and the content could live in separate systems because they would never need to be consistent with each other in real time. For most applications, that assumption holds. For a product whose core interaction is “give me back exactly what I recorded, right now,” it doesn’t. And the gap between those two worlds is where everything started to fail.

The transcript sidecar

In my article on RAG retrieval, I identified an anti-pattern I called the vector sidecar: the habit of standing up a separate vector database alongside your primary store, only to discover that the two systems can’t answer a single query together. The Plaud architecture is the same shape, applied to an AI hardware product. Call it the transcript sidecar.

The transcript sidecar is what you get whenever you split structured metadata from unstructured content across two systems with no shared transaction guarantee. The metadata says a recording exists, is complete, and is ready. The content lives elsewhere, reached via a separate call, with its own latency and failure modes. Nothing ties the two together. There is no transaction boundary that spans “the row that says this transcript is ready” and “the object that contains the transcript.”

This produces three distinct compounding problems.

• Retrieval latency is not a network problem; it’s a data locality problem. S3 is excellent object storage. It is not a database. When you make object storage the primary retrieval path for tens-of-megabytes payloads under concurrent load, you’re trading the consistency and latency guarantees of a database for the simplicity of a blob store. Under light load, the trade is invisible. Under heavy concurrent load, S3 retrieval latency and its variability become the user’s experience of your product.
• Consistency gaps open between two systems that fail independently. The MySQL row can indicate that a recording is ready a moment before the S3 object is durably reachable, or a replica can lag, causing the metadata a user sees and the content they fetch to disagree. With no shared transaction, the application layer inherits the job of papering over the gap, with retries, polling, and reconciliation logic that grows more elaborate every quarter.
• The schema becomes immovable at exactly the wrong time. More on this below, but the short version is that the metadata store hit a scale ceiling where changing the schema required a maintenance window. For a product still evolving its feature set, that’s a database governing the roadmap, rather than the other way around.

The unifying insight is the one I keep coming back to throughout this whole series: any time you split data across systems that can fail independently, you have handed the consistency problem to the application layer, where it is not solved so much as managed, and where it compounds over time.

When the database governs the roadmap

The most interesting failure in the Plaud story isn’t the latency. It’s the schema freeze, because it’s the one that teams least expect and feel most acutely.

With around 300 million rows, the team’s MySQL setup reached a point where schema changes such as adding a column or changing an index — the routine evolution of any growing product — could no longer be performed online without risk. DDL operations on a table that large, on that architecture, meant locking, replication strain, and the real possibility of downtime. So changes had to be batched into maintenance windows.

Sit with what that means. A maintenance window for a schema change is the time the database allows the product team to ship. The roadmap now bends around the database’s limitations. A feature that needs a new column will wait until the next window. The product’s cadence is set by the operational fragility of the data layer. Most teams don’t notice this inversion because it happens gradually, and then one day a product manager asks why a small change is a two-week project, and the answer is “the database.”

“A maintenance window for a schema change is the time the database allows the product team to ship. The roadmap now bends around the database’s limitations.”

The 300-million-row ceiling is a forcing function that teams hit later than they expect and should hit later than they do. Expect, because single-instance MySQL feels limitless right up until it doesn’t. Should, because the architecture that gets you to 300 million rows is rarely the architecture that takes you past it, and the migration is far more disruptive at 300 million rows than it would have been at 30 million. The ceiling is real; it is predictable, and most teams plan for it only after they’ve hit it.

What changing the architecture actually fixed

Plaud’s resolution was to consolidate the metadata layer into a distributed SQL database (they moved to TiDB), which removed the single-instance ceiling and restored online schema changes. I’m going to use their numbers, but not as a product pitch. I want them as evidence for a claim about where architectural debt goes in a product like this.

Before (dual-store-brain at scale)	After (consolidated metadata layer)
Throughput strained under concurrent load	~10x QPS headroom
Tail latency variable under load	P95 under 10 ms
Schema changes require maintenance windows	Online DDL, no downtime
Single-instance ceiling at hundreds of millions of rows	Horizontal scale across multiple clusters

QPS headroom and tail latency matter, but the first result I’d point to is the online DDL. Restoring the ability to change the schema without a maintenance window is what handed the roadmap back to the product team. The database stopped governing the release cadence. That’s not a performance win you can put on a benchmark chart, but it’s the one the product organization feels every week.

Also notice what the fix did not do: it did not attempt to store 30-megabyte audio files in the database. Large objects can still be stored in object storage. The point was never “put everything in one system for its own sake.” The point was that the metadata layer, the part that has to be consistent, queryable, and evolvable in real time, needed to actually behave like a database at scale, instead of becoming the fragile half of a split-brain architecture.

Where architectural debt goes

Here’s the claim the Plaud numbers are evidence for. When the database is the product, architectural debt does not accumulate in some back-office system that only the platform team feels. It accumulates in the product experience, where every user feels it.

This is the structural difference between an AI note-taker and, say, an internal analytics tool. For a product like this, every meaningful user interaction is a database operation. Recording creates rows and objects. Transcription updates state. Retrieval is a read. Editing is a write. Search is a query. There is no part of the product that isn’t, underneath, the database doing something. Architectural problems surface as product problems, one-for-one. A consistency gap becomes a transcript that is briefly missing. Slow retrieval reads as a product that is slow to remember. A frozen schema means features arrive late.

No user will ever file a bug report that says “your metadata store and your object store lack a shared transaction boundary.” Left unaddressed, this class of debt manifests as a product that feels less dependable precisely when someone depends on it. For a product whose entire value proposition is “trust me to remember,” the debt lands on the promise itself. This is why Plaud’s decision to re-architect when it did matter. The team fixed the data layer before the failure became noticeable to users.

“You can have the best transcription model in the category and still ship a product that feels unreliable, because reliability for this kind of product is a data architecture property, not a model property.”

This is why I say the database is the product. Not as a slogan. As a literal description of where the product’s quality is determined. You can have the best transcription model in the category and still ship a product that feels unreliable, because reliability for this kind of product is a data architecture property, not a model property.

The trap is waiting for the whole category

AI hardware is having a moment. Note-takers, wearables, ambient recorders, pendants, badges — each built on the premise that the device will remember so you don’t have to. The category is growing, and the products are getting better at the parts everyone talks about: the models, the form factor, the battery life.

But underneath, almost all of them start with the same architecture Plaud started with, because it’s the reasonable default. Structured metadata in a single-instance relational database. Large content in object storage. No shared transaction boundary. A schema that’s easy to change at ten million rows and frozen at three hundred million. The trap is identical, and it’s waiting at the same place on the growth curve for every team in the category.

The teams that will do well are the ones that recognize this early, understand that, for a memory product, the metadata layer is not back-office plumbing but the spine of the user experience, and plan their data architecture for the scale they’re trying to reach rather than the scale they’re at. Plaud’s migration is the pattern worth studying before you hit 300 million rows, not after. The lesson is cheaper to learn from someone else’s 300 million rows than from your own.

When your database and your product are the same thing, you don’t get to treat the database as someone else’s problem. It is the product. Build it like one.

The post “The database is the product”: What breaks when memory devices scale appeared first on The New Stack.

Meta is preparing to manufacture its own AI chip for the first time. According to an internal memo, the company expects production of its proprietary processor, Iris, to begin in September.

After clearing bug testing in about six weeks, the chip — reported on by Reuters — is expected to take on some of the inference work currently running on third-party GPUs, giving Meta more control over how it builds and scales its AI infrastructure.

It’s unmistakable that this could be the company’s most important move yet toward in-house silicon for AI workloads.

But anyone can see this isn’t really about the hardware. It’s unmistakable that this could be the company’s most important move yet toward in-house silicon for AI workloads. The timing, as Meta is locked in an aggressive multi-billion-dollar infrastructure race, is critical. It’s clear that the company’s CEO, Mark Zuckerberg, wants to grow into the AI titan he believes the company can be, but it’s nearly impossible when the competition controls the core infrastructure.

Custom silicon for inference

Iris is designed for a specific job inside Meta’s AI infrastructure as custom silicon optimized for Meta’s heavy workloads.  Iris expands Meta’s Meta Training and Inference Accelerators (MTIA) program, which is intended to move targeted AI inference workloads onto custom silicon.

The processor would handle workloads that drive content ranking, recommendations, and generative AI services across Meta’s family of applications, including Facebook, Instagram, and WhatsApp.

• The MTIA 300 is already deployed in production to run ranking and recommendation inference across Meta’s platforms.
• The 450 and 500 variants target generative image and video inference through 2027.

By shifting these high-volume inference tasks to custom silicon, Meta can lower data center costs while bypassing the traditional hardware supply bottleneck for its day-to-day operations.

Securing the AI supply chain

Meta’s modular, rapid-fire approach to custom silicon is aggressive versus traditional industry timelines. The company plans to drop a new iteration roughly every six months through 2027.

Meta is working with Broadcom to design Iris, while TSMC will manufacture the chip. But custom silicon is only one piece of the equation. Scaling AI infrastructure also requires a steady supply of memory, storage, and networking components at a time when demand for AI hardware continues to strain global supply chains.

To support that expansion, Meta has also been securing key components across its supply chain. The company has signed long-term agreements for high-bandwidth memory from Samsung Electronics, flash storage from SanDisk, and fiber-optic networking equipment from Sumitomo Electric.

The strategy mirrors similar investments by other hyperscalers. Google continues to expand its TPU program, while Amazon has developed its Trainium and Inferentia processors.

Scaling to 14 gigawatts

The Iris rollout is one component of Meta’s broader AI infrastructure expansion. The company plans to bring roughly 7 gigawatts of computing capacity online this year, then double that to 14 gigawatts in 2027. At that scale, Meta’s AI infrastructure would consume more electricity than many small countries.

At that scale, Meta’s AI infrastructure would consume more electricity than many small countries.

And, scaling AI infrastructure at this level comes with an enormous price tag. Meta has projected 2026 capital expenditures of between $125 billion and $145 billion, making it one of the largest single-year infrastructure investors in corporate history

Meta just pulled off something rare, which is essentially convincing Wall Street that spending more money is actually a good thing.

Wall Street rewards AI spending

Yet Meta just pulled off something rare: essentially convincing Wall Street that spending more money is actually a good thing. Following a trillion-dollar wipeout in tech market cap amid investor nervousness about the sheer scale of AI spending, Meta’s shares climbed roughly 8%.

With new MTIA chips planned roughly every six months through 2027, Meta is betting that vertically integrated AI hardware can deliver lower inference costs and better performance than relying exclusively on merchant silicon. By bringing chip design in-house and securing critical components across its supply chain, the company is slated to scale AI infrastructure with greater control over cost, deployment, and optimization.

The post Meta’s Iris push signals the next phase of AI infrastructure appeared first on The New Stack.

Researchers show that serving AI models with llm-d can boost inference speeds by up to 5 times and double throughput — all while using heterogeneous GPUs.

This sponsored article is brought to you by Wetour Robotics.

A field technician on a wind turbine, harness clipped, both hands on a wrench, needs to send a command to the diagnostic device hanging at her belt. A logistics worker on a loading dock, gloves on, eyes on the pallet, needs to redirect a connected lift. A person using an assistive mobility device on a crowded street wants to nudge it forward without taking out a phone or speaking aloud. None of these moments call for a smarter robot. They call for a smarter way to be heard by the machines that already exist.

The industry has been building from one side

The past three years of Physical AI have been a story of remarkable progress on the robot side of the loop. Companies like Boston Dynamics, Figure, and Unitree have advanced actuators, locomotion, and dexterity to a level that would have seemed implausible a decade ago. Google DeepMind’s Gemini Robotics has redefined what vision-language-action models can do in unstructured settings. The trajectory of the hardware and the foundation models is real, and it is accelerating.

But there is another side to this loop, and it has been treated as a solved problem for too long. The interface between humans and machines has defaulted, for 40 years, to three input modalities: screens, buttons, and voice. Each of those assumes the user can stop, look down, and translate intent into structured commands. That assumption breaks the moment the work moves into a real environment. On a turbine. On a dock. On a sidewalk. In any setting where hands are occupied, eyes are committed, or speaking is impractical, the conventional interface stack quietly fails.

Spatial Intent Fusion is the simultaneous processing of three streams of human-centered information, namely spatial position, visual context, and gestural intent: Your body is the interface.

The bottleneck on the human side of the loop is becoming as important as the one on the machine side. And solving it requires a different question. Not how do we make the robot more capable, but how do we let the human participate in the computing system as naturally as the robot already does.

Wetour Robotics’ bet: put the human back into the computing loop

Wetour Robotics is betting that the next architectural leap in Physical AI is not about making the robot more capable. It is about making the human a first-class node in the computing network, with the same kind of low-latency, high-fidelity participation that connected devices already enjoy.

Wetour Robotics’ engineers frame the problem this way: a wristband that recognizes a gesture is not enough. A camera that recognizes a scene is not enough. The information a human carries about what they are about to do is distributed across multiple channels, including where their body is in space, what their eyes are attending to, and what their muscles are preparing to do, and any single channel observed in isolation is ambiguous. Reconstructing intent reliably means fusing those channels at the operating system level, with latency low enough that the loop feels closed rather than mediated.

This approach has a name. Wetour Robotics calls it Spatial Intent Fusion: the simultaneous processing of three streams of human-centered information, namely spatial position, visual context, and gestural intent, fused into a single real-time command for any connected physical device. It is the technical implementation behind a simpler positioning statement the company uses externally: your body is the interface.

Orchestra is a portable intelligent hub running the operating system that handles sensor fusion, intent inference, command translation, and safety arbitration. The reference compute platform is NVIDIA Jetson Orin Nano Super, which provides enough on-device inference capacity to keep the entire control loop at the edge, with no cloud dependency on the critical path. Wetour Robotics

The architecture: three layers, four engines, one loop

Orchestra is not a single device but a layered platform, designed from the start to be sensor-flexible and actuator-agnostic. The architecture decomposes into three perception layers and four coordination engines.

Orchestra itself is the local compute and orchestration core: a portable intelligent hub running the operating system that handles sensor fusion, intent inference, command translation, and safety arbitration. The reference compute platform is NVIDIA Jetson Orin Nano Super, which provides enough on-device inference capacity to keep the entire control loop at the edge, with no cloud dependency on the critical path. Edge inference is non-negotiable for this application. Full-chain latency from biosignal acquisition to actuator command is held under 100 milliseconds, the envelope inside which closed-loop control feels natural rather than laggy.

VisionLink handles visual and spatial perception. Cameras feed into vision models that identify objects, estimate distances, and track environmental context. VisionLink is designed not as a passive recognition layer but as a real-time command generator: its outputs feed directly into Orchestra OS to be fused with biosignal data.

Conductor is the biosignal pipeline. It ingests raw surface electromyographic (sEMG) data from a wrist-worn device, classifies temporal patterns into discrete gestures or continuous control signals, and outputs actuator commands. The technically interesting property of sEMG for this use case is that the signal precedes visible motion. Motor unit action potentials appear at the skin surface roughly 50 to 80 milliseconds before a finger completes the corresponding gesture. Wetour Robotics calls this property pre-motion intent sensing, and it is what allows Orchestra to anticipate user intent rather than react to it.

On top of the three perception layers, Orchestra OS runs four coordination engines. The Perception Engine ingests and normalizes raw sensor streams. The Intent Engine performs Spatial Intent Fusion across modalities, resolving what the user is trying to do given where they are, what they are looking at, and what their hand is signaling. The Orchestration Engine translates intent into device-specific command sequences for any connected actuator. The Safety Engine arbitrates conflicting commands, enforces operational envelopes, and gates execution against runtime safety conditions.

The trade-offs we’re honest about

No system that bridges the human body and the digital world is finished. Three engineering challenges remain open, and the company addresses each with a deliberate trade-off rather than a claim of having fully solved it.

Baseline stability of sEMG under motion. In a stationary user, continuous gesture recognition from sEMG is reliable. Once the user is walking, climbing, or otherwise moving, motion artifacts and electrode drift degrade the signal in ways that are difficult to fully compensate for. Rather than overpromise on continuous control in dynamic settings, Orchestra defaults to a smaller set of robust discrete gestures in complex operating environments, and reserves continuous control modes for contexts where the signal-to-noise ratio supports them.

Miniaturization of edge AI compute. Running the Orchestra control loop entirely at the edge requires real on-device inference, which has historically meant trading off between compute capacity, battery life, and form factor. Wetour Robotics’ approach has been a compact carrier board paired with a thermal design and a battery module sized for all-day wearability. The result is a hub that travels with the user rather than tethering them to a desk, and that performs the full perception-to-actuation loop without offloading to the cloud.

Heterogeneity of third-party device protocols. The actuator side of the loop is a fragmented landscape. Different manufacturers expose different command interfaces, different communication stacks, and different safety conventions, and a Physical AI operating system has to integrate with all of them. Wetour Robotics uses an AI-agent layer to negotiate connection and protocol translation adaptively, so that Orchestra OS can ingest data from a wide range of devices, run them through neural network models that infer human intent, and emit the right command on the right protocol for the device on the other end.

Why this matters, and why it helps the rest of the field

The history of computing is a history of interface revolutions. Command lines gave way to graphical user interfaces, which gave way to touch, which gave way to voice. Each transition expanded who could participate in the system and what they could do with it. The next transition is not about a new screen or a new microphone. It is about treating the human body itself as a participant in the computing network, capable of contributing intent at the same speed and fidelity that any other connected node can.

The history of computing is a history of interface revolutions. The next transition is not about a new screen or a new microphone — it is about treating the human body itself as a participant in the computing network.

This path is not a competitor to the work being done on humanoid robots, foundation models for embodied AI, and dexterous manipulation. It is the missing complement to that work. The hardest open problem for humanoid systems is the data: every natural interaction between a human and the physical world is a potential training signal, and most of those interactions are currently invisible to any computing system. As more humans become first-class nodes in the loop, those interactions become observable, structured, and ultimately useful for training the next generation of embodied AI, including the humanoid robots being developed today.

In other words: putting the human back into the computing loop is not just about better interfaces for individual users. It is about generating the kind of grounded, in-the-wild human-machine interaction data that the broader Physical AI ecosystem will need to keep advancing. The robot side and the human side of the loop are not two competing futures. They are two halves of the same one.

That is what Wetour Robotics means when it says: Your body is the interface.

Learn more at wetourrobotics.com.