Google has joined Amazon in releasing a developer tool that makes it easy for PyTorch users to move their apps off Nvidia GPUs.

DeepSeek has open-sourced the inference engine it uses to serve its models in production, a step it said last year wasn’t yet practical.

Dubbed DeepInfer, the framework is designed to let developers run DeepSeek’s V4.1 Flash model on their own infrastructure, with many of the same optimizations the company uses behind its API. Support for future DeepSeek models is also planned.

“Running a model efficiently is a different matter altogether.”

DeepSeek has, of course, long made its model weights available for developers to download and run using third-party software such as vLLM. But running a model efficiently is a different matter altogether. The software that manages requests, distributes work across chips, and generates responses can have a significant impact on speed, throughput and computing costs. And so DeepInfer essentially gives developers access to DeepSeek’s own approach to those challenges.

“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.”

‘Making AI more affordable’

Taking to X on Tuesday, DeepSeek said it was making its internal inference technology available to outside developers in an effort to bring down the expense of running large AI models.

“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable,” the company writes.

🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!

Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more…

— DeepSeek (@deepseek_ai) October 20, 2026

Released under the Apache 2.0 license, DeepInfer includes optimized computing kernels and communication libraries, including DeepSeek’s previously open-sourced FlashMLA, DeepGEMM, and DeepEP, alongside mechanisms for distributing inference across multiple chips. It also supports speculative decoding, a technique that speeds up response generation by predicting and verifying multiple tokens at once, and asynchronous scheduling to reduce the time processors spend waiting for new work. The framework also supports structured outputs, allowing developers to specify how model responses should be formatted.

Alongside DeepInfer, the company is also open-sourcing deepseek-recipe, a companion library that handles how API requests are converted into model inputs, and how the resulting output is processed into responses. Available for Rust and Python, it’s designed to help developers reproduce the behavior of DeepSeek’s official API when running their own deployments. The library was quietly released in September, with an updated version published on Tuesday.

The announcement comes just days after DeepSeek released an updated version of V4.1 Flash, called DeepSeek-V4.1-Flash-1017, touting improvements in coding performance and efficiency. That release coincided with the company making its in-house web search technology available to developers through new Search and Fetch APIs, as per The New Stack‘s report on Monday.

Why DeepSeek held back in 2025

DeepInfer goes further than DeepSeek’s previous plans to share its inference technology. Back in April 2025, the company published a document outlining its plans for its inference engine, explaining that it had considered releasing the entire system before deciding to contribute individual components and improvements to existing projects instead.

At the time, DeepSeek identified several obstacles. Its inference software had evolved from an early version of vLLM into a highly customized system, and the company questioned whether its relatively small research team could maintain another major open-source project. There were also practical difficulties in making the software usable outside DeepSeek’s own infrastructure.

“The engine is tightly coupled with our internal infrastructure, including cluster management tools, making it impractical for public deployment without significant modifications,” the company wrote at the time.

So what’s changed now, exactly? Well, DeepSeek hasn’t explained how it overcame those earlier difficulties, although the scope of the new release offers some clues. The DeepInfer documentation makes clear that the publicly available software currently supports offline inference, with “the core features needed for serving at scale,” as the company puts it. However, several components needed to operate a complete production service are still missing. These include a KV cache management system, infrastructure for separating different stages of inference across machines, and load balancing across replicas.

In other words, DeepSeek has opened up much of the engine responsible for running its models, while leaving developers to supply some of the surrounding infrastructure required to operate a full-fledged service. That narrower release may help explain how the company has made its technology available despite the challenges it identified last year.

Nvidia and Huawei, side by side

While the release of DeepSeek’s inference engine is significant enough in its own right, arguably the more interesting tidbit is the hardware supported from day one.

“Arguably the more interesting tidbit is the hardware supported from day one.”

Nvidia is included, as expected, though support is restricted to its SM100 architecture, used in Blackwell data center GPUs. Notably, the other supported platform is Huawei’s Ascend 950 NPU series, putting the Chinese chipmaker alongside Nvidia in DeepSeek’s list of compatible hardware. Older Nvidia architectures, including Hopper, are conspicuously absent.

This also ties back to DeepSeek’s published open-source plans from 2025. While the company was reluctant to release its entire inference engine at the time, it committed to working with hardware manufacturers ahead of future model launches, with the aim of making its models available across different chip architectures from day one.

US export restrictions have complicated Chinese companies’ access to Nvidia’s most advanced chips, adding another dimension to DeepSeek’s growing ties with Huawei. In September, Bloomberg reported that DeepSeek was planning to deploy at least 160,000 Huawei Ascend 950DT chips at a new data center in Inner Mongolia, specifically to run its models. The company has largely relied on Nvidia for training, although The Information reported in September that DeepSeek also plans to increase its use of Huawei chips for training, with deliveries expected as early as the fourth quarter.

Later that month, Reuters reported that DeepSeek and Huawei were collaborating on software tools intended to make Ascend processors easier to program, further underlining the company’s investment in Chinese AI hardware.

The Huawei inclusion has already attracted attention among the wider developer community. In a discussion on Reddit, one commenter singled out the two supported hardware platforms as a highlight of the news, while another pointed to DeepSeek’s origins in quantitative finance as a possible explanation for its technical expertise.

“Once again Deepseek shows how good high-frequency traders are at optimizing performance… On Ascend NPUs nonetheless!!,” the commenter wrote.

Another user seemingly poked fun at Anthropic’s Claude Opus 5.5, following reports that requests involving kernel programming for Huawei’s AI chips could trigger safeguards that redirect certain advanced AI development tasks to an older model.

“Don’t let Opus 5.5 see this! It will delete the Ascend support,” the user joked.

It is worth noting that feature support differs between the Nvidia and Huawei implementations. DeepInfer’s batch-invariant inference and deterministic sampling features, which help ensure reproducible outputs under specified conditions, are available only on Nvidia hardware. DeepSeek’s published performance measurements also use Nvidia GPUs, with four SM100 GPUs used for prefill and 64 for decoding, leaving open the question of how the framework performs on Huawei’s chips by comparison.

For developers without access to either chip family, the immediate value may lie in studying the newly available code and adapting its optimizations to other inference engines. For now, DeepSeek has given the wider open-source community considerably more of its inference technology to experiment with.

The post DeepSeek open-sources the inference engine behind its API. And Huawei gets equal billing with Nvidia. appeared first on The New Stack.

'Dataflow Architectures for AI' is a 6-course specialization from IBM Research that compares leading accelerator architectures through hands-on labs.

Counterfeiters are getting better at replicating brands' marks and passing off dupes, known as "superfakes," as genuine items. A startup called Veritas wants to tackle the problem with its own chip that verifies goods.

Microsoft is positioning the new Surface Laptop Ultra as an alternative to Apple’s MacBook Pros for creatives and developers — and especially AI developers.

Built on Nvidia’s N1X-based RTX Spark platform, with a Blackwell RTX GPU with up to 6,144 cores, a CPU with up to 20 cores, up to 128GB of unified memory, and a price that undercuts Apple, it’s a tempting alternative if you want to run local AI models and also want to have a powerful machine for editing videos and gaming on the go.

This is The New Stack, so I’m leaving the gaming benchmarks and creator use cases to others. My focus in testing the unit was on running AI workloads.

I’ve been using the Surface Laptop Ultra in its high-end $4,300 GPU and CPU configuration with 64GB of RAM and a 1TB SSD for the last week, including while on the road.

Typically, I use a Mac Studio with the M4 Max chip at home, which happily runs most local models I throw at it (as long as they fit into the 48 GB of RAM). I installed all the usual tools I use to run local models on the laptop. Think Unsloth Studio, Ollama, Hermes Agent, etc.

At no point did I feel like I was making a compromise in running those on the laptop. Indeed, if anything, the Blackwell GPU was generally a bit faster than my Mac Studio.

Fast local inference (with the right models)

Since the new Nemotron 3.5 Bolt model launched earlier this week, I also took that one for a spin. It answered queries at an impressive 56 to 60 tokens per second — and sometimes more. That makes it a great model for running in Hermes Agent or OpenCode, for example. It’s a heavily quantized model with 9B active parameters, so you would expect it to be fast, of course.

Yet, while the laptop has the unified memory to run it, the dense Qwen 3.8 27B at 8-bit quantization never achieved more than 8 or 9 tokens per second in Unsloth Studio. That’s not quite usable, but with these dense models, that’s virtually always due to memory bandwidth. But bringing it down to a 4-bit quantization brought the speed up to a respectable 25 to 28 tokens per second.

Using the 8-bit Gemma 4 26B-A4B shows how the mixture-of-experts technique is really what you need to run virtually any model locally at reasonable speeds. For this model, the Surface Laptop Ultra generally hit around 40 tokens per second.

I don’t have an M5-based MacBook or Mac Studio at hand, but Tom’s Hardware reports that the Ultra generally outperforms the M5 Max in a MacBook Pro in AI use cases.

Performance drops when you unplug

All of those previous numbers were with the laptop on its 140W charger. Microsoft says it tuned the system so that running on battery would still give users most of the full performance. Microsoft claims its benchmarks showed that the laptop retained 99.6% of its performance when running on battery, but at least for the AI workloads I tested, that drop was far more noticeable.

The 4-bit Qwen 3.8 model, for example, dropped from its 25-28 tokens per second to around 18 tokens per second on battery. For Nemotron 3.5 Bolt, the performance dropped to around 42 to 45 tokens per second. Overall, it seems fair to expect around a 20% to 25% loss in performance when on battery when running AI workloads (and games show a very similar drop).

I did not run any formal battery life tests. Microsoft says users can expect 12 hours of active web usage. If you are consistently running local AI workloads, though, I wouldn’t expect more than a few hours of battery life based on my observations.

The RAM questions

Having 64GB of RAM puts you in a bit of an odd situation given that most smaller models right now are in the 27B to 30B league. The extra RAM gives you plenty of headroom for the KV cache and other applications, but there aren’t a lot of models in the 32-48GB range, for example, that the machine could also handle.

Going for 128GB of RAM brings the price of the Surface Laptop Ultra to $5,900

The trade-off for all that power: heat and noise

Unsurprisingly, all of the compute power (Microsoft says you get up to 1 petaflop of FP4 AI performance) creates a bit of heat.

Nobody is going to argue that the Surface Laptop Ultra is whisper-quiet when you run AI applications. The noise its dual fans make isn’t extreme, but enough to get a bit on your nerves while you are working. That’s especially noticeable when Hermes or your favorite coding agent is chugging along for a while on a more complex query.

There is a reason the Surface RTX Spark Dev Box, which is built on top of the same N1X platform, has a thousand air vents, after all (which will also likely allow it to push its performance beyond the N1X in the laptop).

Outside of that, the laptop is very quiet, though, and the fans remain off for most day-to-day use cases. It also quiets down very fast once the GPU load drops again.

A few more observations

I would feel remiss if I didn’t highlight a few other things I found while testing the Ultra. This is, after all, Microsoft’s most ambitious laptop yet.

• The touchpad is huge and has haptic feedback. There are only a few applications that support it so far, including DaVinci Resolve, for example. Having that feedback makes editing faster, as you can feel when the playhead snaps into place, for example. I want more touchpads with that feature.
• Whatever Microsoft did with the speakers works wonders. There are side vents near the front of the laptop that create a surprisingly wide stereo field, something you will definitely notice in games.
• The new magnetic USB-C charging connector is nice. It feels a lot like the magnetic Connect on other Surface devices. I just worry I’ll lose the cable sooner rather than later.
• Microsoft is not positioning this as a gaming PC, but with that GPU, people will try to play a few games, and so did I. Gaming on a Windows ARM machine is still in its early days, but thanks to Microsoft’s emulation, it generally works without too much overhead.
  Games that rely more on the GPU than the CPU will fare better, but I was able to play Cyberpunk 2077 in 2K with ray tracing turned on at around 60fps. With frame generation and a few settings tweaks, even 4K gaming is an option.
• With ray tracing off, the remaster of The Witcher 3 also happily ran at 60fps in 4K.
• But since the 15-inch, 3:2 touchscreen (3270×2180) is essentially a better 3K screen, you wouldn’t want to run a game in 4K anyway unless you plug in an external monitor.
• The screen looks great and can get very bright, but it is also highly reflective. That’s often distracting.
• Overall, Windows on ARM feels a lot more solid now than it did a year ago. Surely there are some applications and games that won’t run (or run well), but I haven’t come across any issues. Increasingly, developers (like OpenCode) offer Windows ARM64 versions of the desktop apps.
• Having two USB-C, one USB-A, and an HDMI port is such a joy if you have any older accessories. Add the headphone jack and SD card reader, and you don’t even need to carry a USB-C hub.
• I’ve gotten used to my MacBook Air. The Laptop Ultra is heavy at 4.41lbs (2kg).
• The SSD is replaceable. There are four screws underneath the laptop, and you don’t void your warrenty of you open the machine up. Apple should pay attention.

The post Running local AI models on Microsoft’s Surface Laptop Ultra: What we found appeared first on The New Stack.

Speaking at TechCrunch Disrupt, Amazon devices chief Panos Panay described a future where ambient AI devices work in the background, anticipating users' needs instead of waiting for commands.

Amazon-owned Bee has launched its first new AI devices since its acquisition, including an updated wearable and a credit-card-sized recorder that attaches to smartphones. The devices can summarize conversations, create to-do lists, and integrate with Alexa and Amazon shopping, with prices starting at $74.99.

A federal judge ruled that Apple can access OpenAI's interview materials involving former Apple employees, as well as private messages and testimony from key figures in the case.

Flox's edge devices can identify animals and use species-specific sounds to drive them away from critical infrastructure.

Open models and llm-d enabled IBM Research to explore quantum error correction hardware at 11.5× lower inference cost than an all-proprietary configuration.

One of the core promises of AI is that the more it knows about a person, the better it will be able to assist them with tasks and make appropriate suggestions, which, if you'll believe the tech companies, requires apps and devices to listen, watch, or record users' activity all the time and gather all possible context.

Over the past couple of years, we’ve seen multiple reports that Apple was working on a new smart home product, but now it seems the product has a name and a launch date.

For six years, Danu founder Amy Ma has been working on a better way to sort recyclable waste.

Upside Parts, a Massachusetts-based custom manufacturing company, released an analysis of more than 30,000 3D-printed parts produced across hundreds of customer orders, looking at how robotics companies use on-demand 3D printing. Robotics customers accounted for 82.3 percent of total part volume. The analysis comes as robotics production is expanding across the US, increasing the need […]

Natura’s $99 Interface smart ring lets you summon AI agents with the press of a finger to complete tasks, capture thoughts, and control devices — while doubling as a health tracker.

Microsoft revealed the specs and price for its Surface Laptop Ultra, AI PCs that run on Nvidia chips that are designed to run AI models and agents.

Q.ANT, a startup out of Stuttgart, Germany, builds processors that use light instead of electricity to do some of the math behind AI. The company pitches them as a way to run AI on a fraction of the power today’s chips need.

Now developers can start writing software for those chips without owning one. Q.ANT pushed a free, open-source software kit to GitHub this week that lets developers build and test programs on a normal computer, then run them on the real chips once they get access.

This is a move out of Nvidia’s playbook. Nvidia owes its lead in AI as much to CUDA, the software developers use to program its GPUs, as it does to the chips themselves. 

But with Q.ANT, the catch is the hardware. Q.ANT’s chips are running at a few research computing centers, and everyone else has to wait “the coming months” for cloud access through German provider IONOS or an on-site server from Q.ANT.

The kit, called the Q.ANT Native Computing Toolkit, is free on GitHub under a license that allows commercial use. Developers can work in Python or C. The key piece is a simulator that mimics the chip on a regular computer, with no Q.ANT drivers required.

What can it do today? The AI tools in this first version focus on running models that have already been trained. The examples read handwritten numbers, identify objects in photos and outline shapes in images. Training still happens on regular CPUs and GPUs.

The pitch for photonic computing is power. AI chips burn a lot of energy moving data back and forth between memory and the processor. Q.ANT’s chips do part of the math with light, specifically wave-shaped functions similar to a cosine, which regular chips calculate digitally. Q.ANT says AI models built around those functions get better results with fewer parameters, the settings a model learns during training. Fewer parameters means a smaller model, less data to move and less power. The kit includes examples comparing a standard model with one built Q.ANT’s way. Those comparisons are the company’s own.

“An ecosystem isn’t created by hardware alone. It emerges when the software layer is open and others can build on it,” said Michael Förtsch, Q.ANT’s founder and CEO. He calls the release the “Linux moment” of photonic computing.

Q.ANT is betting light can do the math itself. Lightmatter, one of the best-known companies in the field, now puts its focus on Passage, which uses light to move data between chips. The idea of light-based AI isn’t new, either. TNS covered MIT’s photonic processor for building optical neural networks back in 2017.

Q.ANT raised €62 million in July 2025 in a round led by Cherry Ventures, UVC Partners and imec.xpand. In March, it said its second-generation chips were running at the Leibniz Supercomputing Centre near Munich. The results it published from there compare the new chip with its old one: more than 50 times faster at the kind of math that does most of the work in AI models, and six times less energy on typical jobs, by the company’s numbers. Its bigger claims, like up to 30 times better energy efficiency, don’t say what they’re measured against.

Good software alone won’t carry a new chip. Nvidia has been building CUDA for nearly 20 years and is still adding to it, including deeper native Python support last year. Graphcore, the British AI chip startup, had its own software kit and still ended up being sold to SoftBank in 2024.

Q.ANT calls this the first openly available software kit for programming a photonic processor. That depends on how you count. Xanadu has offered free, open software for its light-based quantum computers since 2018. For now, developers can play with the simulator. What they can’t do yet is test Q.ANT’s power-saving claims on their own models. That has to wait until the chips open up.

The post Q.ANT gives away the software for its light-powered AI chips in a CUDA-style bet on developers appeared first on The New Stack.

The 1.58 in a 1.58-bit language model sounds like a hard limit, but Intel researchers pushed a ternary model below it by changing how its weights are stored rather than changing the model itself.

Their new BITCOS format compressed one checkpoint to 1.485 bits per weight and improved decoding throughput by as much as 18% on CPUs and 27% on GPUs.

The key is that the familiar 1.58-bit figure assumes a model uses its three possible weight values equally, while real ternary models contain far more zeros than that calculation accounts for. BITCOS stores the location and sign of each nonzero weight separately, allowing zeros to take up less space without retraining the model or altering its output — the equivalent of packing the same contents into a smaller box.

Where the 1.58-bit figure comes from

Ternary models use only three weight values — -1, 0, and +1 — and 1.58 bits is the theoretical minimum needed to represent three equally likely options. That number is cleaner than the reality of storing the weights, where the standard approach fits five ternary values into an eight-bit byte for an average of 1.6 bits each. Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.

When Intel’s researchers measured the distribution of weights across 29 checkpoints from seven ternary model families, they found that zeros accounted for between 29.7% and 51.5% of the weights. In 26 of those checkpoints, there were enough zeros for BITCOS to beat five-trit packing.

The sparsest was a ternary version of Qwen3-1.7B produced with CAT-Q post-training quantization, where 51.48% of the weights were zero, and BITCOS brought the storage cost down to 1.485 bits per weight.

Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.

How zeros save space

BITCOS stands for “BITmap and COmpacted Signs” and divides a model’s weights into two streams. The first assigns one bit to every weight to record whether it is zero or nonzero, while the second assigns a sign bit only to nonzero weights.

A positive or negative weight therefore consumes two bits, but a zero needs only the presence bit because it has no sign to record.

If z is the proportion of zero weights, BITCOS uses 2 − z bits per weight, dropping from 1.6 bits at 40% zeros to 1.485 bits at 51.5%. Because it changes only the storage format, unpacking restores the original -1, 0 and +1 values without affecting accuracy.

BITCOS becomes smaller than five-trit packing once more than 37.5% of a model’s weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.

BITCOS becomes smaller than five-trit packing once more than 37.5% of a model’s weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.

Making smaller weights run faster

Built for token-by-token decoding with small batch sizes, the format reduces the weight data moving through memory. Intel developed separate unpacking kernels for AVX-512 and AVX2 CPUs as well as Xe2 GPUs, joining other efforts to fit compressed models into faster inference pipelines for AI agents.

On AVX-512 hardware, the kernel uses the presence bitmap as a mask and pdep to scatter the compacted sign bits across the nonzero weight positions. Because Xe2 GPUs lack an equivalent instruction, Intel implemented the same operation with a 2KB lookup table.

Benchmarks across five systems

Compared with the 2-bit kernels, BITCOS ran 10% to 18% faster on the 64-core Xeon server and 2% to 15% faster on the 24-core Core Ultra 9. Performance improved by 9% to 22% on the integrated Arc 140V and by 2% to 27% on the discrete Arc Pro B70. These results measure decoding after the model has loaded, separate from efforts to cut GPU inference cold starts from minutes to seconds.

The smaller format did not win everywhere

On the eight-core Lunar Lake CPU, Intel’s fixed 2-bit kernel beat BITCOS on every model because the system had enough bandwidth to make unpacking the bottleneck. BITCOS remained faster on the GPUs, although decoding overhead limited the gains. Computer scientist and AI infrastructure author Chip Huyen has made the same point about inference more generally, arguing that the right optimization depends on whether compute, memory or bandwidth is holding back the workload.

On the eight-core Lunar Lake CPU, Intel’s fixed 2-bit kernel beat BITCOS on every model because the system had enough bandwidth to make unpacking the bottleneck.

Format limits and open questions

The paper has not been peer-reviewed; all five test systems used Intel hardware, and the end-to-end benchmarks covered seven models at batch size one. Intel has yet to test the format on Nvidia, AMD, or Arm hardware.

The post Intel squeezed a 1.58-bit LLM down to 1.485 bits without changing a single weight appeared first on The New Stack.

Running an LLM on your PC is easy enough, but putting an agent to work there is a different story. Portable Computer, the local version of Perplexity’s Computer agent, is now available inside the Perplexity app for Windows on compatible Nvidia GeForce RTX and RTX PRO GPUs.

That’s the good news; the catch is, you’ll need an Nvidia GPU with at least 24GB of VRAM.

The Windows launch gives Perplexity three platforms in less than three weeks. Portable Computer debuted on Linux and Nvidia DGX Spark on August 25, followed a week later by hybrid compute for Apple silicon, which splits tasks between local and cloud models on Macs. Now Windows joins the mix, but bringing Portable Computer over took more than simply porting the app. Perplexity had to adapt the model runtime, orchestration, security, and hardware integration for each platform while keeping the user experience the same.

you’ll need an Nvidia GPU with at least 24GB of VRAM to use it.

Orchestration beyond the model

Portable Computer bundles those pieces together. On Windows, it supports PPLX 27B — Perplexity’s post-trained model — and Qwen 3.8 27B, both optimized for RTX GPUs, alongside a built-in browser, tool calling,  and Perplexity’s proprietary SPACE sandbox.

It’s a different lane from LM Studio or Ollama, which make running models locally as painless as possible but stop well short of giving a model autonomy over multistep work. DeepSeek’s recent hiring spree of roughly 150 new roles, nearly all of them focused on agent infrastructure rather than the model, hints at how much engineering sits between a capable model and a capable agent.

DeepSeek’s recent hiring spree of roughly 150 new roles, nearly all of them focused on agent infrastructure rather than the model, hints at how much engineering sits between a capable model and a capable agent.

Connectors blur local boundaries

Perplexity ships connectors for Microsoft Outlook, OneDrive, and Word, plus Google Drive, Gmail, Slack, and GitHub — which tells you something about what “local” actually means here.

The agent can reach external services because it’s not air-gapped. Locally completed tasks can process files without sending documents to a cloud model. Once an agent has access to both local files and remote APIs on the same machine, figuring out which resources it actually needs and where to find them gets harder.

Hybrid cloud as fallback

Perplexity isn’t pretending that a 27-billion-parameter model running on a desktop GPU can handle everything, which explains the hybrid architecture. When the agent determines that a task needs more reasoning power than the local model can deliver, it can escalate to Perplexity’s cloud models.

According to Nvidia, the agent identifies when cloud support would help and asks the user for permission before sending any data off the machine.

For organizations handling sensitive or regulated data, that split can make all the difference. A local agent can grind through source code or financial records without uploading them to a hosted model for basic processing. There’s a cost angle too, since tasks completed locally don’t burn Perplexity Computer credits.

High VRAM floor limits reach

Portable Computer is available with Perplexity Pro ($20/month) and Max ($200/month), across individual and enterprise plans, with Nvidia DGX Station support coming later. The real challenge is taking local agents from developer passion projects to enterprise-ready tools. By baking this into Windows, it immediately gets in front of the scale of users needed to make that happen.

The real challenge is taking local agents from developer passion projects to enterprise-ready tools.

The post Perplexity’s new agent runs entirely on your GPU — with one expensive catch appeared first on The New Stack.

Perplexity wants its AI agents to use more of the computing power already sitting inside a Mac. The company launched Hybrid Compute on Tuesday, a new feature that lets Perplexity Computer move parts of the same task between powerful cloud models and smaller models running directly on Apple silicon.

The timing is significant. Hybrid Compute arrives on the first day of John Ternus’ tenure as Apple’s CEO. Ternus, who previously led Apple’s hardware engineering organization and played a key role in the company’s transition from Intel processors to its own Apple silicon, succeeds Tim Cook after 15 years as CEO. Perplexity is now betting that same hardware can become part of the infrastructure behind autonomous AI agents.

The task starts in the cloud. When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without having to start over.

When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without starting over.

Privacy Gate checks first

A Perplexity-trained Privacy Gate runs locally on the Mac and looks for sensitive information such as names, addresses, account numbers, and secrets. When it flags something, the user can decide whether that part of the task should stay on the device.

Users can review what the system wants to keep local before work begins and ensure it hasn’t missed anything they don’t want sent to the cloud. They can also choose which model handles the local work.

At launch, users can choose between Gemma E4B, Qwen3.6 35B-A3B, and a version of Qwen3.6 35B that Perplexity post-trained itself, with more models planned for later. Perplexity also handles the installation through the desktop app, so users don’t have to open a terminal or set up the model themselves.

Local tokens cost nothing

Once a task is running, the app displays local CPU, GPU, and memory utilization, as well as the number of tokens consumed. Users aren’t charged for tokens generated by models running locally on their Mac. That gives Perplexity another reason to move work onto the device beyond privacy, since running models locally can also cut inference costs. (The question of who bears inference costs and when is becoming a competitive issue across the industry.)

There is a compromise. Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability. Hybrid Compute leaves that choice to the user.

Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability.

Context crosses the boundary

The trickier bit is what happens to context when part of a task moves onto the Mac. The cloud model still needs to know enough about what happened locally to continue the job, without getting access to the private information that was supposed to stay there.

Perplexity says Computer can move a step from the cloud to a local model without restarting the task or losing context, and ultimately combine the cloud and local work into a single result. It does not detail in its announcement exactly what context is passed between those environments or how information produced by the local subagent is filtered before it returns to the broader workflow.

The local subagent can work with private files and data and take actions on the Mac. Users can also start a task on an iPhone and hand off local work to their Mac without having to start over.

For enterprise customers, Perplexity adds company-wide rules for what stays local and a record of what leaves each device, bringing the same governance questions facing AI agents down to the device level.

DGX Spark starts local

Hybrid Compute reverses the approach Perplexity introduced for Nvidia’s DGX Spark last week. DGX Spark starts locally and reaches out to frontier cloud models only with permission, while the Mac version starts in the cloud and moves work onto the device when needed.

In both cases, the agent harness decides where each part of the job runs. That orchestration — deciding which tools and context an agent actually needs — is becoming an increasingly difficult engineering problem as agents gain access to more systems and data.

Apple silicon becomes part of the agent stack

Running this much of an agent locally still requires a fairly powerful Mac. Perplexity recommends at least 32GB of unified memory, Apple silicon, and macOS 15. The feature is available to Pro and Max subscribers as well as enterprise customers.

Those requirements show the limits of local AI at the moment. Smaller models can run on plenty of Macs, but giving an agent enough compute to handle meaningful work still requires relatively high-end hardware.

That will likely change as Macs get better at running larger models. For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.

For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.

The post Your Mac is now part of Perplexity’s AI infrastructure appeared first on The New Stack.