Compute is the lifeblood of AI. He who controls the spice controls the universe the compute will control the production of tokens and reap the benefits of AI. Without compute you do not have a seat at the table. The United States technology community is all in on compute and AI as the next platform and is now adding compute at a staggering pace.

There is competition, and it not only comes from companies but from countries, and the US government has placed a series of export controls to limit China’s rising compute. Today the US controls and is the undisputed leader in compute with more than 70% of the worlds deployed FLOPs. One way to stay ahead is to keep going full steam while hindering your competition. Limiting your competitor nation state from compute, which will limit them from intelligence, is the current policy to stay ahead in the AI race.

These moves have led to backlash, including with China cutting rare earth minerals and magnets off from the US. Secretary of Commerce Howard Lutnick says the resumption of Nvidia GPU sales to China were required to restart China’s shipments of their linchpin supply chain materials.

But constraints have also led to adaptation, and Chinese companies have adapted. High batch sizes and disaggregated serving are but two examples. Despite advances, in the case of DeepSeek, most of their tokens are still inferenced on western hardware. We wrote about this dynamic in our recent DeepSeek debrief. Training of DeepSeek’s next generation model was also delayed by the use of Huawei chips, as we also said in the debrief.

This is not a stable equilibrium. There are always moving pieces in the race for intelligence. Beijing plans for the long term and knows it must secure its own domestic compute destiny. There is an irony: in the 2010s China kicked out Google to enforce its Great Firewall and foster its domestic industry, this time the US government is withholding hardware technology so they cannot seize the lead in AI.

We believe that at China’s core, they want to control not only its internet and AI, but the hardware that supports it. From silicon to tokens, China seeks sovereignty over every layer of the stack, and given recent history will never want to be beholden to foreign powers. Enter Huawei.

China loves national champions, and in it’s characteristic capitalism, tends to funnel resources to a few national champions. Today’s champion is probably Nvidia’s greatest adversary: Huawei. We expect Huawei to be able to make millions of chips this year, and to be bottlenecked by HBM next year. Today we want to talk about Huawei, which can translate as “China’s achievement”.

All In with Huawei’s chips

Huawei is the key piece for China’s compute destiny. Huawei’s chip ecosystem is vertically integrated and capable network of tools, fabs, and design that allows it to express a full stack vision of hardware. This hardware is very impressive although less efficient then western hardware.

Our previous investigation into Huawei’s fab network is a great example of their broad reaching network.

Huawei wants to vertically integrate the entire manufacturing process. The goal is to own not just the manufacturing of the logic die, the brains of the chip, but also the memory and packaging. They have even created their own tool company, SiCarrier to copy foreign firms tools. Huawei has bought over $9B of tools to put in their own fabs as well as reverse engineer to replicate.

Their efforts are not to be underestimated. SiCarrier, for example, has recently raised $2.8B in funding. That money is going into building fabs dedicated for Huawei staffed by Huawei employees. Some reporting on these fabs, which we believe to be Huawei owned and operated, suggests that their combined production by next year could entirely exceed SMIC, the current leader and where production is currently outsourced. With Huawei ramping up their own efforts, allocation at SMIC can be freed up for other chips, including Cambricon. The Cambricon chip is itself popular among Chinese companies, especially ByteDance.

Huawei operating their own fabs will represent a material increase in Chinese production, but also in their ability to iterate, control, and improve on processes. Huawei and SMIC will directly work on increasing yields, refining R&D for the next node, and bolstering Chinese semiconductor manufacturing capability.

Currently, all high volume chip production is outsourced to SMIC, the leading Chinese pure-play foundry. This includes the Ascend series of accelerator chips along with the Kirin mobile processors. Yields are poor for SMIC’s 7 nm-class processes due to a combination of immaturity, export controls, and the inherent difficulty in yielding large die such as the Ascend. Thus a relatively low percentage of SMIC’s overall capacity is allocated to producing Ascend die, since smaller mobile processors just make better business sense at this point. But this can change quickly. Let’s discuss what is possible for Huawei….

Huawei Production Numbers and SMIC’s Ramp

Our data shows Huawei’s production is 507k Ascend units shipped in 2024, the majority of which are 910Bs, and 805k this year, 653k of those being 910C. The 910C is the more advanced version. This includes die made by TSMC and SMIC.

SMIC, hampered by export controls, struggled to get production off the ground. But Huawei got Ascend die fabricated at TSMC while SMIC ramped. This was a violation of export controls and Huawei ended up receiving more than 2.9M Ascend die, which can be used for both 910B and 910C. We detailed this here.

It is specifically this “Die Bank” of foreign chips from TSMC that gets them through 2024 and 2025. Without this Die Bank, Huawei’s Ascend production numbers would be much lower.

We expect that the TSMC die bank to run out within the next 9 months. SMIC, however, now has more than enough capacity to produce meaningful volumes of chips. We forecast that SMIC will no longer be the bottleneck for Ascend production, as they will have sufficiently ramped capacity by end of year.

The chart above shows a baseline scenario with a modest increase in SMIC capacity allocation for Ascend. It requires at most 20k wafers per month (wspm) of SMIC capacity to produce millions of Ascend die per month.

For reference, a conservative estimate of SMIC’s total advanced node capacity (7nm and below) is 45k wspm by end of 2025, increasing to 60k wspm in 2026 and 80k wspm in 2027. In addition, Huawei is building their own fabs, not all of which are export controlled, and collaborating with SMIC on process technology, so production could ramp higher for advanced process technologies.

If 100% of capacity was allocated to Ascend die, their production capability would be in the tens of millions per year. They are on track to be more than capable of supporting a large domestic demand for China-produced compute die.

Our forecasts above use a conservative estimate for both yield and its rate of improvement in the future. It’s very likely that SMIC can exceed these estimated as its 7nm nodes mature. Our estimates for yields are lower than TSMC, Intel, Samsung, ASE, Amkor, etc for front end wafers and packaging.

Indeed, yield is an important lever that can be pulled to increase production without giving up more allocation. With just small increases in yield beyond our forecast above, SMIC will be able to produce several million Ascend die at lower allocation amounts than they would otherwise. Every percentage counts and SMIC has their best engineers working on exactly this problem.

In other words, SMIC only needs small amounts, as low as single digit percentages, of allocation to produce more than a million die as soon as early next year. Production to a couple million is possible, more allocation is all that is needed. For the reasons we articulate above, we believe the reported number of 200k Ascend chips to be significantly off the mark.

While SMIC is expanding and is no longer the bottleneck and Ascend production, Huawei’s long-term ambitions include its own fab network buildout. We noted in our Fab Whack-A-Mole report:

Huawei is clearly taking full advantage to the tune of $7.3B of WFE expenditure in 2024, up 27% year-on-year. They’ve gone from effectively zero in 2022 to the 4^th largest WFE customer globally in two years.

Since we published that report in fall of 2024, the Huawei fab buildout has, if anything, accelerated. Their ecosystem of semiconductor-related shell companies is expanding. Massive new cleanroom buildouts are in progress. And improved domestic options and possible diversion of $30B+ of imports of wafer fabrication equipment means they are likely able to equip these fabs as well. The details are worthy of their own report, but suffice to say Huawei continues with a large, concentrated effort to own every vertical in the Ascend supply chain.

TSMC Access

Huawei currently makes many mobile chips, but there is zero reason to do so from a strategic geopolitical perspective. Oppo and Xiaomi currently fabricate mobile SoCs at TSMC. They are ramping up their own designs to decrease dependence on companies like MediaTek and Qualcomm.

Huawei can reduce their mobile SoC production while not consuming, in any meaningful volume, SMIC allocation. The continued access to TSMC for other Chinese entities decreases the pressure for SMIC to make mobile SoCs, meaning more can be allocated to AI chips.

Huawei and SMIC can ramp up AI production without having to worry about serving all the nation’s demand for mobile chips due to other companies’ access to TSMC.

Export Control Lag Benefits SMIC

Another part of SMIC’s expansion strategy is stockpiling considerable amounts of semiconductor tooling. The controls are released on an announced schedule and it’s usually easy to anticipate what will be included.

This is due to timing differences in controls that they can take advantage of. For example, the U.S. routinely exempts Japanese and Dutch companies from its equipment export controls. Ostensibly this is done because they are allied countries and have their own export control regimes and semiconductor industries.

The problem, however, is that when new export controls come out, Japan and the Netherlands do not immediately follow. In many instances, matching controls are delayed by 6 months or never. Chinese companies can rush order years’ worth of equipment to stockpile, while American vendors are shut out of the process.   Japanese WFE vendors are happy to sell into this gap with margins fattened by order expedites. Many key suppliers are seeing well above 40% revenue share from China:

In cases where controls do follow the US, Japan and the Netherlands do not have any matching controls on re-exporting. This means that it is possible for restricted tooling to reach China if it goes through a different country first.

In addition, the use of “advanced ICs” as the bar for restricting tools leaves room for issues. ASML’s NXT:1980 scanners are perfectly capable of 7nm-class logic, and likely beyond if economics (throughput and yield) can be ignored or subsidized. These are allowed into China and even to certain SMIC facilities.

Export controls, as noted, can be expanded in addition to very stringent enforcement of existing mechanisms. This includes tighter coordination with allies on timelines for matching controls and getting buy in on re-exporting.

We are broadly encouraged by the administration’s Action Plan. As an example, we are pleased to see controls on semiconductor subsystems being called out. This is a drum SemiAnalysis has been beating for a while and think it is the right direction. With that said, many subsystem firms that supply western players such as VAT Group in countries like Switzerland without controls will not be stopped from shipping critical chambers to China.

The Action Plan also included concrete suggestions around aligning protection measures globally, which we think if implemented, will greatly ameliorate the issue of alliance’s lags in export controls described above. The issue is that SMIC and CXMT are able to continue to expand production because the sanctions on them are flimsy. International co-operation, especially with partners in Korea, is key to get right and ensure that Chinese commercial bottlenecks are not eased. Korea manufactures a huge amount of memory and Samsung has historically supplied large amounts of memory into China, so tight alignment of goals and enforcement is critical.

The other part of the supply chain that is just as critical is memory. This is where we believe the key constraint lies.

HBM Is the Bottleneck

We believe HBM production is the bottleneck. China does too, which is why they have asked US officials to relax controls on HBM as part of the recent trade talks. What the ask omits is telling: it does not include more TSMC access or lithography tools. Beijing is specifically asking Washington to loosen HBM restrictions.

Much like how Huawei was able to stockpile TSMC logic wafer inventory, they were also able to stockpile HBM inventory. Samsung, due to their failures in entering the accelerator supply chain for Western chips, sold their product to Chinese customers that funneled this inventory to Huawei. This is why it is so critical to work with Korea on enforcement of memory controls moving forward.

Samsung alone has directly provided 11.4 million stacks of HBM to China, including a staggering 7M stacks in the 1-month gap between controls announcement and enforcement dates. When including other providers and methods of shipment, that is 13 million stacks of HBM.

Specifically, on December 2^nd 2024, the Bureau of Industry and Security (BIS) announced controls on anything more advanced than HBM2E. Full compliance was required on December 31^st 2024. Samsung exported as much as possible to China in that one quarter. This comprises the majority of China’s HBM. They were able to achieve this due to the US Government and media telecasting the restrictions for many months before they came out.

After the ruling, Samsung HBM still made it into China. We previously detailed how companies like CoAsia Electronics and Faraday supplied non-functional chips with HBM into China, though we believe due to our efforts in exposing this both privately in late January and publicly later that it has now stopped with revenue numbers returning to normal. This does not mean that HBM smuggling has stopped entirely, though. There could be other sources.

In sum, China has procured 13M HBM stacks which is sufficient for 1.6M Ascend 910C packages. Despite this, we expect that China will be bottlenecked by HBM by the end of the year as they run out of foreign HBM.

China can easily make more than 805k Huawei Ascends this year from TSMC and SMIC capacity, but they will not because they do not have enough HBM.

We expect SMIC’s production to be 1M 910Cs and almost half a million 910Bs this year, however, not all of them will be turned into ASICs as the HBM is not available. If some HBM enters through smuggling, then Huawei can produce more Ascend AI ASICs.

Without Foreign HBM, China has no Domestic AI Accelerator Industry

Without access to more foreign HBM, Huawei will not be able to fabricate even 1 million Huawei Ascend chips next year. They must rely entirely on domestic production which we will detail below. Nvidia and AMD have effectively 0 competition in China once these HBM banks run out.

The other option available to China is utilize slower GDDR and LPDDR memory, but this is not suitable for the leading language models with modern reinforcement learning techniques or for large scale inference deployments.

Domestic HBM Industry – CXMT

China’s main DRAM player, CXMT, has caught up with the west rapidly. This is due to a combination of extremely strong domestic engineering capabilities, poaching engineers from Samsung, SK Hynix, and Micron, as well as the leading tool vendors Applied Materials, Lam Research, and Tokyo Electron teaching them sub-processes.

CXMT is able to ship DDR5 memory, only a couple years behind SK Hynix, Micron, and Samsung, and is winding down profitable DDR4 production previously reserved for PCs and mobile. While they have not shipped much HBM, their roadmap is aggressive. By next year, their production capacity will rival that of Micron’s, though not fast enough to save Huawei Ascend’s production. In 2026, we expect them to be producing 257k WPM, which would be just under 15% of global DRAM production. In our estimates, this scales to 490k in 2030.

CXMT’s shift in focus and production ramp are driven in part by investments from the CCP. China’s “Big Fund III”, which started in May 2024, invested $2B into the company. CXMT is also expanding in Shanghai as well as Beijing, with HBM packaging subsidiaries being set up in the former.

Expecting, though ultimately averting, an entity listing by USG, CXMT stockpiled years of tools in 2024 and is likely still adding HBM-specific gear. CXMT is still not entity listed! While HBM was noted as a key focus in that salvo of controls, the previous administration failed to include China’s HBM champion. The Trump administration needs to solve this failure from the Biden administration immediately.

The stockpiling matters as advanced tools, like Hanmi’s TCB systems for HBM3, are restricted, but they are not that instrumental. Older systems can be run slower without impacting cost too much. More importantly, CXMT can still procure leading edge equipment for Through Silicon Via (TSV) formation – that is critical for making HBM -through Japanese suppliers. In addition, Chinese OSATs such as JCET and Tong Fu are also racing ahead with their R&D efforts and building capacity for the critical TSV and stacking processes to package front end HBM wafers from CXMT. This is unlike the Western memory incumbents who have vertically integrated these processes. This is a typical example of Chinese industrial development where multiple players are encouraged to develop domestic manufacturing capabilities: creating cutthroat competition that accelerates the speed of development. It is therefore critical to not just focus controls on CXMT, but the whole of China as critical process steps can always be outsourced.

Because front-end logic is currently not the binding constraint, Beijing’s main asks in the trade deal target HBM and the relaxation of controls relating to this set of tooling. Given the importance of HBM, it is critical to understand CXMT’s future production depending on various allocation scenarios.

CXMT Production Forecast

This can play out in several ways, depending on how much wafer capacity is given to HBM. China can easily produce tens of millions of stacks with less than 50% of wafer capacity. CXMT is at a bit over ~250,000 wafers a month of production capacity and is expected to reach 300,000 wafers by the end of the year.

Currently they have not built out the tooling required to convert standard DRAM production lines to HBM, but it is inevitable. Properly designed, targeted sanctions can slow this conversion massively.

Different scenarios lead to different amounts of Huawei Ascends produced. As noted, SMIC can produce the die needed to match the HBM.

To be clear, this could change. The rate of production for CXMT could increase if they continue stockpiling key tooling or significantly improve yields. The might of China’s manufacturing capability and capacity knows no bounds. This estimate is somewhat conservative, it is likely that CXMT is able to produce the significantly more capable HBM3e in 2026.

We believe CXMT will only be able to make ~2 million stacks of HBM next year, which is only sufficient for 250,000-300,000 Ascend 910C’s. Yields and capacity conversion will take some time to improve for CXMT to commit significant capacity.

If all leading edge logic die capacity could be paired with HBM, Huawei production of Ascend would grow from 805k this year to 1,175k in 2025. More importantly, production next year would grow from 300k to over 5 million Ascend 910C!

Our analysis indicates that export controls have been effective in constraining and limiting Chinese chip production capabilities. Assuming no smuggling, China will be able to make less Ascends next year, not more. CXMT is squeezed tight. Had controls not been present, the Ascend ramp would be fully realized, Chinese models would be served on Huawei Ascend at scale, and there would be enough compute capacity that advanced models like DeepSeek R2 and V4 would already be here. Not to mention, with more capacity, China would be better positioned to export their AI on their chips.

As such, it is absolutely critical to ensure the application, enforcement, and continued updating of export controls to prevent CXMT and related entities from ramping production. As mentioned, this includes not just CXMT but OSATs and subsidiaries they work with. Second, it is also important for the intelligence community to track and identify any instances of HBM smuggling, like the Faraday + CoAsia scheme which we disclosed privately in January and publicly reported on later.

By no means should HBM be allowed to be shipped into China. Production of AI chips over the next few years is heavily gated by CXMT’s ramp and foreign shipments of HBM.

There is very strong incentive to find ways to ship HBM into China. This is why enforcement of controls are so critical.

There is another pillar to China’s production strategy, separate from memory and logic. This pillar is about networking the chips together and where they are made.

Networking and Datacenter CPU

Chips do not exist in isolation. We have argued that the system matters more than the microarchitecture for years. Clusters are comprised of many tens of thousands of interconnected chips and how they are interconnected matters.

We detailed Huawei’s CloudMatrix 384, or CM384, system below.

We believe the networking equipment used, specifically the scale up switches, are being made at TSMC and not at SMIC through shell companies. We also think they are stockpiling this equipment.

We believe Huawei has also been able to manufacture their datacenter CPU at TSMC. This showcases that current controls aimed to limit Huawei’s access to TSMC are insufficient. The Ascend AI ASIC is 7nm, but because of poor screening, Huawei has been able to procure technology from the more advanced TSMC 5nm node.

By producing these chips at TSMC, this frees up capacity at SMIC, meaning more allocation can be given to the Ascend or Cambricon programs (HBM permitting). This, as mentioned earlier in the report, is a critical variable. If SMIC had to make the networking equipment, this would squeeze the wafer allocations towards Ascend significantly.

There is all the incentive in the world for China to offload as much production as possible to TSMC. They have used shell companies before and there is ample evidence they are still doing it. While some ideas around KYC and red flags have been implemented, we believe that existing frameworks are insufficient.

Nvidia and Huawei: Blackwell and the H20

The US government previously banned the Nvidia H20 chip from going into China, but recently indicated that Nvidia will be granted a license to be able to export the chip. As we have previously noted, the H20 is a capable chip with more memory than the banned the H100, though it has less much fewer FLOPs. Having more memory is helpful for inference performance.

Nvidia is poised to sell its existing inventory of the H20 and H20E (a variant with even more memory) at a minimum. This will amount to more than several hundreds of thousands of chips and billions of dollars of revenue.

There have also been reports of a more advanced version based on the Blackwell series. This is slated to have just as much memory as the H20 but will also contain significantly more FLOPs. Specifically, the B30A could have more than 10x the FLOPs than the H20, significantly above the export control threshold. The B30A could be positioned at half the price and have half the performance of the B300. The solution to this is rather simple: just buy 2x the chips for equivalent performance. Given how strong and weak scaling work in machine learning workloads, this is a valid path to remain competitve.

The B30A is argued as needed to be shipped to China due to decreased interest from China in the H20. A lot of this is due to government pressure of China’s major tech firms. While there is significant messaging that China has decreased interest in the H20, we do not believe this to be true.

Specifically, we do not believe this is reflective of China’s demand for compute or foreign chips. It is an incorrect policy being pushed from the top down which will be reversed as soon as Huawei runs out of HBM to produce more Ascend chips, or it could even be orchestrated brinkmanship to get approval to a more powerful chip.

This is on the tail of other news, like DeepSeek unable to get acceptable performance on Huawei’s chips. That is no coincidence. Export controls are working and the biggest blocker for more progress is compute.

China may be trying to psyop their way into getting the significantly more performant Blackwell chip. While the H20 and the H20E have much, much better software than the 910C, it is arguable that they are in the same league. However, it is unquestionable that the B30A would be in a league of its own. Given that the H20, H20E, and especially the B30A are good at inference, this also allows Chinese companies more compute to serve their models and applications.

It will allow them to export Chinese AI on hardware they own. This will increase the proliferation of Chinese AI and Chinese apps, inevitably taking market share away from American applications. The American AI stack is American AI on American chips, not just the chips.

The US needs to balance having China on the US AI stack and slowing the development of their own, while also limiting the quality and volumes that are shipped into China.

The H20 is allowed into China, but the progression of shipping more powerful chips into China should be closely guarded and watched considering China’s domestic capabilities.

The US should only move the quality of a chip shipped to China when it becomes clear China can ship something competitive to the H20E in significant volumes.

Regardless of the exact SKU, the export licenses have several implications discussed below.

Implications of Compute Diplomacy

First, it means that there will be more chips for top players like DeepSeek and Alibaba to use for Reinforcement Learning (RL). RL is a big driver of progress right now. The majority of RL compute is inference, which the H20 and especially the H20E, are good at. The B30A will have even better performance.

Players like DeepSeek had enough compute to keep improving – R1 received a major update with performance increases on a timeline that is like o1 -> o3. We firmly believe GPT-5, Claude 4, and Grok 4 are considerably ahead of R1 though, especially in agentic tasks.

However, problems in AI are solvable by two things: talent and compute. DeepSeek has always had the former, now they will have more of the latter. We expect DeepSeek’s rate of progress to materially increase as they receive substantial volumes of any chip that gets into China. DeepSeek has ambitions to release a multimodal model in V4, but scarce compute is slowing progress.

Alibaba (Qwen) and Moonshot (Kimi K2), have also yet to produce a large multimodal model, primarily focusing on text only due to compute limitations. They will release multimodal models this year, but they are still squarely behind OpenAI, Anthropic, and Google in many capabilties. If Blackwell GPU shipments come too early, that will accelerate China’s pace.

The second major implication of this is that China will now have more compute to serve and inference models to its population. Compute constraints affect user experience, with DeepSeek deliberately serving R1 at low speeds to users to preserve compute.

Poor user experience from the lack of compute has greatly limited China’s ability to deploy AI. As user experience improves, so will adoption, and the economic benefits from AI. It is easy see a future in which models are pre-trained via H800s, post-trained on H20s, then served to the population on Ascends and H20s. There is so much demand for AI powered products that should more capacity to serve these models exist, it would immediately be satisfied. One of the reasons Chinese models are open sourced is that it allows for other people to serve their models for them. With more capacity, models can be closed sourced and dependence on American providers cut off.

Claims that models trained on one chip must be inferenced on the same type are entirely false. Anthropic did different stages of research and training for Claude 4 on GPUs and TPUs. Additionally, Claude 4 inference is offered on Nvidia GPUs, Google TPUs, and Amazon Trainiums. This complexity is all while Anthropic is also the fastest growing AI company in terms of revenue with by far the most capable model for software engineering.

DeepSeek, Alibaba, Moonshot, etc primarily train their models on Nvidia chips and we do not think this will change anytime soon. If the Blackwell version ships, gains across all levels will be more pronounced.

China’s compute at a snapshot

With the H20 approved, this represents a material increase in the number of FLOPs and memory that China has access to. The 910C will be the first meaningful effort of indigenous production in China in terms of realized FLOPs and memory. We also expect some re-exportation into China by various bad actors, constituting meaningful volumes of H100 and low volumes of B200.

Should the Blackwell version ship, gains around FLOPs will be even more pronounced as the chip has significantly more FLOPs. China will end up having more FLOPs in that scenario, in addition to more memory.

This is not to mention that Chinese players, including major ones like ByteDance, can still access compute through renting in non-restricted countries. For example, ByteDance can still access top of the line Blackwell GPUs from providers like Oracle and Google. We have previously covered this relationship in detail. Malaysia, in no uncertain terms, has become a huge market for Nvidia. Export controls were also effective in ensuring that it remains that way, with Malaysia immediately distancing themselves from any effort around the Huawei Ascend.

Renting enables a path for Chinese dependence on Nvidia without Nvidia chips being in China, which is possible because the chips do not need to be in China. Bytedance’s Seed models were trained in the US on a US cloud.

Inspecting GPUs is much easier in Malaysia than in China. As such, it is an effective way to limit any unauthorized re-exporting by bad actors, as we believe methods like location tracking are technically intractable and easy to bypass. Note that renting will still allow China access to the capability of training advances models. If allowed to be used for serving the models, this allows China the chance to proliferate their AI through new and improved AI powered applications, taking market share away from American products. The critical difference, though, is that access can be cut off.

With that said, China will not allow their data to be shipped out of China in large volumes hence there is still significant need for domestic AI capabilities. Selling chips into China does not change China’s drive for total silicon self sovereignty and will only serve to help provide a buffer while local production meets demand.

It is worth remembering that China’s focus on silicon self sovereignty predates US export controls. The argument Blackwell needs to be sold into China is a false narrative as Huawei will soon run out of HBM. Selling chips into China only served to bail them out until domestic production ramps. The production ramp will not slow until self sufficiency is achieved. Shipments of Blackwell must be weighed with a close eye on production of domestic capabilties. If Huawei, Cambricon, and CXMT accelerate production faster than expected, then the US should raise the bar sooner.

Next, we dive into Nvidia’s future in China. This includes expected revenue from the H20 and newer China specific chips on the horizon. We also compare how much compute China has relative to the US how the is absolutely dominating China when it comes to both memory and FLOPs.

Frontier model training has pushed GPUs and AI systems to their absolute limits, making cost, efficiency, power, performance per TCO, and reliability central to the discussion on effective training. The Hopper vs Blackwell comparisons are not as simple as Nvidia would have you believe.

In this report, we will start by present the results of benchmark runs across over 2,000 H100 GPUs, analyzing data on model flops utilization (MFU), total cost of ownership (TCO) and cost per training 1M tokens. We will also discuss energy use, examining the energy in utility Joules consumed for each token trained and compare it to the average US household annual energy usage, reframing power efficiency in societal context. We will also show the results of this analysis when scaling the GPU cluster from 128 H100s to 2048 H100s and across different versions of Nvidia software.

Later in this report, we will also analyze GB200 NVL72 benchmark results across Llama4 400B MoE and DeepSeek 670B MoE and compare this data to our earlier results from the H100. We will discuss whether the GB200 NVL72 performance per $ advantages survives once reliability issues are factored in.

Downtime from poor reliability and lost engineering time is one of the main factors that we will capture in our perf per TCO calculations. Currently there are no large-scale training runs done yet on GB200 NVL72 as software continues to mature and reliability challenges are worked through. This means that Nvidia’s H100 and H200 as well as Google TPUs remain the only GPUs that are today being successfully used to complete frontier-scale training. As it stands today, even the most advanced operators at frontier labs and CSPs are not yet able to carry out mega training runs on the GB200 NVL72.

With that said, every new architecture naturally requires time for the ecosystem to ramp software to effectively utilize the architecture. The GB200 NVL72 ramp is slightly slower than prior generations, but not by much, and we are confident that before the end of the year, GB200 NVL72 software would have improved considerably. Combined with frontier models architecture being codesigned with the larger scale up world size in mind, we expect that there will be significant efficiency gains from using the GB200 NVL72 by the end of the year.

On the reliability front, there will continue to be significant challenges that Nvidia must work even closer with its partners to rapidly solve, but we think the ecosystem will quickly rally its resources towards tackling these reliability challenges.

SemiAnalysis Is Hiring

We are looking for a new grad engineer to join our engineering team. This is an unique opportunity to work on an high-visibility special projects with support from many industry leaders and CEOs. If you’re passionate about performance engineering, system reliability, and want to work at the intersection of hardware and software, this is a rare chance to make industry wide impact.

What you’ll work on:

• Building and running large-scale benchmarks across multiple vendors (AMD, NVIDIA, TPU, Trainium, etc.
• Designing reproducible CI/CD pipelines to automate benchmarking workflows
• Ensuring reliability and scalability of systems used by industry partners

What we’re looking for:

• Strong skills in Python
• Background in Site Reliability Engineering (SRE) or systems-level problem solving
• Experience with CI/CD pipelines and modern DevOps practices
• Curiosity about GPUs, TPUs, Trainium, multicloud, and performance benchmarking

Link to apply: https://app.dover.com/apply/SemiAnalysis/2a9c8da5-6d59-4ac8-8302-3877345dbce1

Benchmarking and Analysis Methodology

For our benchmarking and analysis, we rely on Nvidia’s DGXC Benchmarking Team’s new DGX Cloud Benchmarking Scripts executed on NVIDIA’s internal H100 EOS cluster, configured with 8×400 Gbit/s InfiniBand networking. These results serve as the official reference numbers against which Neocloud environments can be compared when defining service-level agreements (SLAs) between Neoclouds and their customers.

Clouds can also submit benchmarks to NVIDIA and if they are able to meet these EOS reference numbers then they can earn the NVIDIA Exemplar Cloud designation. Our upcoming ClusterMAXv2 will heavily weight providers’ Exemplar Clouds status when evaluating service quality as this status is a stamp of approval that a provider can deliver reference performance numbers across many workloads for large scale GPU deployments.

The aforementioned benchmarks are conducted using NeMo Megatron-LM, but given that many end users of GPUs do not exclusively rely on NeMo Megatron-LM, the DGXC benchmarking team has plans to extend coverage to native Torch DTensor frameworks such as TorchTitan.

We would like to thank the Nvidia DGCX benchmarking team for creating these sets of benchmarks and providing reference numbers to help lift up the GPU Cloud industry!

H100 and GB200 NVL72 Capex, Opex, Total Cost of Ownership Analysis

The price of an H100 server has dropped somewhat in the past 18 months to around $190k per server. Including storage, networking and other items, the total upfront capital cost per server comes up to $250k for a typical hyperscaler.

Turning to the GB200 NVL72, the rack scale server alone costs $3.1M for a typical hyperscaler. Including networking, storage and other items, all in cost comes up to about $3.9M per rack.

When comparing across all three buyer types, from Hyperscalers to Neocloud Giants to Emerging Neoclouds, the GB200 NVL72’s all-in capital cost per GPU comes to about 1.6x to 1.7x the all-in capital cost per GPU of the H100.

Comparing the two systems’ operating cost of ownership, we find that the Opex per GPU for the GB200 NVL72 is not that much higher than that of the H100. The cost difference comes from the fact the GB200 NVL72 has a higher all-in power consumption per GPU than the H100. This is primarily driven by the fact that the GB200 chip consumes 1200W per chip vs 700W for the H100.

When factoring in both capex and opex in order to arrive at the total cost of ownership (TCO), we see that TCO for the GB200 NVL72 is about 1.6x higher than TCO for the H100. This means that the GB200 NVL72 needs to be at least 1.6x faster than the H100 in order to have an performance per TCO advantage when compared to the H100.

Three things Nvidia could do better for the ML community

Before we deep dive into the benchmarks and results, we will present three key suggestions to Nvidia.

First, we recommend that Nvidia expand their benchmarking efforts and increase transparency even more. In order for Nvidia to continuously raise the bar across the entire GPU cloud industry, it needs to benchmark across both its Hyperscaler partners and Nvidia Cloud Partners (NCPs) and make the data publicly available. With this, anyone in the ML community can factor the benchmarking data into their decision making process before signing contracts worth tens or hundreds of millions of dollars.

As an example, in the first release of our ClusterMAX rating system, we pointed out that GCP’s older a3-mega H100 delivered 10% worse than average MFU for O(Llama 70B) size training and 15-20% worse than average for MFU of O(8x7B) mixture of experts spare models. Thus, end users should be paying 10-20% lower than average rental cost to GCP in order to achieve the same performance per dollar as the market average. Having a publicly available set of benchmark results across the Hyperscaler and NCP providers will dramatically increase the ease of negotiating fair contract prices and speed up decision making. This can save considerable time and money on both sides by obviating the need for extensive, costly and time-consuming proof of concept runs.

Our second recommendation to Nvidia is that they expand their benchmarking focus beyond NeMo-MegatronLM as many users prefer to use Native PyTorch with FSDP2 and DTensor instead of NeMo-MegatronLM. One advantage of using NeMo-MegatronLM is that at any given time, there are many performance features in NeMo-MegatronLM that aren’t yet available in native PyTorch. It is reasonable for the latest features to be rolled out to NeMo-Megatron first, but all of these features should be upstreamed to native PyTorch after a month’s time at most. To this end, more Nvidia engineers should be allocated towards PyTorch core development instead of being tasked with adding more features to NeMo. Nvidia expanding benchmarking focus should include runs employing PyTorch will dovetail perfectly with this initiative as well.

Instead of having engineers optimize NeMo, they should be optimizing TorchTitan. The new NeMo AutoModel library is a step in the right direction it as supports native PyTorch FSDP2 backend in addition to Megatron-LM, noticeability missing is native PyTorch 3D+ Parallelism with DTensor and a lot of pretraining features is absent and most of the features is for finetuning.

Our third recommendation is that Nvidia continue to accelerate development of diagnostics and debugging tools for GB200 NVL72 backplane. Unfortunately, even after an extensive burn-in process, the NVLink copper backplane still is not that reliable. Operators of the GB200 NVL72 also lament that this problem is compounded by the fact that the tools used to diagnose and debug back-plane related errors are behind and sub-optimal. Nvidia can also improve the situation by insisting on even stricter acceptance tests across their ODM/OEM partners before handing GB200 NVL72 racks over to their customers.

GPT-3 175B Token/s/GPU, Training Performance and Power. Cost Improvements from January 2024 to December 2024

In the table below, we present the results of our benchmark runs in which we train GPT-3 175B on a 128 H100 cluster at different points in time. We chose to display results across different NeMo-Megatron LM Versions starting from January 2024 and ending in December 2024, representing one year and two years respectively from the start of H100 mass deployment.

The benchmark setup uses 128 H100s with 4 data replicas. Each data replica consists of 32 GPUs parallelized with each layer tensor parallelized using the NVLink domain across 4 GPUs (i.e. TP=4) and then pipelined. One might think that it would be best to do TP=8 to match the entire NVLink domain world size of 8 GPUs for the H100, but for GPT-3 175B model, it is better to use TP=4 as this will have a higher arithmetic intensity.

To elaborate, GPT3 175B’s hidden dimension is 12,288, which means if one were to use TP=8, the result will be a small K reduction dim of 1,536. By comparison, when using TP=4, the hidden reduction dim will instead be 3,072.

The sequence length of the benchmark follows the original GPT-3 paper setup and uses 2,048 seq length as well as a global batch size of 256 samples. This means the model will see 500k (Global Batch Size * Seq Len) tokens before each optimizer step.

When looking at BF16 model flops utilization (MFU), we see a considerable improvement from 34% MFU to 54% MFU over the course of 12 months, amounting to a 57% improvement in training throughput solely from software improvements across the CUDA stack. This improvement results from NVIDIA CuDNN/CuBLAS engineers writing more optimized fused wgmma kernels, NCCL engineers writing more optimized collectives that use fewer SMs for communication among other improvements. At the end of the day, it is the full software stack optimization that matters.

We see the same trend for FP8 MFU, improving from 29.5% MFU to 39.5% MFU in that same time, for a 34% improvement in throughput from just software gains alone.

Turning to costs, assuming a cost of $1.42/hr/GPU excluding any rental margin, we see that the cost to train GPT-3 175B on FP8 went from 72 cents per 1M tokens trained in Jan 2024 to just 54.2 cents per million tokens by Dec 2024. That means that the cost to train GPT-3 175B when using the original training token count of 300B improved from $218k in Jan 2024 to only $162k by Dec 2024.

Finally, we examine the power consumption from training GPT-3. We estimate the all in power draw for the 128 H100 cluster inclusive of GPUs, CPUs, networking, storage and other components. We then gross this up by the power usage effectiveness (PUE) of a typical colocation data center to arrive the all-in utility Joules per token.

As an unwelcome flashback to high school physics, a Joule is a unit of energy that is equivalent the work done when a force of 1 Newton moves an object 1 meter in the direction of the force. Lighting an incandescent 60W light bulb for one second consumes 60 Joules (a Watt (W) is a unit of energy consumption per second) and consumes 216kJ per hour. An alternative way to express units of energy is to use watt-hours or kilowatt-hours, which is just the power of a device multiplied by the number of hours which it is utilized over. The average annual US household in 2022 consumed 10,791kWh of energy or approximately 38,847,600,000 Joules. Dividing this 10,791 kWh by 8,760 hours per year gives us 1,232 W of power on average over the year – a little more than the 1,200W used by a single GB200 GPU!

We see that each token trained consumes 2.46 Joules for FP8 and 3.63 Joules for BF16 when using the December 2024 version of NVIDIA software. If we had an energy budget equivalent to the average US household’s annual energy consumption, we could train 15.8B FP8 tokens. Extending this calculation further, training 300B tokens on GPT3 175B would require 19 annual US households’ worth of energy consumption for FP8 and 28 households’ worth of annual energy consumption for BF16.

GPT-3’s total training cost of $162k and 19 households’ annual energy consumption doesn’t sound excessive, but it is the many experiments and many failed training runs that add up to the ballooning energy growth from AI Training we are seeing now in the United States.

Weak vs Strong Scaling

Strong and weak scaling describe the performance improvement of scaling compute resources for different problem setups, for instance, different batch sizes.

Strong scaling refers to scaling compute resources while keeping the model size and global batch size the same. In such a case, Amdahl’s Law, which describes the speedup that can be achieved by parallelizing computing steps, can be used to quantify the speedup of strong scaling.

On the other hand, weak scaling refers to scaling compute resources to solve larger problems at a constant time. AI Training inherently utilizes weak scaling since you can scale up your model size and your global batch size (depending on convergence) by scaling up the number of GPUs used a training job.

Llama3 405B Token/s/GPU, Cost Per Million Tokens, Joules Per Token vs Number of GPUs (Weak Scaling)

In this benchmark, we examine how training performance for Llama3 405B varies as we increase the number of H100 GPUs in the cluster – an example of weak scaling.

In the table below, we see how as we increase the GPU cluster size from 576 H100s to 2,304 H100s, both FP8 MFU and BF16 MFU hover around 43% MFU and 54% MFU respectively across all sizes. In the training run published in the Llama 3 Herd of Models Paper, researchers used 16k H100s to train Llama 3 405B, achieving a BF16 MFU of 41% for pretraining using a similar parallelism strategy. Note that the above pre-training runs used a sequence length of 8192, whereas for mid-training context extension, each sample’s sequence length is 131,072 instead of 8,192. This longer sequence length requires context parallelism across 16 nodes resulting in MFU dropping to 38% due to the additional communication needed for ring attention.

Turning to total cost of training, we see that carrying out just the pre-training run, training Llama 3 405B over 15T tokens, costs $1.95 per million tokens when training using BF16 using a 2,304 H100 cluster. This adds up to $29.1M just for the pretraining phase, which is dramatically higher than mixture of expert models such as DeepSeek – which cost only $5M per training run.

Of course, we stress once more that this cost reflects the cost for a single final successful training run and the cost of the many experiments needed to get to that final stage as well as the cost of employing researchers, among other costs.

Since Llama3 405B is approximately 2.3x larger than GPT3 175B in terms of total parameter count, the all-in utility Joules per token is about 2.3x greater for Llama 3 405B vs GPT3 175B at 8.8 Joules per token vs 3.6 Joules per token respectively.

This means that for the same energy as the average US household consumes in a year, Meta can train 4.4B tokens on Llama3 405B on BF16. To train to convergence using 15T tokens, Meta would require an amount of energy equal to the annual consumption of an entire neighborhood of 3,400 US households.

Llama3 70B Training Performance Token/s/GPU, Cost Per Million Tokens, Joules Per Token vs Number of GPUs (Weak Scaling)

Next, we look at Llama3 70B training performance for different cluster sizes. As we increase the cluster size from 64 H100s to 2,048 H100s, we see that performance for FP8 drops by 10%, dropping from 38.1% for 64 GPUs down to 35.5% for 2,048 GPUs. It is quite interesting that the MFU drops by so much (on a percentage basis – which is what really matters given the low MFU base) because the per data replica batch size doesn’t change as we scale up and the parallelism also strategy doesn’t change. All runs still continue to use TP=4,PP=2, and context parallel=2 – the only real change is adding more data replicas. Interestingly, for BF16, the drop in MFU is far smaller at only 1-2%, dropping from 54.5% MFU for 64 H100s down to 53.7% for 2,408 GPUs.

Llama3 405B is 5.7x larger than Llama3 70B, and as with any dense models, the number of FLOPs required is linear with respect to the number of parameters. As such, the cost to train Llama 3 405B should be 5.7x greater than that of Llama 3 70B. In practice, at the ~2k H100 scale, Llama3 405B is 5.4x more expensive in terms of the cost per million tokens using BF16.

In terms of power consumption, we see that for FP8, training consumes 10% more energy per token when training on 2,408 H100s vs 64 H100s. To train Llama 3 70B to convergence with 15T tokens on FP8 using at 64 H100s would only use energy equal to 440 US households’ annual energy consumption, whereas at the 2,048 H100 scale, we would require energy equivalent to 472 US households’ annual energy consumption.

Llama3 8B Training Performance Over Time

Larger models like Llama3 405B and Llama3 70B both use tensor parallelism, pipeline parallelism and data parallelism, but training Llama3 8B only requires context parallelism across the 8,192 sequence length for each pair of GPUs within the NVLink domain and uses data parallelism to spread the work beyond across other pairs of GPUs. In this analysis – we also look at training performance with respect to time in order to gauge how software improvements across the stack have affected training performance. We see that performance has only improved slightly from November 2024 to April 2025, the latter date being a full 23 months after Hopper began mass deployment.

In the next section, we deep dive into the current state of GB200 NVL72 training performance as compared to training on the H100. We will discuss benchmarks from training DeepSeek 670B MoE and Llama4 400B MoE, analyzing GB200’s performance per total cost of ownership (TCO) vs that of the H100.

We will also zoom into the aforementioned lack of effective GB200 NVL72 diagnostic and debugging tools and discuss the many issues contributing to the unreliability of GB200 NVL72. These are the challenges that NVIDIA, CSPs, Neoclouds and the end users at frontier labs must solve in order to successfully and cost effectively train frontier models on the GB200 NVL72 before the end of the year.