Frontier model training has pushed GPUs and AI systems to their absolute limits, making cost, efficiency, power, performance per TCO, and reliability central to the discussion on effective training. The Hopper vs Blackwell comparisons are not as simple as Nvidia would have you believe.

In this report, we will start by present the results of benchmark runs across over 2,000 H100 GPUs, analyzing data on model flops utilization (MFU), total cost of ownership (TCO) and cost per training 1M tokens. We will also discuss energy use, examining the energy in utility Joules consumed for each token trained and compare it to the average US household annual energy usage, reframing power efficiency in societal context. We will also show the results of this analysis when scaling the GPU cluster from 128 H100s to 2048 H100s and across different versions of Nvidia software.

Later in this report, we will also analyze GB200 NVL72 benchmark results across Llama4 400B MoE and DeepSeek 670B MoE and compare this data to our earlier results from the H100. We will discuss whether the GB200 NVL72 performance per $ advantages survives once reliability issues are factored in.

Downtime from poor reliability and lost engineering time is one of the main factors that we will capture in our perf per TCO calculations. Currently there are no large-scale training runs done yet on GB200 NVL72 as software continues to mature and reliability challenges are worked through. This means that Nvidia’s H100 and H200 as well as Google TPUs remain the only GPUs that are today being successfully used to complete frontier-scale training. As it stands today, even the most advanced operators at frontier labs and CSPs are not yet able to carry out mega training runs on the GB200 NVL72.

With that said, every new architecture naturally requires time for the ecosystem to ramp software to effectively utilize the architecture. The GB200 NVL72 ramp is slightly slower than prior generations, but not by much, and we are confident that before the end of the year, GB200 NVL72 software would have improved considerably. Combined with frontier models architecture being codesigned with the larger scale up world size in mind, we expect that there will be significant efficiency gains from using the GB200 NVL72 by the end of the year.

On the reliability front, there will continue to be significant challenges that Nvidia must work even closer with its partners to rapidly solve, but we think the ecosystem will quickly rally its resources towards tackling these reliability challenges.

SemiAnalysis Is Hiring

We are looking for a new grad engineer to join our engineering team. This is an unique opportunity to work on an high-visibility special projects with support from many industry leaders and CEOs. If you’re passionate about performance engineering, system reliability, and want to work at the intersection of hardware and software, this is a rare chance to make industry wide impact.

What you’ll work on:

• Building and running large-scale benchmarks across multiple vendors (AMD, NVIDIA, TPU, Trainium, etc.
• Designing reproducible CI/CD pipelines to automate benchmarking workflows
• Ensuring reliability and scalability of systems used by industry partners

What we’re looking for:

• Strong skills in Python
• Background in Site Reliability Engineering (SRE) or systems-level problem solving
• Experience with CI/CD pipelines and modern DevOps practices
• Curiosity about GPUs, TPUs, Trainium, multicloud, and performance benchmarking

Link to apply: https://app.dover.com/apply/SemiAnalysis/2a9c8da5-6d59-4ac8-8302-3877345dbce1

Benchmarking and Analysis Methodology

For our benchmarking and analysis, we rely on Nvidia’s DGXC Benchmarking Team’s new DGX Cloud Benchmarking Scripts executed on NVIDIA’s internal H100 EOS cluster, configured with 8×400 Gbit/s InfiniBand networking. These results serve as the official reference numbers against which Neocloud environments can be compared when defining service-level agreements (SLAs) between Neoclouds and their customers.

Clouds can also submit benchmarks to NVIDIA and if they are able to meet these EOS reference numbers then they can earn the NVIDIA Exemplar Cloud designation. Our upcoming ClusterMAXv2 will heavily weight providers’ Exemplar Clouds status when evaluating service quality as this status is a stamp of approval that a provider can deliver reference performance numbers across many workloads for large scale GPU deployments.

The aforementioned benchmarks are conducted using NeMo Megatron-LM, but given that many end users of GPUs do not exclusively rely on NeMo Megatron-LM, the DGXC benchmarking team has plans to extend coverage to native Torch DTensor frameworks such as TorchTitan.

We would like to thank the Nvidia DGCX benchmarking team for creating these sets of benchmarks and providing reference numbers to help lift up the GPU Cloud industry!

H100 and GB200 NVL72 Capex, Opex, Total Cost of Ownership Analysis

The price of an H100 server has dropped somewhat in the past 18 months to around $190k per server. Including storage, networking and other items, the total upfront capital cost per server comes up to $250k for a typical hyperscaler.

Turning to the GB200 NVL72, the rack scale server alone costs $3.1M for a typical hyperscaler. Including networking, storage and other items, all in cost comes up to about $3.9M per rack.

When comparing across all three buyer types, from Hyperscalers to Neocloud Giants to Emerging Neoclouds, the GB200 NVL72’s all-in capital cost per GPU comes to about 1.6x to 1.7x the all-in capital cost per GPU of the H100.

Comparing the two systems’ operating cost of ownership, we find that the Opex per GPU for the GB200 NVL72 is not that much higher than that of the H100. The cost difference comes from the fact the GB200 NVL72 has a higher all-in power consumption per GPU than the H100. This is primarily driven by the fact that the GB200 chip consumes 1200W per chip vs 700W for the H100.

When factoring in both capex and opex in order to arrive at the total cost of ownership (TCO), we see that TCO for the GB200 NVL72 is about 1.6x higher than TCO for the H100. This means that the GB200 NVL72 needs to be at least 1.6x faster than the H100 in order to have an performance per TCO advantage when compared to the H100.

Three things Nvidia could do better for the ML community

Before we deep dive into the benchmarks and results, we will present three key suggestions to Nvidia.

First, we recommend that Nvidia expand their benchmarking efforts and increase transparency even more. In order for Nvidia to continuously raise the bar across the entire GPU cloud industry, it needs to benchmark across both its Hyperscaler partners and Nvidia Cloud Partners (NCPs) and make the data publicly available. With this, anyone in the ML community can factor the benchmarking data into their decision making process before signing contracts worth tens or hundreds of millions of dollars.

As an example, in the first release of our ClusterMAX rating system, we pointed out that GCP’s older a3-mega H100 delivered 10% worse than average MFU for O(Llama 70B) size training and 15-20% worse than average for MFU of O(8x7B) mixture of experts spare models. Thus, end users should be paying 10-20% lower than average rental cost to GCP in order to achieve the same performance per dollar as the market average. Having a publicly available set of benchmark results across the Hyperscaler and NCP providers will dramatically increase the ease of negotiating fair contract prices and speed up decision making. This can save considerable time and money on both sides by obviating the need for extensive, costly and time-consuming proof of concept runs.

Our second recommendation to Nvidia is that they expand their benchmarking focus beyond NeMo-MegatronLM as many users prefer to use Native PyTorch with FSDP2 and DTensor instead of NeMo-MegatronLM. One advantage of using NeMo-MegatronLM is that at any given time, there are many performance features in NeMo-MegatronLM that aren’t yet available in native PyTorch. It is reasonable for the latest features to be rolled out to NeMo-Megatron first, but all of these features should be upstreamed to native PyTorch after a month’s time at most. To this end, more Nvidia engineers should be allocated towards PyTorch core development instead of being tasked with adding more features to NeMo. Nvidia expanding benchmarking focus should include runs employing PyTorch will dovetail perfectly with this initiative as well.

Instead of having engineers optimize NeMo, they should be optimizing TorchTitan. The new NeMo AutoModel library is a step in the right direction it as supports native PyTorch FSDP2 backend in addition to Megatron-LM, noticeability missing is native PyTorch 3D+ Parallelism with DTensor and a lot of pretraining features is absent and most of the features is for finetuning.

Our third recommendation is that Nvidia continue to accelerate development of diagnostics and debugging tools for GB200 NVL72 backplane. Unfortunately, even after an extensive burn-in process, the NVLink copper backplane still is not that reliable. Operators of the GB200 NVL72 also lament that this problem is compounded by the fact that the tools used to diagnose and debug back-plane related errors are behind and sub-optimal. Nvidia can also improve the situation by insisting on even stricter acceptance tests across their ODM/OEM partners before handing GB200 NVL72 racks over to their customers.

GPT-3 175B Token/s/GPU, Training Performance and Power. Cost Improvements from January 2024 to December 2024

In the table below, we present the results of our benchmark runs in which we train GPT-3 175B on a 128 H100 cluster at different points in time. We chose to display results across different NeMo-Megatron LM Versions starting from January 2024 and ending in December 2024, representing one year and two years respectively from the start of H100 mass deployment.

The benchmark setup uses 128 H100s with 4 data replicas. Each data replica consists of 32 GPUs parallelized with each layer tensor parallelized using the NVLink domain across 4 GPUs (i.e. TP=4) and then pipelined. One might think that it would be best to do TP=8 to match the entire NVLink domain world size of 8 GPUs for the H100, but for GPT-3 175B model, it is better to use TP=4 as this will have a higher arithmetic intensity.

To elaborate, GPT3 175B’s hidden dimension is 12,288, which means if one were to use TP=8, the result will be a small K reduction dim of 1,536. By comparison, when using TP=4, the hidden reduction dim will instead be 3,072.

The sequence length of the benchmark follows the original GPT-3 paper setup and uses 2,048 seq length as well as a global batch size of 256 samples. This means the model will see 500k (Global Batch Size * Seq Len) tokens before each optimizer step.

When looking at BF16 model flops utilization (MFU), we see a considerable improvement from 34% MFU to 54% MFU over the course of 12 months, amounting to a 57% improvement in training throughput solely from software improvements across the CUDA stack. This improvement results from NVIDIA CuDNN/CuBLAS engineers writing more optimized fused wgmma kernels, NCCL engineers writing more optimized collectives that use fewer SMs for communication among other improvements. At the end of the day, it is the full software stack optimization that matters.

We see the same trend for FP8 MFU, improving from 29.5% MFU to 39.5% MFU in that same time, for a 34% improvement in throughput from just software gains alone.

Turning to costs, assuming a cost of $1.42/hr/GPU excluding any rental margin, we see that the cost to train GPT-3 175B on FP8 went from 72 cents per 1M tokens trained in Jan 2024 to just 54.2 cents per million tokens by Dec 2024. That means that the cost to train GPT-3 175B when using the original training token count of 300B improved from $218k in Jan 2024 to only $162k by Dec 2024.

Finally, we examine the power consumption from training GPT-3. We estimate the all in power draw for the 128 H100 cluster inclusive of GPUs, CPUs, networking, storage and other components. We then gross this up by the power usage effectiveness (PUE) of a typical colocation data center to arrive the all-in utility Joules per token.

As an unwelcome flashback to high school physics, a Joule is a unit of energy that is equivalent the work done when a force of 1 Newton moves an object 1 meter in the direction of the force. Lighting an incandescent 60W light bulb for one second consumes 60 Joules (a Watt (W) is a unit of energy consumption per second) and consumes 216kJ per hour. An alternative way to express units of energy is to use watt-hours or kilowatt-hours, which is just the power of a device multiplied by the number of hours which it is utilized over. The average annual US household in 2022 consumed 10,791kWh of energy or approximately 38,847,600,000 Joules. Dividing this 10,791 kWh by 8,760 hours per year gives us 1,232 W of power on average over the year – a little more than the 1,200W used by a single GB200 GPU!

We see that each token trained consumes 2.46 Joules for FP8 and 3.63 Joules for BF16 when using the December 2024 version of NVIDIA software. If we had an energy budget equivalent to the average US household’s annual energy consumption, we could train 15.8B FP8 tokens. Extending this calculation further, training 300B tokens on GPT3 175B would require 19 annual US households’ worth of energy consumption for FP8 and 28 households’ worth of annual energy consumption for BF16.

GPT-3’s total training cost of $162k and 19 households’ annual energy consumption doesn’t sound excessive, but it is the many experiments and many failed training runs that add up to the ballooning energy growth from AI Training we are seeing now in the United States.

Weak vs Strong Scaling

Strong and weak scaling describe the performance improvement of scaling compute resources for different problem setups, for instance, different batch sizes.

Strong scaling refers to scaling compute resources while keeping the model size and global batch size the same. In such a case, Amdahl’s Law, which describes the speedup that can be achieved by parallelizing computing steps, can be used to quantify the speedup of strong scaling.

On the other hand, weak scaling refers to scaling compute resources to solve larger problems at a constant time. AI Training inherently utilizes weak scaling since you can scale up your model size and your global batch size (depending on convergence) by scaling up the number of GPUs used a training job.

Llama3 405B Token/s/GPU, Cost Per Million Tokens, Joules Per Token vs Number of GPUs (Weak Scaling)

In this benchmark, we examine how training performance for Llama3 405B varies as we increase the number of H100 GPUs in the cluster – an example of weak scaling.

In the table below, we see how as we increase the GPU cluster size from 576 H100s to 2,304 H100s, both FP8 MFU and BF16 MFU hover around 43% MFU and 54% MFU respectively across all sizes. In the training run published in the Llama 3 Herd of Models Paper, researchers used 16k H100s to train Llama 3 405B, achieving a BF16 MFU of 41% for pretraining using a similar parallelism strategy. Note that the above pre-training runs used a sequence length of 8192, whereas for mid-training context extension, each sample’s sequence length is 131,072 instead of 8,192. This longer sequence length requires context parallelism across 16 nodes resulting in MFU dropping to 38% due to the additional communication needed for ring attention.

Turning to total cost of training, we see that carrying out just the pre-training run, training Llama 3 405B over 15T tokens, costs $1.95 per million tokens when training using BF16 using a 2,304 H100 cluster. This adds up to $29.1M just for the pretraining phase, which is dramatically higher than mixture of expert models such as DeepSeek – which cost only $5M per training run.

Of course, we stress once more that this cost reflects the cost for a single final successful training run and the cost of the many experiments needed to get to that final stage as well as the cost of employing researchers, among other costs.

Since Llama3 405B is approximately 2.3x larger than GPT3 175B in terms of total parameter count, the all-in utility Joules per token is about 2.3x greater for Llama 3 405B vs GPT3 175B at 8.8 Joules per token vs 3.6 Joules per token respectively.

This means that for the same energy as the average US household consumes in a year, Meta can train 4.4B tokens on Llama3 405B on BF16. To train to convergence using 15T tokens, Meta would require an amount of energy equal to the annual consumption of an entire neighborhood of 3,400 US households.

Llama3 70B Training Performance Token/s/GPU, Cost Per Million Tokens, Joules Per Token vs Number of GPUs (Weak Scaling)

Next, we look at Llama3 70B training performance for different cluster sizes. As we increase the cluster size from 64 H100s to 2,048 H100s, we see that performance for FP8 drops by 10%, dropping from 38.1% for 64 GPUs down to 35.5% for 2,048 GPUs. It is quite interesting that the MFU drops by so much (on a percentage basis – which is what really matters given the low MFU base) because the per data replica batch size doesn’t change as we scale up and the parallelism also strategy doesn’t change. All runs still continue to use TP=4,PP=2, and context parallel=2 – the only real change is adding more data replicas. Interestingly, for BF16, the drop in MFU is far smaller at only 1-2%, dropping from 54.5% MFU for 64 H100s down to 53.7% for 2,408 GPUs.

Llama3 405B is 5.7x larger than Llama3 70B, and as with any dense models, the number of FLOPs required is linear with respect to the number of parameters. As such, the cost to train Llama 3 405B should be 5.7x greater than that of Llama 3 70B. In practice, at the ~2k H100 scale, Llama3 405B is 5.4x more expensive in terms of the cost per million tokens using BF16.

In terms of power consumption, we see that for FP8, training consumes 10% more energy per token when training on 2,408 H100s vs 64 H100s. To train Llama 3 70B to convergence with 15T tokens on FP8 using at 64 H100s would only use energy equal to 440 US households’ annual energy consumption, whereas at the 2,048 H100 scale, we would require energy equivalent to 472 US households’ annual energy consumption.

Llama3 8B Training Performance Over Time

Larger models like Llama3 405B and Llama3 70B both use tensor parallelism, pipeline parallelism and data parallelism, but training Llama3 8B only requires context parallelism across the 8,192 sequence length for each pair of GPUs within the NVLink domain and uses data parallelism to spread the work beyond across other pairs of GPUs. In this analysis – we also look at training performance with respect to time in order to gauge how software improvements across the stack have affected training performance. We see that performance has only improved slightly from November 2024 to April 2025, the latter date being a full 23 months after Hopper began mass deployment.

In the next section, we deep dive into the current state of GB200 NVL72 training performance as compared to training on the H100. We will discuss benchmarks from training DeepSeek 670B MoE and Llama4 400B MoE, analyzing GB200’s performance per total cost of ownership (TCO) vs that of the H100.

We will also zoom into the aforementioned lack of effective GB200 NVL72 diagnostic and debugging tools and discuss the many issues contributing to the unreliability of GB200 NVL72. These are the challenges that NVIDIA, CSPs, Neoclouds and the end users at frontier labs must solve in order to successfully and cost effectively train frontier models on the GB200 NVL72 before the end of the year.

Long time readers will recall that SemiAnalysis covers more than just datacenters and AMD. Today we’re back to semiconductors with a tech-focused roundup of the best from this year’s VLSI conference, the premiere design and integration. That includes the latest in chips manufacturing: fab digital twins, the future of advanced logic transistors and interconnects, DRAM architectures beyond the 1x nm nodes, and more. We’ll discuss Intel’s 18A process and compare with TSMC, where backside power will be adopted (and where it won’t), and the likely winners in 4F² versus 3D DRAM.

Digital Twins: From Atoms to Fabs

Semiconductor design and fabrication is getting exponentially more complex, increasing development costs and lengthening design cycles. Digital twins allow for design exploration and optimization to be done in an accelerated virtual environment. With this, engineers can ensure that designs work before any silicon is run through the fab.

Digital twins span the entire scale of semiconductor design:

• Atomic-level: Simulate the quantum and Newtonian interactions between atoms in materials engineering of transistor contacts and gates
• Wafer-level: Optimize tool chambers and process recipes in virtual silicon for yield and performance
• Fab-level: Maximize fab productivity with orchestrated maintenance and management across the fleet

On atomistic simulations, Synopsys provided an overview of their QuantumATK suite, used in materials engineering in transistor contacts and gate oxide stack design, which are critical to device performance. Traditional Density Functional Theory (DFT) modelling of quantum effects between atoms is the most accurate but computationally expensive, while conventional force field simulation of Newtonian atomic interactions is quick but with limited accuracy. GPU accelerated DFT-NEGF (Non-Equilibrium Green’s Function) demonstrated a 9.3x speedup using only 4x A100 vs CPU, while Machine-Learned Force Field simulation using Moment Tensor Potentials demonstrated near-DFT accuracy with 17 min compute cost vs 12 days with traditional DFT.

These atomic models are critical in understanding the electrical interactions occurring at the interface between different material layers. In contact engineering, MLFF is used to generate the contact interface between crystalline silicon and amorphous silicides, simulating the depth of interdiffusion where the boundary undergoes silicidation. DFT-NEGF is then used to calculate contact resistance and current-voltage curves across the interface. For gate oxide design, the complex multi-layer work function metal stack is built using MLFF and simulated to check its structure and chemical composition. Dipole dopants can then be introduced and optimized with DFT, which also does electrostatic analysis to calculate key parameters such as effective work function, Schottky barrier height and equivalent oxide thickness. As we move forward into Gate All Around design schemes, these atomic simulations become even more important in choosing the right materials.

On wafer-level optimization with virtual silicon, Lam Research presented work on their Coventor SEMulator3D software. With transistor geometries increasing in complexity from planar to FinFET to GAA, the number of possible process recipe combinations only grows exponentially, which they’ve marketed as Lam’s Law. Virtual wafer fabrication is done with process simulation using trained models with optimized parameters, allowing engineers to widen process windows and improve yield while reducing the number of physical test wafer cycles required to validate changes. Lam also builds their deposition and etch tools as a digital twin, building virtual chambers with plasma flow simulation to help in recipe prediction while also optimizing chamber design for uniformity across the wafer.

These simulation tools have been used in process window studies to select backside contact schemes with the widest process window while investigating how each recipe affects the stress and strain across the nanosheet transistors. High Aspect Ratio etch schemes also use virtual environments to predict tool output etch profiles for a given input mask pattern. These etch profiles are compared to a target output profile and given a distance which is then minimized with further tests in the digital twin.

Going up to the fab-level, Lam also presented the work needed to achieve a ‘lights-out’ fab. That is a fab that requires no human intervention and as such one can turn the lights off. Tool fleets need to be orchestrated in a virtual twin at near real-time speeds to coordinate tool downtime and maximize fab productivity. The tools themselves need to be ‘self-aware’ with their predictive maintenance, using built-in metrology tools that detect tool alignment and process drift over its lifetime. For a lights-out fab, each tool should target at least 1 year of uninterrupted operation with no human intervention, with automatic recovery after failure and self-requesting maintenance.

Maintenance of tools would be carried out automatically with robotic parts delivery and installation of consumables and wear items, with tools being designed around robotic maintenance. While a conceptual target of 2035-2040 was given by Lam, the main barrier facing lights-out fabs is with data and connectivity across tools from different vendors and the standardization of maintenance processes.

TSMC DRAM in BEOL

With SRAM bit density no longer increasing with new process nodes, TSMC R&D sought to revive eDRAM to boost chip cache densites. Embedded DRAM was last seen in IBM’s z15 on GlobalFoundries 14nm. The main innovation here is that TSMC can fabricate the entire memory array within the BEOL metal layers, with the DRAM transistor and capacitor formation scheme being within the 400 degree C limit of BEOL flows. This frees up the front-side transistors and lower metal layers for functional logic blocks. With modern processor designs increasing the ratio of SRAM to logic area, being able to stack a DRAM-based last level cache on top of active logic would represent a breakthrough in scalability and design.

However, the demonstration shown is still early in the R&D phase, with the available advanced logic area below just used to house the DRAM peripheral logic (Wordline drivers and sense amplifiers) to boost memory density. The 4Mbit macro that was fabricated only had a bit density of 63.7 Mb/mm2, not even 2x that of modern high density 6T SRAM. For reference, Micron’s latest 1-gamma DRAM would be about 9x denser than this, but without the performance and accessibility to serve as on-die cache.

While TSMC did not give any hints on when this will be ready for productization, it does show huge potential for future generations of this technology, which will fundamentally change how chips are designed.

DRAM: 4F² and 3D

DRAM has two inflection points on its 5-year roadmap: 4F² and 3D. The current 6F², in use for more than a decade, will only scale until the 1d node. With 1c in high volume now, 1d should debut in the next 1-2 years. SK Hynix highlighted a few key challenges in scaling beyond 1d:

Cell contact areas, in particular the storage node contact where the storage capacitor connects to the control transistor below, shrink quadratically with cell critical dimension. The contacts must be large / well aligned enough to provide a good electrical connection between the transistor and capacitor, but not be so large or misaligned that they short out to any neighboring cells. This is the “cell contact open margin” in the chart above, shrinking with every node. At 1d the process and tooling reach their limits for a workable, high-yield process.

The resistance in the shrinking devices and interconnects also grows as they scale down. This is the “cell external resistance” referenced above. It’s a catch-all sum of all resistive elements between the memory cell and the sense amplifier. The bitline contact and local bitline (metal) wire itself are two main contributors. Both increase in resistance as they scale. This slows down cell operation and reduces the read margin of the cell, both undesirable. Operation speed is affected by charge transfer between the cell and bitline, which slows as the resistance of that path increases. Resistance also saps the voltage differential that the sense amplifier sees. Too little and the cell cannot reliably be read – the memory doesn’t work:

The Memory Wall: Past, Present, and Future of DRAM

4F² solves these issues and more. We won’t rehash the basics of the architecture – see our Memory Wall report linked above – but a few specifics are interesting:

The cell contact challenge in 6F² comes from congestion where the bitline and storage node contact are on the same level (storage node contacts are denoted as BC for buried contact in the image below).

From a side-on view, its easy to see how little margin there is between the bitline and contact:

Compare this to vertical channel transistors (VCT) in a 4F² layout; the buried bitline has its own real estate well out of the way of any other components. The current path is also much shorter, directly down from capacitor, through the vertical channel, directly to the bitline. In 6F² the path is down through the bottom of the “U” shaped channel and back up, longer and consequently with higher resistance.

Of course, there are challenges for implementing 4F² or it would’ve been adopted already. Both the buried bitline and vertical channel transistor are high aspect ratio, difficult for etch and deposition tools. Until just a few years ago deposition tools were not capable of filling a deep trench with the required metals for the bitline, likely Ru or Co. The cell layout, although it reduces some of the alignment challenges, is still denser and thus requires EUV patterning. Last, there was simply no reason to take the risk of changing architectures when 6F² was still scalable.

There are still a few wildcards in 4F² development that may determine which fab can achieve lowest cost per bit and good yields, and which tool vendors might benefit. The gate structure, crucial to the performance of the memory cell, might be dual or even gate all around. SK Hynix and others are still deciding.

There is also a choice between peripheral-under-cell and peripheral-on-cell. Conventionally the peripheral circuitry would’ve been adjacent to the memory cells on the wafer frontside, but for increased overall density it will be moved underneath the cell array. Peri-under-cell is similar to backside power for logic, requiring fusion bonding of a second wafer. The control transistors are built in an array on the frontside before a support wafer is bonded on, the wafer flipped, and the peri built up. Then everything is flipped once more to add the storage node contacts and capacitors themselves. The tool vendors who would see incremental benefit are similar to the BSPDN supply chain – CMP, fusion bonding, TSV etch.

Peri-on-cell is simply hybrid bonding of completed storage node array and peripheral wafers. While this offers some process latitude – the peri can be made without worrying about damage to the array and vice versa – it requires hybrid bonding at a pitch of well below 50nm. That’s an order of magnitude lower than the current leading edge. Still, Hynix at least is looking at it in R&D, and other applications will drive hybrid bonder development regardless.

Last, 3D DRAM is being developed in parallel. Current progress suggests a few nodes of 4F² are likely before 3D is ready. Chinese chipmakers are a potential disruptor here, as they have strong incentive to develop 3D because it is not dependent on advanced litho.

NVDRAM

Micron’s NVDRAM (NV for non-volatile) resurfaced 18 months after its first appearance at IEDM 2023. This is their ferroelectric (HZO) DRAM using a 4F² architecture, Ruthenium wordlines, and CMOS under array. If you were trying to make an expensive memory using all the latest tricks, this is probably how you’d do it.

The bitcells were scaled by an impressive 27% since the previous paper, to 41nm on a side, without a degradation in performance. That brings density to nearly 0.6 Gb/mm², well above any high volume commercial DRAM available today.

Theoretically NVDRAM has a small advantage over traditional DRAM since it does not waste power and time performing refresh cycles. Unfortunately, the electricity savings amount to roughly $1 per year. When a single DIMM is on the order of $300+, the lifetime energy savings are nowhere near enough to justify the higher price of this exotic product. At the very least, the work on Ru wordlines, 4F², vertical channel transistors, and CMOS under array are all applicable to upcoming DRAM nodes.

2D Materials

Replacing silicon is a high bar. Any replacement must not only yield better performing, denser transistors, it must also be practical to use. Silicon wafers are a commodity and can easily be doped in select areas to form transistor channels. 2D materials are not yet practical to work with on an industrial scale. We’ve written many different times that on-wafer growth is they key blocker. But if chipmakers or labs are solving this problem, they’re keeping it quiet. The papers we saw on other innovations – Intel improving contact formation, Samsung building CFETs with 2D channels – are impressive, but ultimately fall flat if the material can’t be grown in an economic way in the first place.

Forksheet

Gate-all-around is no longer the “next big thing” in logic, it’s ramping towards high volume. Forksheet and CFET have taken the mantle of exciting next-gen architectures. An evolution of GAA, forksheet entails putting the N- and P- halves of a CMOS closer together by adding a dielectric wall between them.

In a traditional architecture, the spacing between NMOS and PMOS devices is limited by parasitic capacitance and the threat of latch-up. Increasing parasitic capacitance means the chip runs slower and uses more power. Latch-up is an outright failure of the transistors giving the input voltage V_dd an uncontrolled path directly to ground. Some techniques already exist – most importantly shallow trench isolation – for mitigating these effects.

Forksheet is a new, theoretically better technique along these same lines. While the material between NMOS and PMOS has always been some sort of insulating dielectric, a forksheet requires an exquisite layer of ultra low-k material to enable tighter spacing. This introduces new integration challenges and extra cost into the manufacturing process.

It’s not trivial to develop a material that can be deposited in nm-thick, high-quality films but also withstand subsequent processing as the rest of the transistor is formed. Plasma-induced damage from etch or deposition is a particular problem. Most papers aren’t detailing their material solutions here but it’s a good bet that AMAT, traditionally the leader in ultra-low-k dielectrics, is playing a part.

Forksheet also nominally has worse gate control than gate-all-around. This is because the gate only wraps around three sides of the transistor channel, the fourth abuts the forksheet wall. It’s basically a finFET turned sideways. Increased density but worse electrostatic control vs. GAA isn’t necessarily a good tradeoff. There are a few workarounds: 1) slightly etch back the forksheet wall, leaving room for gate material to envelop the fourth side of the channel, sacrificing some scaling benefit 2) add additional nanosheets to improve electrostatics control, adding cost and integration complexity.

TSMC, IBM, and IMEC all talked extensively about forksheets. For IBM and IMEC this has limited commercial relevance. For TSMC, a willingness to discuss in detail might even be a negative signal for real adoption. No publicly announced node – through the 14 angstrom families at present – is using forksheet.

CFET Timeline

Even so, the potential successor to forksheet is already being discussed. CFET has been in vogue for a few years and we’ve covered the basics before:

Current work is towards industrialization. Lab demonstrations of a single device work well and look great on slides, but the cost is high and yield low. Even though they are popular at conferences, we think real adoption of CFETs in high volume is still a decade out. A presenter from Intel, in a talk about “beyond RibbonFET,” outright said “We will probably see gate all around for another decade.” As with copper interconnects and finFETs, the core technologies of logic tend to extend 1-2 nodes beyond what is expected.

China’s FlipFET Design

Despite various sanctions, China is not slowing down in semiconductor research and development. Out of all the academic papers presented, Peking University’s FlipFET design caught the most attention, showcasing a novel patterning scheme to achieve similar PPA to CFETs without the headaches of monolithic or sequential integration.

Essentially, the FlipFET concept starts with forming the fins or nanosheets for both top and bottom transistors, but only does the high temperature Source/Drain epitaxy for the top transistor before flipping the wafer and exposing the backside for processing. The contacts and BEOL metal layers are patterned before the wafer is flipped again to complete the lower temperature processes on both sides. This method produces a self-aligned transistor stack that does not need high-aspect ratio processing that monolithic CFETs have to overcome. Forming the gates from both sides also allows easier threshold voltage tuning differences between the top and bottom device.

However, the main drawback to FlipFET is cost, trading off easier integration of the active transistors at the expense of multiple backside process flows with a greater susceptibility to wafer warpage and overlay errors, lowering yield. So far, the lab has only fabricated frontside and backside transistors on separate wafers, leaving doubts on whether the fabrication of the other transistor would affect device performance of the first. Alignment of fine-pitch contacts and metals after wafer flipping is also a concern, but should not be any more challenging than other CFET options.

While the Chinese labs have already demonstrated FlipFETs in silicon, they’re not stopping there. Further innovation of FlipFET designs were presented and modelled such as FlipFET with self-aligned gate, FlipFETs using forksheets with an embedded power rail within the isolation wall, and even applying the FlipFET concept to monolithic CFETs with high aspect ratio Vias to achieve a 4-stack transistor design.

18A Process Details

The star paper was Intel’s 18A presentation. This is the first detailed public look at a real high volume backside power process.

Intel claims 30% SRAM scaling for 18A against an Intel 3 baseline. A large one-time benefit like this is expected when changing from finFET to GAA. The cell diagrams clearly show the shrink achieved by replacing 2 fins with a single stack of wide ribbons:

Comparing high density (HD) cell areas, 18A is on par with TSMC N5 and N3E at 0.0210 µm². N2 should also see at least some benefit from the finFET -> GAA transition, but most of the claimed 22% SRAM scaling (vs. N3E) likely comes from the periphery not the bitcells themselves. Overall 18A density is likely to be slightly less than N3P, and close to 30% less than N2.

18A Cost vs. Leading Nodes