Multi-Die Assemblies Dominate At 2nm And Below

Key Takeaways:

• Orders of magnitude performance improvements required for AI cannot happen without multi-die assemblies, new packaging options, and the underlying technologies and methodologies to enable them.
• Research initiated by IEEE’s International Roadmap for Devices and Systems is starting to gel across different market and technology segments.
• Massive compute power will be used to optimize these systems for performance, power, and cost.

Multi-die assemblies and heterogeneous integration are becoming standard in leading-edge servers and high-end edge devices, forcing broad changes in long-established design and manufacturing processes while fueling a slew of innovations and enabling technologies, many of which are still in development.

The primary driver behind this long-anticipated shift is the inability to pack enough transistors into a single reticle-sized die to meet the performance demands across data-heavy AI workloads. In the past, performance was improved by shrinking digital logic, which let more compute horsepower fit into a given die area. But demand for faster processing has far outpaced the ability to scale transistors and wires, creating a performance gap that feature shrinks can’t close.

The rapid adoption of agentic AI only complicates matters, because it’s no longer just about packing in more identical GPUs, which are the mainstay for generative AI. Agentic AI requires CPUs, in addition to GPUs and various accelerators, to orchestrate how agents behave in highly customized, shifting workloads. The result is so many variables and options to sort through that AI is needed even to validate chip architectures and ensure what gets designed is manufacturable, testable, and still relevant by the time it’s integrated into a larger heterogeneous system.

“As an industry, we are embarking on a journey of extraordinary opportunity,” said Prahalad Parthangal, technical director for advanced packaging at Lam Research, during a presentation at the recent SEMI Strategic Materials Conference. “AI is like a tidal wave. It’s sweeping across multiple industries. Every new model, every new application demands more from our chips. And with the expansion of large model training, real-time inference, and the beginning of a proliferation of the build-out of sensors on edge AI, the amount of silicon in each of these devices and their complexity is growing at an exponential pace.”

Many engineering teams aim for at least two orders of magnitude improvement in performance per watt. Achieving that goal will require a complete realignment of the chip industry. It involves more processing area, which today depends heavily on chiplets and advanced packaging, as well as continued logic scaling for power reasons. In addition, and especially as AI moves to the edge, it will likely include in-memory, near-memory, and in-sensor pre-processing to reduce the amount of data that needs to be processed, smaller language models that can deliver faster results using more targeted data, and the quantization and pruning of all language models.

“Logic compute scaling is increasingly shifting from monolithic devices to becoming more heterogeneously integrated chiplet architectures,” Parthangal said. “It looks very different in terms of the pace of innovation that is needed for every portion of the system or the package. With each generation, the increasing complexity requires us to innovate new materials, new architectures, and new packaging technologies, which are larger and larger, and becoming more and more advanced.”

Chiplet inspiration
The original idea behind chiplets, as the name implies, was to slice a single die into smaller dies to improve yield. That was the stated reason for Xilinx’s planar Virtex-7 chip in 2011, featuring four chiplets connected to an interposer. But priorities have changed since then.

Today, many chiplets are at, or close to, full reticle size, typically connected to other chiplets and memories through some type of interposer, substrate, and/or one or more bridges based on a 2.5D or 3.5D architecture. And to accommodate more chiplets and higher performance, the interposers are getting larger. In fact, when foundries and packaging houses talk about reticle limits these days, they typically are referring to the much larger interposer size, not the maximum lithography exposure, which is fixed at 858mm² for EUV.

“The reticle size is really increasing a lot,” said C.P. Hung, ASE’s vice president of corporate research and development, during a presentation at the SEMI Strategic Materials Conference. “Now we are on 5.5X, and 9X, 12X, and 40X are ahead. For all the advanced packages, we started with 2.5D, but people are saying, ‘Wow, that design is really expensive because all the devices are sitting on top of an interposer. That’s a huge interposer.'”

Fig. 1: Bigger interposers, higher costs. Source: ASE/SEMI Strategic Materials Conference.

This explains why 2.5D adoption is heavily weighted toward large data centers. A bespoke multi-die assembly that can be deployed in tens of thousands of servers achieves its own economies of scale, but it doesn’t work outside of that particular application. Costs need to be reduced through a steady stream of innovations that can be proven in silicon and, ideally,  integrated into existing manufacturing and advanced packaging processes.

In the past, much of this was achieved through the consensus of dozens of companies working in concert on the International Technology Roadmap for Semiconductors (ITRS). That effort ended in 2015 because it became clear that planar device scaling no longer provided sufficient power, performance, and area benefits. The following year, it was replaced by the IEEE International Roadmap for Devices and Systems (IRDS), which began looking at everything from packaging options to new materials, quantum computing, yield enhancement, and lithography.

Some of those efforts are now well underway and being integrated with others, a necessary step to achieve economies of scale beyond just one company. The key here is less about developing a single chip or technology and more about developing methodologies and processes that are extensible for the entire chip industry. That’s where the economies of scale kick in. For example, organic and glass interposers are inexpensive alternatives to silicon interposers, but the innovation lies in the materials and processes needed to handle them consistently and reliably.

Interposer alternatives gain traction
But proving and integrating these changes takes time. The chip industry is extremely risk-averse because the cost of failure in the fab — or worse, in the field — is enormous. Research into organic interposers began across the industry in 2010 as a low-cost alternative to silicon, but they were slow to catch on because of CTE mismatch, handling issues with flexible materials, and much lower density than silicon.

Likewise, Georgia Tech professor Rao Tummala and Venky Sundaram began researching glass interposers in 2011, but only in the past couple of years has glass been discussed as a replacement for silicon interposers. The main problem with glass is its brittleness, which requires special attention when drilling vias that connect multiple layers. But glass has fewer parasitics than silicon, and it’s extremely flat, which is important in advanced packages.

Skipping the interposer entirely and running wires through redistribution layers (RDL) in a high-density fan-out is another option, but until recently the tradeoff has been a significantly lower interconnect density. That could change by integrating bridges, which provide more connections than RDL alone.

“You can have three, six, or nine layers, and there are prototypes with up to 12 and 15,” said ASE’s Hung. “And there is a lot of discussion about replacing those high-density, multiple RDL layers with a bridge, so the rest of the less-dense fan-out is taken care of by only one or three layers of RDL.”

Fig. 2: Using a bridge to improve performance in a fan-out (upper right). Source: ASE/SEMI Strategic Materials Conference.

Thermal innovations
One of the thorniest issues in packaging is thermal dissipation, and it’s especially challenging as logic density in a package increases and logic utilization pushes into the 70% to 80% range in large data centers. As a point of reference, server utilization at the turn of the Millennium was in the 5% to 15% range, when cooling only required a fan.

Today, no single approach is sufficient to control the heat, so chipmakers must choose from a menu of options, including liquid cooling of server racks and microchannels inside chips, direct cooling, cold plates, immersion, vapor caps, and TSV chimneys. And if that isn’t sufficient, some type of throttling, like checkerboarding, can be used, where every other transistor or block of transistors is turned on and off.

A new approach combines several of these options into two-phase cooling, where a non-conductive liquid with a low boiling point is channeled across a cold plate in a closed loop. The liquid boils off into a vapor and then condenses back into a liquid.

“Two-phase cooling is coming, and that significantly reduces the interface resistance of getting the heat out,” said David McCann, senior vice president and chief of staff for business units at Amkor Technology. “You have liquid that boils off within the cooling surface, so it stays at a constant temperature. And you get a lot of turbulence, which helps dissipate the heat, which gets absorbed into the cooling medium. That’s the next big transition. Getting the heat out is a huge challenge because the power of these modules is skyrocketing. We see that when we power up the different parts to do the test. That’s an interaction between us, the customer designing the [test] program, and the manufacturer. They’re turning on different parts of the chip at different times so we don’t overwhelm the ability to test from a heat standpoint. The test program has to be designed so it captures all of that.”

Better floor-planning can help here, as well. “Just by multi-physics modeling, as you are floor-planning and you get some deviation in the design, we have gotten very good at detecting hot spots ahead of time and telling the silicon, ‘You might want to rearrange this. Pay attention here.’ Even the modeling techniques are extremely improved,” said Lalitha Immaneni, vice president of semiconductor R&D in Intel’s Assembly Test Technology Development organization.

Additionally, the amount of research underway to further simplify and de-risk different processes is enormous.

“The thermal team is looking at various ways of removing heat with integrated heat spreaders and new material that goes directly on top of the IHS,” Immaneni said. “Mechanical warpage and thermal dissipation will be the biggest issues. In the industry, people are talking about liquid cooling and thermal interface materials. But at the same time, some amount of smart floor-planning will help.”

New stacking approach
Not all stacking is vertical. IBM Research has developed a hybrid approach that stacks transistors at an angle using beveled edges and fewer metal routing tracks. The result is increased transistor density — 50% improved performance, 70% more energy efficiency, and 40% more SRAM. The company expects to begin production with this approach in the early 2030s.

Fig. 3: A beveled design using a “nanostack” approach. Source: IBM Research

“We believe that with the introduction of different substrates for the nFET and the pFET, for the top FET and the bottom FET — along with a layout innovation in standard cell libraries — for the first time we can do a three-track (3T) library design,” said Huiming Bu, vice president of global semiconductor R&D at IBM Research. “No one has done that before. Five-track today is the industry best, then it moves to four-track, and then three-track. That’s one key innovation.”

5T designs are used primarily with GAA FETs down to 2nm. They balance routability and area density and are commonly used in 3nm and 2nm designs. 3T offers much higher density due to lower cell heights and a smaller footprint. The challenge with 3T has been leakage between power rails because the materials are so tiny, but IBM claims to have solved that problem.

“The second innovation is a material innovation in the top FET and the bottom FET,” Bu said. “That opens the door for many innovations for a performance boost. Then there’s a third innovation. If you think about a standard stack design, you access both sides of the transistor from top to bottom. If the transistors are aligned like CFETs, you cannot do that. You can only access the top of the top transistor and the bottom of the bottom transistor. Everything else has to go laterally, which takes up area in your layout. That wastes design efficiency. We can put signal routing and power routing on both sides of the wafer. We have at least three generations of nanostack without stacking more layers.”

Bu said the number of stacks can be increased to at least four layers. That will require more sophisticated thermal channeling, handled by a combination of liquid microchannel cooling and thermal vias, and possibly a new substrate material with better thermal conductivity.

“Previously, the most advanced designs were done with design technology co-optimization,” Bu said. “That is not going to be sufficient. We need to add mechanical and thermal into that equation. Mechanical cannot be addressed after the fact. It has to be part of the technology. Mechanical stability and thermal conductivity are the key innovations that have to happen. We are working with our EDA partners on how to design certain features.”

Fig. 4: Counting atoms in IBM’s sub-1nm technology. Source: IBM Research

Chiplets
Even though chiplets have been used in production chips since 2011, they were slow to catch on — and not for lack of interest. In 2017, DARPA funded the Common Heterogeneous Integration and Intellectual Property Reuse Strategies (CHIPS) program to shrink the development cycle using standardized components.

But integrating chiplets into an advanced package is far more complicated than snapping LEGOs together. So far, nearly all of the designs that incorporate chiplets are custom. Beyond standard I/Os and high-bandwidth memory, a successful commercial chiplet marketplace has yet to emerge. That hasn’t diminished the need for chiplets, but it has limited their use to large systems companies, which deploy highly customized versions in AI data centers.

“We’ve looked at chiplets, and at first it seems like the perfect dream architecture,” said Robbie Williamson, vice president of customer engineering for Cloud AI at Arm. “There are standards for how they talk and consortia. But Nvidia pushes one thing, we push another, Google pushes another. So where do we meet? When you talk to customers and really start digging into it, it becomes more expensive than you thought. And then, how reusable is this thing? These are issues we’re still working through, and the industry is still working through.”

The underlying problem is that when chiplets are encapsulated in an advanced package, each chiplet needs to be fully characterized in the context of what physically surrounds it to account for a variety of physical effects — noise, heat, vibration, and voltage changes — as well as the workloads they will be running. Any or all of those factors can affect chiplet utilization, size, and the overall performance per watt of the systems they’re used in.

While customized chiplets will always be the optimal approach, they are too expensive for most applications. Big systems companies have been able to justify those costs with higher performance, which reduces the area per watt, and faster time to results, which is a competitive advantage. But even they appear to be facing cost pressures, as shown by recent moves to sell chiplets designed for internal use. This includes Google’s TPUs, Meta’s inference accelerators, Amazon’s CPUs and AI training and inference chips, and Microsoft’s training and inference accelerator. It’s still unclear how these in-house designs will fare against designs from other companies.

Large automotive companies, such as BMW, Stellantis, Volkswagen, Rivian, and Tesla, have also been developing chiplet-type architectures that can use chiplets for different options across all of their car models. Alongside this, there are ongoing discussions about whether more generic hardware can be used, running either a virtualization layer or a common programming interface.

The challenge here is that a chiplet is only one piece in a complex, multi-die assembly, and chiplets need context. Performance can vary greatly, depending on how and where it will be used, and for what purpose. In addition, the systems they are used in need some type of prioritization scheme for sending and receiving data, and accessing it in the first place.

“There’s a lot of work when we’re building chiplets where we need to have discussions with the partner that is building them,” said Kristof Beets, vice president of product management at Imagination Technologies. “How much data are you looking to move from all these different regions? How do we make sure you have enough bandwidth, that the latency is okay, and that you create something that will actually work? If it’s all local data, fantastic. That’s like a normal SoC. But if we need to do three or four hops to get to main memory, and it takes hundreds of cycles for each hop, it’s much more problematic.”

Big changes with massive simulations
Chiplets are just one of many interacting components and dependencies in a multi-die assembly. Until recently, the only way to manage all of them was a highly iterative, rigidly siloed, and time-consuming development process. What’s changed is the massive compute power that has since become available, allowing chipmakers to use that same compute power to brute-force complex design and manufacturing challenges, then integrate all the steps end-to-end.

“We have a much-improved understanding of the physical mechanisms we’re modeling,” said David Fried, chief AI officer and corporate vice president for Semiverse Solutions at Lam Research. “We keep discovering new physical mechanisms — and actually leveraging new mechanisms for process advantage — but we’ve gotten much better at understanding those physical mechanisms and the compute capabilities. The compute horsepower we’re applying to these problems has finally caught up to a level where simulation can make predictions on an hour-by-hour, day-to-day basis. Simulation used to be an academic pursuit. Somebody used to take nine months to run simulations to prove a mechanism or effect. Today, we’re doing in an hour what was completely unimaginable 20 years ago, even in academic circles. That acceleration of simulation and virtualization has had a dramatic effect on R&D. Even 10 to 15 years ago, it would have been intractable. It would have required too much memory, too much compute power, and algorithms that weren’t well understood. So nobody even tried.”

Virtual fabs accelerate process development
Orchestration of those various pieces with manufacturing processes is being automated, as well. This is driving a relatively new concept called fab technology co-optimization (FTCO), which essentially is to manufacturing what system and design technology co-optimization are to system and chip design.

“We’ve got this long history of multi-physics solvers that have gone into TCAD (technology computer-aided design), and so now two things have happened,” said Wally Rhines, CEO of Silvaco. “One is that processes get more and more complex, so if you want to get the optimum for any process step, you’ve got to test more and more variables. The second thing is that cycle times have increased, so if you want to run prototype wafers, you’re talking about months in the wafer fab. A lot more of the development and analysis has to be virtual. We’re seeing enormous demand for building surrogate or machine learning models, digital twins, whatever you want to call them, for particular processes, then collecting a lot of data and building a model. You’re doing hundreds of queries to figure out whether the temperature in tenth-of-degree steps is optimum at one point or another, and you might take 12 other variables and query them.”

The challenge, and one that has persisted for decades, is how to share data between different players in the ecosystem without giving away proprietary information. Progress is being made, finally, but it’s still not fully solved. “The trend is for everything to go to machine learning models, and to generate those models you need an enormous amount of synthetic data. But you also need to calibrate it with actual physical data,” Rhines said. “And the problem with the physical data is nobody in the semiconductor industry allows anyone else to see their physical data.”

Continued scaling
All of these developments don’t change the focus on device scaling. Traditional scaling continues, but that’s not the only path forward. In fact, multi-die assemblies typically have a variety of processing elements developed at different nodes, with the most advanced nodes running the highest-priority computations.

Transistor density still increases processing speed, though not with the percentage gains seen when scaling was the only path forward, and it shortens the distance signals need to travel to SRAM, which is critical for L1/L2 cache. The problem is that SRAM doesn’t scale, and no other memory type works as well, so what gets stored in SRAM versus off-chip DRAM needs to be prioritized. That reduces the performance benefits of shrinking features, but scaling still reduces power at each node, which is particularly important for compute-intensive AI.

At 2nm and below, problems compound. Process variation, faster circuit aging, greater susceptibility to noise and voltage swings, and much harder-to-control gate leakage all make it harder to manage. And that’s just for starters.

“The challenges are basically more severe versions of the 3nm and 4nm nodes, which are amplified at 2nm and below,” said Kostas Adam, vice president of engineering at Synopsys. “One of those challenges is the amount of computation required to deal with each mask — to compute what needs to go on the mask. That’s a major new complication on the manufacturing front. From a fab perspective, they need to deliver very high yield, which is more difficult on advanced nodes, and they need to do it as cheaply as possible. If they can avoid a lot of multi-patterning, they will reduce cost and complexity and make manufacturing easier. But if you cannot avoid it, the more you go into an advanced node, the more you need different tricks.”

Precision becomes essential at every step of the flow. “Another thing at play is the level of hitting the dimensions that you are trying to manufacture accurately,” Adam said. “It becomes even more challenging, because you are still working with the same hardware, more or less, the same DUV and EUV tools that were used for 7, 5, and 4 nanometers. A lot of the weight is shifting to doing more heavy computation and optimization to compute the optimal mask, so that if you stick it into the scanner, it will give you the best process window. And if you get a good process window, that immediately translates to yield. In my group, day in and day out, we deal with how to deliver these more advanced computations and optimizations that are needed for the next node, and to try to contain the computational cost, which frankly is pretty insane. To compute a full reticle-size chip, it takes roughly 1 million CPU hours for one mask. For a relatively hard layer, like a middle-metal layer mask, you need 1 million CPU hours. If you have 10 or 20 of those hard layers, you are talking about 20 million CPU hours per product. So if you are a foundry and you have to process 10 products per month, you’re talking about 200 million CPU hours per month, which translates to 6.5 million CPU hours per day. That means you need roughly 300,000 cores in your data centers, which translates to about 120 racks in a data center at 15 kilowatts per rack. So we are dealing with daunting calculations, and the expectation is that you need more accuracy, but without increasing the computational cost of the previous generation.”

Conclusion
The good news is that all of these problems have — or will have — one or more solutions. A slew of options are available, and many more are on the way. In the short term, the availability of some of these may be determined by fab and packaging capacity, which will limit the number of companies with access to advanced-node technology, but that will even out over the next five years.

Longer term, the big question isn’t whether the leading-edge technology will be available everywhere. It’s how and where the industry will use a long list of options, which will vary greatly by region, market segment, and workload. The semiconductor requirements for a humanoid robot will be very different than those of a data center or a smartphone, and the focus on power will vary greatly from an AI data center in China to one in the United States.

There are more puzzle pieces available, more ways to put them together, and more possibilities for how to use them, and those options will only increase over the next couple of decades.

The post Multi-Die Assemblies Dominate At 2nm And Below appeared first on Semiconductor Engineering.