Gen AI is hitting limits but these new chipmakers can help

Corporate investors are backing chip startups with solutions for the generative AI's memory wall, interconnect and edge problems.

At times, the development of generative AI has seemed unstoppable. Products like ChatGPT have become a fixture of everyday life while the world’s largest tech companies scramble to build ever larger data centres for more powerful models. Corporate venture capital funds have dived in too. Over the last few years they have invested billions in startups innovating on the chip technology that allows AI to function.

But AI, like any developing technology, has its stumbling blocks. Chips with vast computing power are running below capacity because of memory restrictions, inefficiencies in design mean burning through unthinkable amounts of energy, and the technology is held remotely in large data centres rather than directly accessible on our devices.

These are all problems a wave of corporate-backed startups are hoping to solve.

CVC-backed investment rounds in startups developing AI chips increased by 43% in 2023 from the previous year, according to data from PitchBook. Behind the headline figure is a range of different market niches being targeted, from new methods of information transfer on the wafer itself, to designs that could find their way into your smartphone, up to the architecture for the large chips that go into the most power-hungry data centres.

(excerpt on Celestial)

The memory wall

A limitation in AI chip design comes from a problem known as the memory wall, and any innovative startups tackling it could gain a foothold in the market.

It was once thought that the key challenge in chip technology was increasing computing power.

“Today, we know that’s almost a useless metric,” says Christian Patze, an investment director at M Ventures, the CVC arm of the German materials company Merck, which supplies materials used in semiconductor manufacturing. The memory wall means that chips with a high amount of compute cannot make full use of it because the time spent transferring data from memory is time not spent doing computations.

“These models get so big that you run into memory issues where chips only reach 20% or 30% utilisation. They’re sitting there idle and that costs the data centre a lot of money. If you have faster interconnect, you could improve the utilisation of the GPUs.”

This is why M Ventures invested in Celestial AI, which is trying to speed up data transfer with optical interconnect technology, using photons for swifter communication on the chip between memory and compute.

“The hyperscalers do their own thing,” says Patze, referring to the large tech companies driving AI innovation such as Microsoft, Alphabet and Amazon, which are all trying to develop their own chips to reduce their dependency on Nvidia.

“This significantly tightens the market for selling datacenter AI accelerators but also opens an opportunity for companies with a solution to the interconnect problem to collaborate with the hyperscalers. That is what Celestial AI is doing.”

Tackling the memory wall is one area where corporates are hoping to see innovative startups flourish. In March, Celestial AI raised $175m in series C funding. Apart from M Ventures, other CVC funds involved in the deal include Samsung Catalyst Fund and AMD Ventures, the CVC arm of the US chip company AMD.

(full article at link)