AMD buys Taalas to reshape AI inference chips

AMD has agreed to acquire Toronto-based chip startup Taalas, adding technology that embeds artificial intelligence models directly into silicon as the semiconductor group intensifies its push into the fast-growing AI inference market.

The deal, announced on August 6, will bring Taalas’s specialised inference technology and engineering team into AMD’s Artificial Intelligence Group. Financial terms were not disclosed. Completion remains subject to customary regulatory approvals and closing conditions.

Founded in 2023, Taalas has developed processors that effectively turn an AI model into dedicated hardware. Instead of repeatedly moving model weights between high-bandwidth memory and computational units, its architecture encodes the weights and associated dataflows into the chip itself.

That approach targets one of the central constraints in modern AI computing: moving vast quantities of data between memory and processors. Large language model inference can become limited by memory bandwidth even when substantial computational capacity is available, forcing accelerator makers to invest heavily in high-bandwidth memory, advanced packaging and interconnect technology.

Taalas attempts to remove much of that bottleneck by combining model storage and computation within specialised silicon. Its architecture uses a hard-wired mask-ROM fabric alongside SRAM that supports functions including key-value caching, model adaptations and fine-tuning.

The company’s first technology demonstrator, HC1, runs Meta’s Llama 3.1 8B model and is fabricated using TSMC’s 6-nanometre process. The 815-square-millimetre chip contains about 53 billion transistors. Taalas says its implementation can generate about 17,000 tokens per second for an individual user.

Those performance figures were produced by Taalas and should therefore be viewed as vendor benchmarks rather than independent measurements. The company nevertheless demonstrates how specialised hardware could deliver substantially lower latency than general-purpose accelerators when serving a fixed model.

HC1 also illustrates the technology’s biggest trade-off. Conventional GPUs can run many different models after software changes. A Taalas processor is substantially tied to the model for which it was designed. Moving to another architecture or set of model weights can require new silicon.

Taalas says it has reduced the cost and time involved by designing its platform so that only a small number of metal layers need to change when adapting the processor to another model. Its development process is intended to convert model weights into deployable hardware in roughly two months.

The company has also maintained some flexibility around its hard-wired approach. Its Llama implementation supports configurable context lengths and low-rank adapters for fine-tuning. A second-generation platform is designed around standard 4-bit floating-point formats and higher model density.

AMD intends to integrate the technology into its accelerator roadmap rather than position it as an outright replacement for GPUs. Taalas processors could operate alongside Instinct accelerators, EPYC server processors, networking hardware and AMD’s ROCm software platform, creating systems that assign different AI workloads to different types of silicon.

The strategy reflects the growing distinction between AI training and inference. Training requires highly programmable hardware because models and techniques are continually changing. Large-scale inference can favour efficiency once a model has stabilised, particularly when millions of queries are being served using the same weights.

Inference has consequently become an increasingly important battleground between AMD, Nvidia and specialist accelerator developers. Higher token generation speeds, lower energy consumption and reduced infrastructure costs can translate directly into improved economics for cloud providers and companies deploying generative AI applications.

AMD has been expanding its presence across that market through its Instinct accelerator family and Helios rack-scale architecture. Its MI455X accelerator is designed with 432GB of HBM4 memory and theoretical memory bandwidth reaching 23.3 terabytes per second, illustrating how conventional high-performance GPU designs are addressing the same underlying memory challenge through increasingly powerful memory systems.

Taalas offers AMD a markedly different architectural option. Instead of continuously increasing the bandwidth between compute engines and external memory, specialised silicon can minimise the need to move model parameters at all.

The acquisition also reunites AMD with engineers who have extensive links to the company. Taalas co-founder and chief executive Ljubisa Bajic previously worked on AMD CPU-GPU designs before founding AI chipmaker Tenstorrent. Co-founders Lejla Bajic and Drago Ignjatovic also held engineering roles at AMD before moving into AI accelerator development.