Techjockey News

SambaNova Systems, a company specializing in AI hardware and software systems, has revealed its latest AI chip, the SN40, which is set to improve its SambaNova Suite, a comprehensive large language model (LLM) platform.

Introduced just months ago in March, the SambaNova Suite distinguishes itself by deploying processors and operating systems for AI inference training. It aims to provide a cost-effective alternative to power-hungry GPUs.

Surprisingly, SambaNova has swiftly upgraded its hardware, a significant leap in performance. The SN40L, the latest offering, increases the capacity to handle a whopping 5 trillion parameters in an LLM, supporting sequences of over 256,000 in length on a single system node, as reported by the company.

This remarkable performance boost is attributed to the SN40L’s 51 billion transistors per processing unit, with the total of 102 billion per package, a substantial increase compared to the previous SN30 model.

The SN40L incorporates 64 GB of HBM memory, a new addition to SambaNova’s product line, which offers over three times the memory bandwidth for faster data processing. It also features 768 GB of DDR5 memory per processing unit, totalling 1.5 TB, compared to the SN30’s 512 GB (1.0 TB).

SambaNova’s processor sets itself apart from Nvidia’s GPU by providing a reconfigurable dataflow unit (RDU)-based environment.

On the software front, SambaNova introduces a turnkey solution for generative AI. Their full AI stack comprises pre-trained open-source models like the Meta Llama2 LLM model, which organizations can customize with their own content to create internal LLMs. The package also includes SambaFlow software, designed to automatically analyze and optimize processing based on specific task requirements.

Dan Olds, Chief Research Officer at Intersect360 Research, regards this upgrade as a substantial advancement in both hardware and software. Notably, the SN40’s 5 trillion parameter capability is nearly three times larger than the estimated 1.7 trillion parameters of GPT-4.

“The larger memory, plus the addition of HBM, are key factors in driving the performance of this new processor. With larger memory spaces, customers can get more of their models into main memory, which means much faster processing. Adding HBM to the architecture allows the system to move data between main memory and the cache-like HBM in much larger chunks, which also speeds processing,” said Olds.