本文作者 Zhongzhu Zhou 是 TogetherAI 的 Senior Research Scientist，悉尼大学博士，研究方向为高效机器学习系统，方向覆盖 模型训推算法与系统协同设计，LLM 压缩与量化。团队成员均来自 ...

尽管量化已成为大模型性能优化的常规技术手段，但由于很难评估模型量化的实际效果，依然有人质疑量化模型的准确度与生成质量。 对此，基于Llama 3.1系列模型，AI模型优化与加速推理服务商Neural Magic进行了超五十万次的实测，以对比模型量化与原始模型的效果 ...

Discover how a 12-year-old Raspberry Pi successfully runs a local LLM using Falcon H1 Tiny and 4-bit quantization.

Reducing the precision of model weights can make deep neural networks run faster in less GPU memory, while preserving model accuracy. If ever there were a salient example of a counter-intuitive ...

Why workflow optimization matters more than massive hardware specs.

The reason why large language models are called ‘large’ is not because of how smart they are, but as a factor of their sheer size in bytes. At billions of parameters at four bytes each, they pose a ...

One-bit large language models (LLMs) have emerged as a promising approach to making generative AI more accessible and affordable. By representing model weights with a very limited number of bits, ...

U of T Engineering researchers examine ways to make the use of language models more resource efficient by replacing their ...

Google researchers have published a new quantization technique called TurboQuant that compresses the key-value (KV) cache in large language models to 3.5 bits per channel, cutting memory consumption ...