AI chip startup Groq recently made significant progress in AI with its Language Processing Unit (LPU)。Groq's LPU is designed for large language models (LLMs) and uses the Tensor-Streaming Processor (TSP) architecture to demonstrate excellent performance metrics, reaching 750 TOPS (INT8) and 188 TeraFLOPS (FP16), with 320 × 320 fused point-by-point matrix multiplication and 5,120 ALU vectors.。The company's LPU surpassed GPU-based cloud service providers in inference LLM as a direct competitor in the AI inference hardware market such as NVIDIA, AMD, and Intel.。

Groq's LPU also has impressive results in market performance。The company announced that its LPU system has twice the inference performance of the Llama-2 70B model, running at more than 240 tokens per second per user.。Groq also demonstrated the low-latency performance of its LLM and deployed its hardware to the ALCF AI test bed at Argonne National Laboratory, giving researchers around the world access to AI accelerators.。In addition, Groq's LPU inference engine led the way in the first independent LLM benchmark, surpassing the key performance indicators of the eight largest cloud service providers.。Groq provides Tokens-as-a-Service through the Groq API to provide customers with experimental and production applications.。Groq's breakthrough in AI chips shows its competitive strength and potential in the AI hardware market。