Groq style light trails showing speed

Groq Review: The Fastest Open-Model API, After the NVIDIA Deal

Groq serves open models at hundreds of tokens per second on its LPU chips. Speeds, prices, the model catalogue, rate limits and what the NVIDIA licensing deal changed.

This review is built from Groq’s published pricing and documentation plus third-party benchmarks and coverage rather than a formal test of our own. Speeds and prices are attributed with dates; the model list changes often.

Groq runs open-weight language models on its own LPU chips and sells access through an OpenAI-compatible API called GroqCloud. Its pitch has always been speed: hundreds to a thousand tokens per second on models that run at a fraction of that on typical GPU clouds. In December 2025 the company signed a reported $20 billion licensing deal with NVIDIA, and most of its leadership and engineers moved there.

The honest verdict: Groq remains one of the fastest and cheapest ways to run popular open models through an API, and for real-time features such as voice agents, autocomplete and interactive tools, that speed changes the product. The catch is a small, shifting catalogue and a future that now runs through NVIDIA. Use Groq where latency matters and the models you need are on the menu, and keep a second provider ready.

Table of Contents

Groq speed concept lightning bolt
Speed is the whole pitch. The questions that matter now are which models stay on the menu and who sets the roadmap.

What Groq Is

AI Coolies described the service well: open-weight models such as Llama, GPT-OSS, Qwen, Kimi, DeepSeek and Gemma served on custom LPU chips with high token throughput and low-latency streaming. You call it with the OpenAI SDK by changing the base URL, and responses stream back faster than most people can read.

BenchLM made a point worth repeating because people mix them up: Groq is not a model. It is an inference host. Grok is xAI’s model; it runs other labs’ open models.

Toolradar summarised the appeal: very fast inference, transparent per-token pricing with no monthly fees, and a drop-in replacement for the OpenAI API with minimal integration work.

How the LPU Gets Its Speed

Generating text on a GPU is limited mostly by how fast model weights can be read from memory for each token. Groq’s LPU, short for language processing unit, was designed around that problem, keeping data on chip and running a deterministic schedule rather than the dynamic scheduling of a GPU.

OpenAI Tools Hub described the result as something more durable than demo speed: a deterministic compute architecture that turns inference latency into a contract instead of a guess. AI2Work framed it the same way, as a chip built for the decode phase of inference, which is exactly the part that GPUs handle least efficiently.

The trade-off is that models must be spread across many chips, which shapes which models the service can economically serve. That is part of why its catalogue is narrower than GPU-based hosts.

Speed in Practice

Model Reported speed Source
GPT-OSS 20B About 950 to 1,000 tok/s AI Coolies, BenchLM
GPT-OSS 120B About 500 tok/s BenchLM, from Groq’s production table
Llama 3.3 70B About 394 tok/s AI Coolies
Llama 4 Scout Over 460 tok/s, against 100 to 150 on H100s Voiceflow

CloudZero put the comparison in general terms: standard GPU providers deliver around 50 to 150 tokens per second, while its LPU delivers roughly 394 to 1,000 depending on model size.

Speed is not the whole story, though. Metavert cited independent benchmarks where Fireworks reached about 747 tokens per second against Groq’s 456 on some workloads, thanks to speculative decoding and batching. Throughput depends on model, prompt length and load, so test with your own requests.

Pricing in 2026

Pricing is per token, with no subscription. Spheron listed the non-enterprise catalogue at six models priced from $0.05 to $0.90 per million input tokens. AI Coolies gave examples: Llama 3.1 8B at $0.05 input and $0.08 output per million tokens, and Llama 3.3 70B at $0.59 and $0.79. We The Flywheel listed GPT-OSS 120B at $0.15 input and $0.75 output in January.

Metavert noted a 50% discount for batch processing and prompt caching. A free tier exists for testing, with lower rate limits.

Those prices are competitive with or below GPU hosts for the same models, which is remarkable given the speed advantage. AI Coolies called the pricing aggressive by 2026 standards, and that is fair.

The Model Catalogue

This is the main weakness. BenchLM listed the production text models on August 18, 2026 as GPT-OSS 20B and GPT-OSS 120B, plus Whisper Large v3 for speech and the Compound agent systems. Qwen 3.6 27B and MiniMax M2.7 were preview or enterprise. Llama 3.1 8B Instant was shut down for free and developer tiers on August 16.

That is a short list compared with the hundreds of models available through GPU hosts and gateways. Models also come and go, so an application built on one model can find it deprecated with limited notice. If you depend on a specific model, check its status regularly and plan a fallback.

What the NVIDIA Deal Changed

CloudZero reported that founder and chief executive Jonathan Ross, president Sunny Madra and roughly 90% of the engineering team moved to NVIDIA, in a deal structured to avoid a formal merger review that later drew Senate scrutiny and FTC attention. Voiceflow summarised the outcome: NVIDIA has the technology, Groq still has the cloud business.

We The Flywheel noted that the roadmap now runs through NVIDIA. NVIDIA showed a Groq 3 LPU and LPX rack at GTC in March 2026 for the second half of the year, with further generations planned for 2027 and 2028. AI2Work reported that NVIDIA paid 2.9 times Groq’s last valuation, a sign of how much it valued the decode-focused design.

For customers, the practical questions are continuity and investment. The service runs, prices remain low and the API is stable. Whether the cloud service will keep expanding its catalogue, or gradually give way to NVIDIA-hosted LPU offerings, is not yet clear. That uncertainty is a reason to avoid depending on the service alone.

Rate Limits and Reliability

BenchLM’s paid test on September 1, 2026 is instructive. On Qwen 3.8 27B, the service returned a median response time of 0.43 seconds and a p95 of 2.0 seconds across 450 requests. It was also the only platform in that run to return rate-limit errors mid-run, with an on-demand limit of 250,000 tokens per minute.

That matches the general pattern: very fast responses, but capacity limits that bite under load unless you have an enterprise agreement. Build retries and a fallback provider into any production use, and request higher limits before launching something popular.

Where Speed Changes the Product

  • Voice agents. Combined with fast speech recognition, sub-second responses make spoken conversations feel natural.
  • Autocomplete and inline suggestions. Suggestions that appear as you type need near-instant generation.
  • Agents with many steps. An agent that makes ten model calls per task finishes in seconds rather than minutes.
  • Interactive tools. Classification, extraction and rewriting inside a user interface feel instant.

For batch work where nobody waits, such as nightly document processing, speed matters less and price and model choice matter more.

Latency Budgets Explained

Tokens per second is the number everyone quotes, but users experience latency in three parts. The first is network time to reach the provider. The second is time to first token, which includes queueing and processing the prompt. The third is generation time, which depends on output length and speed.

For a short answer of 100 tokens, generation at 1,000 tokens per second takes a tenth of a second, so time to first token and network delay dominate. For a long answer of 2,000 tokens, generation takes two seconds at that speed and twenty seconds at 100 tokens per second, so raw speed dominates. Know which kind of answers your product produces before deciding how much speed matters.

Measure all three parts from where your users are. A provider that is fastest from a US data centre may be slower than a rival for users in Europe or Asia, simply because of distance. Percentile figures matter too: a median of half a second with a p95 of two seconds means one user in twenty waits four times longer than typical.

Building a Voice Agent

Voice agents are the clearest case for very fast inference. A spoken conversation feels natural when the reply starts within roughly a second of the user stopping. That budget has to cover speech recognition, the language model and speech synthesis.

A typical pipeline transcribes speech with a fast Whisper model, sends the text to a small or mid-sized language model, and streams the reply into a text-to-speech engine as tokens arrive. With fast inference, the language model step shrinks to a small slice of the budget, leaving room for better speech quality or a larger model.

Keep responses short and conversational, stream everything, and start speaking before the full answer is generated. Those design choices matter as much as raw speed. Test on real phone lines or browsers, because network jitter adds delay that benchmarks never show.

Batch Jobs and Caching

Not every workload needs speed. Summarising last night’s support tickets, classifying a backlog of documents or generating product descriptions can wait minutes or hours. For those, the batch discount matters more than tokens per second.

Metavert noted a 50% discount for batch processing and prompt caching. Caching helps whenever many requests share a long prefix, such as the same system prompt and instructions, which is common in agents and classification pipelines. Structure prompts so the shared part comes first and the variable part last, so the cache can reuse as much as possible.

For large batch jobs, compare total cost across providers rather than headline rates. A slower GPU host with a larger model catalogue and its own batch discounts may finish the job just as well for a similar price.

Planning for Model Changes

With a short catalogue that changes during the year, plan for models disappearing. Keep the model name in configuration rather than code, so switching takes minutes. Maintain a small evaluation set of real prompts and expected answers, so you can test a replacement quickly when a deprecation notice arrives.

Keep a second provider configured for the same or a similar open model. For GPT-OSS models in particular, several GPU hosts and gateways serve them, which makes failover straightforward. A gateway such as OpenRouter can automate that fallback for you, at the cost of its credit fee.

Finally, watch the provider’s changelog and email notices. Developer tiers sometimes lose access to a model before enterprise tiers, as happened with one small Llama model in August 2026, so the first sign of change may come sooner than you expect.

A quick way to decide is to time your current feature end to end. If users wait more than a couple of seconds for something that feels like it should be instant, faster inference will probably improve the experience noticeably. If they already get answers quickly, or never watch the output appear, spend your effort on model quality and cost instead. Speed is a feature only when someone is waiting for it, and most products have a mix of both kinds of request.

Groq vs Cerebras, Fireworks and Together AI

Provider Strength Weakness
Groq Very fast, low prices Small, shifting catalogue; rate limits
Cerebras Even faster on some models Also a small catalogue
Fireworks Fast with batching, broad catalogue Speed varies by model
Together AI Huge catalogue, fine-tuning, GPUs Slower than LPU or wafer-scale

AI Coolies noted that Cerebras benchmarks at 1,800 to 2,600 output tokens per second on Llama 3.1 8B. For pure speed on the models both serve, Cerebras is the main rival. For breadth and fine-tuning, Together AI and Fireworks win.

Against Renting GPUs

Spheron compared Groq’s per-token prices with renting GPUs and running models yourself. For steady, high-volume workloads on one model, a rented GPU running vLLM can undercut per-token prices. For bursty traffic, or when latency matters, the service is usually cheaper and much faster, with no infrastructure to run.

Our RunPod review and Modal review cover the self-hosted side.

Against Running Models Locally

A local GPU running GPT-OSS 20B will not come close to its speed, but it costs nothing per token and keeps data private. Many developers use both: local models for private or routine work, Groq for latency-sensitive features. Interfaces such as Open WebUI and LibreChat can connect to both.

Who Should Use What

If you are Start with Why Link
Building voice agents or real-time features Groq Hundreds of tokens per second, low prices Try it
Needing the very fastest on supported models Cerebras Thousands of tokens per second on some models Try it
Needing many models or fine-tuning Together AI Broad catalogue and training Try it
Wanting one key for everything OpenRouter Routes to Groq and others Try it

Frequently Asked Questions About Groq

How fast is Groq?
About 1,000 tokens per second on GPT-OSS 20B and 500 on GPT-OSS 120B, according to its production table.

How much does it cost?
Non-enterprise models range from $0.05 to $0.90 per million input tokens, with batch and caching discounts.

Is Groq owned by NVIDIA?
NVIDIA licensed Groq’s technology and hired most of its team in a reported $20 billion deal. GroqCloud continues as a service.

Which models are available?
A short, changing list led by GPT-OSS 20B and 120B, plus Whisper and preview models. Check the current list.

Does it have rate limits?
Yes. Independent testing hit limits mid-run on the on-demand tier. Enterprise agreements raise them.

Is Groq the same as Grok?
No. Grok is xAI’s model. Groq is an inference provider.

Getting Started With Groq

Create an account, generate an API key and point an OpenAI-compatible client at Groq’s base URL. Try GPT-OSS 20B first to see the speed for yourself.

Measure time to first token and total response time for your real prompts, not just tokens per second. Then build retries and a fallback provider before you depend on it.

Check the model deprecation page monthly if your product relies on a specific model.

Verdict

Use Groq if latency shapes your product: voice agents, autocomplete, interactive tools and multi-step agents all feel better at hundreds of tokens per second, and its prices are among the lowest for the models it serves.

The honest caveat is breadth and certainty. The catalogue is short and changes, on-demand rate limits bite under load, and the NVIDIA deal leaves the long-term direction of GroqCloud unclear.

Next step: test your latency-critical feature on Groq’s GPT-OSS models today, and set up a second provider, such as Cerebras or a GPU host, as a fallback before launch.

Sources and Further Reading

On this site: compare providers in our AI inference API providers review, see the wafer-scale rival in our Cerebras review, and route between hosts with OpenRouter.