Groq, inference on custom LPU silicon

Groq, inference on custom LPU silicon

Groq is an inference provider that uses its own chip, called an LPU or Language Processing Unit, instead of graphics cards. The effect is single and measurable: open-weight models answer faster. The documentation quotes roughly a thousand tokens per second for openai/gpt-oss-20b and roughly five hundred for openai/gpt-oss-120b.

What Groq is and what it is not

Before you install anything, two naming confusions need clearing up, because they cost people real time.

The first is Grok. Grok is xAI's family of models. Groq is a different company, a different product, and a different line of business: xAI trains its own models, Groq serves other people's models on its own hardware. The names differ by one letter, both come up in the same conversations about language models, so the mix-up is common and better caught early, before it lands in project documentation or in a contract.

The second confusion lives in the npm registry. The groq package at version 6.10.1, MIT licensed, describes itself as "Tagged template literal for Sanity.io GROQ-queries" and has nothing to do with Groq. It is Sanity's query language. The official client library is called groq-sdk. In Python it is the other way round: there pip install groq gives you the Groq client, because no collision exists. Running npm install groq instead of groq-sdk ends in an import that will never work, and the error message will not explain why.

Groq is not a model lab. It does not train what it serves. openai/gpt-oss-120b and openai/gpt-oss-20b come from OpenAI, qwen/qwen3.6-27b from Alibaba, whisper-large-v3 also from OpenAI, the Orpheus models from Canopy Labs, and minimaxai/minimax-m2.7 from MiniMax. Groq adds the hardware, the execution layer, and the API.

One thing is genuinely its own: the groq/compound and groq/compound-mini systems. These are not models but compositions of openly available models with built-in tools, web search and code execution, invoked automatically depending on the query. Both run at around 450 tokens per second, hold a 131,072 token context window and an 8,192 token completion cap, and carry no per-million rate in the model table.

Speed as a design parameter

Generation speed is easy to treat as a pleasant property, something like a shorter progress bar. In practice it moves the boundary of what can be built at all, and it does so in two ways.

The first concerns agents. An agentic task is a loop: the model generates, calls a tool, receives a result, generates again. Total time is the sum of the steps, so it multiplies with their count. A ten-step agent where each step generates eight hundred tokens spends roughly eighty seconds on generation alone at a hundred tokens per second. At five hundred it drops to sixteen. Same loop, but the first version belongs in a background job and the second in an interaction where a human waits at the screen. The arithmetic is illustrative, since real numbers depend on your prompts, but the ratio does not depend on anything.

The second concerns live responses. Transcription in Whisper billed per hour of audio, ticket classification inside the HTTP request path rather than in a queue, rewriting search results before they reach the browser. In each case the decision reads "can we afford a model call inside the request or not", and the answer moves with latency.

On mechanism, Groq's documentation is sparing. It describes a quantisation scheme called TruePoint Numerics, which lowers precision only where doing so does not affect accuracy, and links to a blog post about how the LPU is built. The rest of the architecture sits on the vendor's side and cannot be independently checked from outside.

Speed does not fix three things, though, and better to know that up front. It does not fix model quality, because a fast weaker model will not replace a slower stronger one on a task that demands reasoning. It does not fix end-to-end latency, because the quoted tokens per second describe generation, while the default on_demand tier per the documentation admits occasional queue latency at peak times. It also does not cover long prompts: the latency guarantee on the Enterprise plan requires uncached context to stay below 8,192 tokens, so large-input workloads fall outside it even after signing a contract.

Model catalogue and rates

Here sits the main limitation, and it deserves to be seen whole, because the catalogue is far narrower than the label "inference provider" suggests.

There are exactly two production text models and both are gpt-oss. Add two Whisper variants, billed per hour of audio, plus the two Compound systems. Everything else sits in preview, which the documentation describes as intended for evaluation only and liable to be discontinued at short notice, or is marked Enterprise with a "Contact Sales" price, as minimaxai/minimax-m2.7 is with its 196,608 token window.

The deprecation history says more than the catalogue itself. Within six months the platform closed four rounds of models: meta-llama/llama-4-maverick-17b-128e-instruct on 9 March 2026, moonshotai/kimi-k2-instruct-0905 on 15 April, qwen/qwen3-32b together with meta-llama/llama-4-scout-17b-16e-instruct on 17 July, and llama-3.1-8b-instant plus llama-3.3-70b-versatile on 16 August. In every round the recommended replacement was gpt-oss. The deprecations cover the free and developer plans, while Enterprise customers on a committed-spend contract are exempt, which is itself information about what stability looks like without a contract.

The consequence for your code is concrete. Keep the model identifier in configuration rather than in code, and check the deprecation page before every sizeable release. Notices go by email to the organisation address, so if the account was opened by somebody who has since changed teams, the message will not arrive.

One detail shows how fast this catalogue moves. As this is written, the documentation pages for the Batch API, the Flex tier and the Performance tier still list llama-3.3-70b-versatile and llama-3.1-8b-instant, although both were shut down on 16 August 2026. Examples from the documentation can therefore return an error when copied without changing the identifier.

Versions, licences and what is actually open

The client libraries are open, and checking the licence against three sources reveals no discrepancy here, which after a run of projects where it does is a pleasant change.

The groq-sdk package in the npm registry stands at version 1.5.0 with the license field set to Apache-2.0. The unpacked tarball contains a LICENSE file holding the full Apache License 2.0 text and the notice "Copyright 2026 Groq". The groq/groq-typescript repository carries the same file, and the GitHub programming interface reports Apache-2.0 for it. Python matches exactly: the groq package at version 1.6.0 on PyPI, Apache 2.0 in the metadata, the same file in the source archive and in the groq/groq-python repository. Three sources, one answer.

The scale of the projects is modest and fits what they are, namely generated client libraries. The Python repository holds around 613 stars, 61 forks and three open issues. The TypeScript one holds around 264 stars, 35 forks and five issues. Neither is archived, both saw changes on 20 August 2026. The TypeScript client carries no production dependencies. The Python one requires language version 3.10 or newer plus anyio, distro, httpx, pydantic, sniffio and typing-extensions.

This is where the open part ends and the misunderstanding that recurs most often in conversations about Groq begins. An Apache licence on the client says nothing about the service. The LPU itself, the execution layer and the whole platform are closed, and the only route to that speed is an account with Groq. There is no self-hosted variant.

The licences on the weights are a separate matter and belong to the model authors rather than to Groq. openai/gpt-oss-120b and openai/gpt-oss-20b are marked Apache 2.0 on Hugging Face, as is qwen/qwen3.6-27b. Meta's models, present on the platform until August 2026, carry their own community licence, which is not an open source licence in the Open Source Initiative sense and imposes conditions on use. Open weights and open source are two different things, and the licence is checked on the model card, not at the inference provider. More on that distinction sits in the pieces on Meta Llama models and on Hugging Face.

Vendor lock-in risk here is asymmetric, and that is fairly good news. The API shape follows OpenAI, so leaving means changing a base URL and a key. The harder part is that model identifiers are Groq specific, and your application's behaviour depends on a speed you may not get elsewhere. You move the code without friction and the timing characteristics much less so.

The first call and OpenAI compatibility

Installation and a first call take two minutes. The key is generated in the console and goes into an environment variable.

pip install groq
export GROQ_API_KEY=gsk_...

# in a Node project, mind the package name
npm install groq-sdk

Groq's own client looks like every client of this shape. The service_tier parameter matters more here than usual and I return to it in the section on limits.

import os
from groq import Groq

client = Groq(api_key=os.environ["GROQ_API_KEY"])

response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    service_tier="auto",
    max_completion_tokens=4096,
    messages=[
        {"role": "system", "content": "Answer concisely."},
        {"role": "user", "content": "Classify this ticket and assign a priority."},
    ],
)

print(response.choices[0].message.content)
print(response.usage.prompt_tokens, response.usage.completion_tokens)

Existing code written against OpenAI moves across by swapping the base URL. The library does not have to change.

import os
import openai

client = openai.OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)

stream = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "List five risks in this migration."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Compatibility is high but not complete, and the documentation names the gaps plainly. The logprobs, logit_bias, top_logprobs and messages[].name fields return a 400. The N parameter, if supplied, must equal one. A temperature of zero is converted to 1e-8. Audio transcription and translation do not support the vtt and srt output formats. The Responses API is supported, so code written against the newer shape has somewhere to land as well.

Tool calling works in the standard way, and it is where the speed pays off most visibly, because every turn of the loop is a separate generation.

import Groq from 'groq-sdk'

const groq = new Groq({ apiKey: process.env.GROQ_API_KEY })

const tools = [
  {
    type: 'function' as const,
    function: {
      name: 'get_order_status',
      description: 'Call this when the user asks about order status by number.',
      parameters: {
        type: 'object',
        properties: { orderId: { type: 'string' } },
        required: ['orderId']
      }
    }
  }
]

const completion = await groq.chat.completions.create({
  model: 'openai/gpt-oss-120b',
  service_tier: 'auto',
  tools,
  messages: [{ role: 'user', content: 'What is happening with order 88213?' }]
})

const call = completion.choices[0].message.tool_calls?.[0]
console.log(call?.function.name, call?.function.arguments)

Limits, plans and the token bill

Limits are measured in several units at once and apply at the organisation level rather than per key. The abbreviations are RPM and RPD for requests per minute and per day, TPM and TPD for tokens, ASH and ASD for audio seconds, and some organisations additionally get separate ITPM and OTPM, meaning distinct per-minute caps on input and output tokens. Whichever threshold you cross first decides, so fifty short requests can exhaust a limit sooner than one long one.

On concrete numbers the documentation says two different things, and it is more honest to give both. The table on the rate limits page, described in the surrounding text as the base limits for the developer plan, gives openai/gpt-oss-120b 30 requests per minute, a thousand per day, 8 thousand tokens per minute and 200 thousand per day. The table on the models page, in a column headed plainly "RATE LIMITS (DEVELOPER PLAN)", gives the same model 250 thousand tokens per minute and a thousand requests per minute. The gap is more than thirtyfold and cannot be reconciled from outside. Binding are the values shown on your organisation's limits page in the console, and for capacity planning treat the lower set as the cautious assumption.

The plans stack in three levels. Free grants API access and lower limits. Developer, which is paid, raises the limits and unlocks two things unavailable before: the Batch API and the Flex tier. Enterprise opens the Performance tier and individually negotiated limits.

The tier is chosen through the service_tier parameter. The default is on_demand, with LPU speed and occasional queueing at peak. The flex tier is available to paying customers only, gives ten times higher limits at the same price, and charges for it differently: when capacity runs out, the request fails fast with status 498 and error capacity_exceeded, so retry handling with jittered backoff is mandatory here rather than optional. The auto tier picks the best one available at the moment.

Throughput guarantees do exist, but only in the Performance tier and only on the Enterprise plan. The documentation states a 99.9 percent availability SLA and a 99 percent latency guarantee, with the caveat that details live in the agreement. Billing there is not per token: you buy provisioned input and output capacity and pay for that, and there is no price on the page, only a route to the sales team. On top of that comes the context condition mentioned earlier, below 8,192 tokens without caching. On the free and developer plans there is no latency guarantee at all.

Two mechanisms bring the bill down, and both work with no code changes or nearly none.

Prompt caching switches itself on, costs nothing extra, and gives a fifty percent discount on cached input tokens. For openai/gpt-oss-120b that means 0.075 USD instead of 0.15 USD per million. Matching works on the prefix, so the static part of the prompt, meaning the system instruction, tool definitions and examples, has to sit at the start, and the variable part, meaning the user question, timestamps and session identifiers, at the end. The cache expires after two hours without use, currently covers only openai/gpt-oss-20b, openai/gpt-oss-120b and openai/gpt-oss-safeguard-20b, and a hit is not guaranteed. Cached tokens do not count against limits, although they are subtracted only after processing, so with many parallel requests a limit can still be crossed.

The Batch API gives a fifty percent discount and leaves synchronous limits untouched. The processing window is set from one day to seven, and a longer window raises the chance that a job completes rather than expires. Discounts do not stack: every token in a batch is billed at the batch rate regardless of whether it hit the cache. The Batch API also does not accept the service_tier parameter.

{"custom_id": "ticket-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-oss-20b", "messages": [{"role": "user", "content": "Classify: customer cannot log in"}]}}
{"custom_id": "ticket-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-oss-20b", "messages": [{"role": "user", "content": "Classify: invoice with the wrong amount"}]}}

Limit state shows up in the headers of every response, and that is the simplest way to spot a problem before it turns into an outage.

import httpx

response = httpx.post(
    "https://api.groq.com/openai/v1/chat/completions",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "model": "openai/gpt-oss-120b",
        "service_tier": "flex",
        "messages": [{"role": "user", "content": text}],
    },
    timeout=60.0,
)

if response.status_code == 429:
    wait = float(response.headers.get("retry-after", "2"))
elif response.status_code == 498:
    wait = 1.0

print(response.headers.get("x-ratelimit-remaining-requests"))
print(response.headers.get("x-ratelimit-remaining-tokens"))
print(response.headers.get("x-ratelimit-reset-tokens"))

The x-ratelimit-limit-requests header always refers to requests per day and x-ratelimit-limit-tokens always to tokens per minute, which is inconsistent and easy to misread. The retry-after header appears only on a 429 response, the others are always present.

Groq against the alternatives

The choice turns on one question: whether latency is a product constraint for you. If it is, because you are building a multi-step agent or an interface where a human waits, Groq offers something prompt optimisation alone will not produce. If it is not, the narrow catalogue and four deprecation rounds in half a year are a price there is no reason to pay.

A sensible middle arrangement is to avoid choosing. A brokering layer such as LiteLLM or OpenRouter lets you keep Groq for time-sensitive paths and route harder reasoning to Claude or another stronger model, without scattering the code across two libraries. For local work and data that cannot leave the company there is Ollama, where the same gpt-oss weights run on your own machine, only much more slowly.

The closest competitor on the open-weight side is Together AI, which bets not on its own hardware but on catalogue breadth, and adds fine-tuning and GPU cluster rental. For the same model the rates are often identical, so the difference comes down to what you are after: single-call speed or a choice among hundreds of models. Two things to know there: it has no free plan, because the account is prepaid from five dollars, and the model licence stays your problem, with some catalogue entries carrying revenue thresholds above which a separate agreement with the weight author is required.

Common mistakes

The first is npm install groq. The package under that name belongs to Sanity and is a query language, not an API client. The correct name is groq-sdk.

The second is a model identifier hardcoded into the source. With four deprecation rounds in six months it is a matter of time before the call starts returning an error. Keep the identifier in configuration and check the deprecation page before releasing.

The third is copying documentation examples without reading them. The Batch API, Flex tier and Performance tier pages still list llama-3.3-70b-versatile and llama-3.1-8b-instant, shut down on 16 August 2026.

The fourth is using the flex tier without handling error 498. This is not an ordinary network failure but documented behaviour when capacity runs out, and without retries with jittered backoff you will meet it at every peak.

The fifth is variable content at the start of the prompt. A timestamp or session identifier placed before the system instruction invalidates the prefix and caching stops working, with no message at all. Static first, variable last.

The sixth is capacity planning based on the table from the models page. It gives 250 thousand tokens per minute where the rate limits page gives 8 thousand for the same plan. Binding are the values in your console.

The seventh is assuming the quoted tokens per second are a guarantee. Without the Enterprise plan there is no SLA at all, and even there the latency guarantee covers only prompts below 8,192 tokens without caching.

The eighth is counting on stacked discounts. Tokens sent in a batch are billed at the batch rate regardless of cache hits, so the two discounts will not apply together.

FAQ

Is Groq the same thing as xAI's Grok?

No. Grok is xAI's family of models. Groq is an inference provider that trains nothing and instead serves other people's open-weight models on its own LPU silicon. The names differ by one letter and that is all they share.

Can existing code written against OpenAI be moved over?

Yes, by changing the base URL to https://api.groq.com/openai/v1 and swapping the key. The exceptions are listed in the documentation: the logprobs, logit_bias, top_logprobs and messages[].name fields return a 400, the N parameter must equal one, a temperature of zero becomes 1e-8, and audio does not support the vtt and srt formats.

What does a call actually cost?

On openai/gpt-oss-120b the rate is 0.15 USD per million input tokens and 0.60 USD per million output, on openai/gpt-oss-20b 0.075 and 0.30 USD respectively. Cached input costs half. The Batch API takes half off the total but does not stack with caching. Whisper is billed per hour of audio, from 0.04 USD on the turbo variant.

Does Groq guarantee throughput and latency?

Only in the Performance tier available on the Enterprise plan, where the documentation states a 99.9 percent availability SLA and a 99 percent latency guarantee, billed as provisioned capacity rather than per token. A condition applies: uncached context below 8,192 tokens. On the free and developer plans there is no guarantee.

What happens when Groq deprecates a model I use?

You get an email to the organisation address and an entry on the deprecation page with a shutdown date and a recommended replacement. After that date, requests to the old identifier return an error. Deprecations apply to the free and developer plans; Enterprise customers on a committed-spend contract are exempt.

Is the free plan enough for production?

No. It offers neither the Batch API nor the Flex tier, the limits are low, and there are no guarantees. It suits a prototype and a check on whether the speed genuinely changes your use case. Production needs at least the developer plan.

Current models, rates and limits are described in the Groq documentation, the deprecation history on the deprecations page, and the client library source sits in the groq-python repository.