Together AI, Inference on Somebody Else's Models

Together AI, Inference on Somebody Else's Models

Together AI is an inference provider: it serves other people's open-weight models through an OpenAI-compatible API, and adds fine-tuning and GPU rental on top. The Python client together is at version 2.31.0 under Apache 2.0, the TypeScript client together-ai at 0.49.0 under the same license. The client license says nothing about the license of the model you call, and that is the single most important thing to understand about this platform.

What Together AI sells and what it does not

The offering splits into four separately billed products. Serverless inference means per-token calls on shared infrastructure with nothing to reserve. Dedicated inference means a model placed on carved-out hardware, billed per GPU hour. GPU clusters are raw machines by the hour or on longer reservation terms. Fine-tuning sits alongside, billed on training-set tokens.

The serverless catalog is narrower than the talk of hundreds of models suggests. In the documentation table of chat models I counted twenty-four entries, among them openai/gpt-oss-120b, meta-llama/Llama-3.3-70B-Instruct-Turbo, Qwen/Qwen3.5-397B-A17B, moonshotai/Kimi-K3, zai-org/GLM-5.2 and deepseek-ai/DeepSeek-V4-Pro. Image and video models number a few dozen each. Embedding models in serverless, on the other hand, number exactly one, intfloat/multilingual-e5-large-instruct at 0.02 USD per million tokens, and rerank models are absent entirely: the documentation points you to dedicated inference for those. If you are building search with a reranking stage, that one line in the docs decides your architecture.

What Together does not do: there is no self-hosted variant. There are also no in-house models that would be a reason to stay. The whole value lies in somebody else maintaining the servers and refreshing the catalog while you pay per token. That is also the main risk: the model your product rests on can leave the catalog, and then you are rewriting a model identifier and re-running quality evaluation.

Packages, versions, and client licenses

The package names are asymmetric, and that is the first trap. On PyPI the official package is called together, on npm it is together-ai. The reversed names do not work: together-ai does not exist on PyPI, and together on npm is an unrelated HTML-assembly package at version 0.0.11, last published on 25 November 2014, under MIT. Installing the wrong name fails silently and pulls in something else entirely.

# Python: the package is called together, not together-ai
pip install together

# TypeScript: the package is called together-ai, not together
npm install together-ai

# check what actually got installed
pip show together | head -3
npm view together-ai version license repository.url

# the Python client installs two terminal commands
together --help
tg --help

License verification from three sources comes out clean here, which in this collection is not the rule. The LICENSE file in the togethercomputer/together-py repository holds the full Apache 2.0 text with a filled-in "Copyright 2026 Together" notice. The license field in both registries reads Apache-2.0. The published packages carry the same file: the source archive and the Python wheel have LICENSE at the root and under dist-info/licenses, the npm package has LICENSE at the root. There is real code inside the packages, not empty stubs.

Two details from the unpacked archives deserve a mention. The npm package holds an extra src/internal/qs/LICENSE.md under BSD 3-Clause, belonging to a vendored copy of the neoqs library; a dependency audit has to list it separately. The second detail is on the Python side: the required dependency detect-agent at version 0.6.0 has neither a license field nor a license classifier on PyPI, even though its wheel contains an Apache 2.0 file with the same "Copyright 2026 Together" notice. A license scanner that reads registry metadata only will flag that dependency as undetermined.

Repositories are a separate trap. The Python 2.x client code lives in togethercomputer/together-py. The older togethercomputer/together-python still exists at its own address, does not redirect, and its last release is v1.5.35 from 21 January 2026. The LICENSE file in that old repository carries an unfilled Apache template with the phrase "Copyright [yyyy] [name of copyright owner]". Citing it today as the client repository is simply out of date. On top of that, README.md inside the 2.31.0 package states a requirement of Python 3.9 or newer, while pyproject.toml in the same package has requires-python = ">= 3.10". The pyproject.toml value is the one that counts, because it is the one that blocks installation.

The SDK license is not the model license

Apache 2.0 on the client covers the client code. The model you call through that code has its own terms, and you are their addressee, not Together. The platform takes no responsibility for you using a model against its license.

Together publishes the model license in the API but not in the documentation. The GET /v1/models endpoint returns an object with an optional license field next to organization, context_length, link, and a nested pricing carrying input, output, cached_input, base, finetune, and hourly. The catalog table in the documentation, by contrast, has columns for context, prices, quantization, and function calling, and no license column at all. Individual model pages on the website do not show license text either. The practical conclusion: pull the license from the API or check it with the author of the weights.

import os

from together import Together

client = Together(api_key=os.environ["TOGETHER_API_KEY"])

for model in client.models.list():
    if model.type != "chat":
        continue
    price = model.pricing
    print(
        model.id,
        model.organization,
        model.license or "no license field",
        price.input if price else None,
        price.output if price else None,
        sep=" | ",
    )

Three models from the serverless catalog show why this is not a formality. openai/gpt-oss-120b and Qwen/Qwen3.5-397B-A17B carry Apache 2.0 on Hugging Face, zai-org/GLM-5.2 and deepseek-ai/DeepSeek-V4-Pro carry MIT. Up to that point nothing happens. But meta-llama/Llama-3.3-70B-Instruct carries the tag license: llama3.3, meaning Meta's own community license with its own conditions rather than one of the permissive licenses.

Two models go further and write revenue thresholds into the text. The Qwen/Qwen3.8-2.4T-A95B weights fall under a bespoke license named the Qwen3.8-Max License. It requires that for a product with more than 100 million monthly active users or more than 20 million dollars in monthly revenue, the model name be prominently displayed in the interface. Separately: if the licensee runs a Model as a Service business or an AI work assistant, and the aggregate revenue of the licensee and its affiliates exceeds 50 million dollars over any consecutive twelve months, it must obtain a separate license from Qwen before any commercial use. Similarly, moonshotai/Kimi-K3 carries the Kimi K3 License with a 20 million dollar aggregate-revenue threshold over any consecutive twelve months for Model as a Service businesses, plus the same display duty above 100 million monthly users or 20 million dollars in monthly revenue.

Both licenses exempt internal use from these requirements, meaning use that does not make the model, its outputs, or its capabilities available to third parties. If, on the other hand, you build your own customer-facing API on Together, you are exactly the addressee the Model as a Service clause describes. More context on the model families sits in the pieces on Qwen, Meta's Llama, and Hugging Face, where the weights themselves live. Picking an entry from the model list is always also picking a license.

The first call and OpenAI compatibility

The base address is https://api.together.ai/v1. The client reads TOGETHER_API_KEY, TOGETHER_BASE_URL, TOGETHER_PROJECT_ID, and TOGETHER_CUSTOM_HEADERS; the TypeScript version adds TOGETHER_LOG. The compatibility layer lets you attach existing code written against the OpenAI client by swapping the key and the address.

import Together from 'together-ai'

const client = new Together({
  apiKey: process.env.TOGETHER_API_KEY,
})

const completion = await client.chat.completions.create({
  model: 'openai/gpt-oss-120b',
  messages: [{ role: 'user', content: 'Give three advantages of FP4.' }],
  max_tokens: 512,
  reasoning_effort: 'low',
})

console.log(completion.choices[0]?.message?.content)

Compatibility is not complete, and the documentation lists the exceptions. Unsupported are assistants, threads and runs, batches in the OpenAI shape, fine_tuning.jobs in the OpenAI shape, and moderations.create. The files resource works partially, because Together has its own files API for training sets and batch jobs. The logit_bias parameter does not work on most models. The service_tier, store, metadata, and prediction parameters are accepted and ignored, as is the detail field on images.

The costliest difference concerns the shape of the usage field and can quietly zero out your metrics. Reasoning models, for example zai-org/GLM-5.2 or Qwen/Qwen3.6-Plus, nest the counters the new way: cached prompt tokens under usage.prompt_tokens_details.cached_tokens, reasoning tokens under usage.completion_tokens_details.reasoning_tokens. Some non-reasoning models, including meta-llama/Llama-3.3-70B-Instruct-Turbo, return cached_tokens flat at the top level of usage, with no *_details objects. A client prepared for one shape only returns zero, with no error whatsoever. The chain of thought comes back in a reasoning field on the assistant message, and the older reasoning_content key is still accepted on input.

Pricing worked out on concrete numbers

Serverless rates per million tokens, from the provider's pricing page as of 22 August 2026:

The most useful comparison is the same model at other providers. For openai/gpt-oss-120b the OpenRouter interface lists twenty endpoints. Together appears there at 0.15 USD for input and 0.60 USD for output, exactly matching its own price list. Groq has an identical 0.15 and 0.60. The cheapest entry on that list is CoreWeave at 0.03 USD input and 0.17 USD output at FP4 quantization, with DeepInfra at 0.037 and 0.17 at BF16. For an output-dominated workload the gap between Together and the cheapest entry is roughly three and a half times. The caveat matters: providers differ in quantization, context length, and real throughput, so a lower rate does not always mean the same quality of result.

Batch API prices look identical to serverless on the pricing page for the models listed in both tables. MiniMax M3 is 0.30 and 1.20 USD in both, Kimi K3 is 3.00 and 15.00 USD in both, Gemma 4 31B is 0.39 and 0.97 USD in both. The rate-limits documentation, meanwhile, speaks of discounts on most models in batch mode. That is a discrepancy between two surfaces of the same provider, and for budgeting it is safer to take the table rates and treat the discount as unconfirmed.

Dedicated endpoints and GPU clusters bill per GPU hour, and the arithmetic there can surprise. A dedicated endpoint on NVIDIA HGX H100 costs 5.49 USD per GPU hour, on HGX B200 8.99 USD; the remaining hardware is listed as contact sales. A GPU cluster with the same H100 costs 3.99 USD per GPU hour on demand, so the endpoint is just under 38 percent more expensive than the raw machine for the same card. Reserving a cluster lowers the rate in steps: 3.69 USD for 7 to 30 days, 3.45 USD for 31 to 90 days, 3.19 USD for 91 to 180 days, and above 180 days the price list refers you to sales. A month of one H100 on a dedicated endpoint comes to 3952.80 USD over thirty days. For the same amount on Llama 3.3 70B in serverless you get about 3.8 billion tokens, counting at the flat 1.04 USD per million in both directions. Until you are pushing billions of tokens a month through a single model, dedicated hardware does not pay for itself.

endpoint = client.endpoints.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    hardware="2x_nvidia_h100_80gb_sxm",
    autoscaling={"min_replicas": 1, "max_replicas": 3},
    display_name="llama-33-prod",
    inactive_timeout=30,
    disable_speculative_decoding=False,
    state="STARTED",
)

print(endpoint.id, endpoint.state)

Take the hardware identifier from client.endpoints.list_hardware(), which returns an id field alongside pricing.cents_per_minute and specs.gpu_count; the 2x_nvidia_h100_80gb_sxm used above comes from the examples built into the package CLI and means two cards, so double the hourly rate. Setting min_replicas to 1 means you pay overnight too. Setting inactive_timeout to a number of minutes shuts the endpoint down after an idle period, while zero, null, or omitting the field disables the shutdown itself. The disable_prompt_cache field still exists in the schema, but the documentation describes it as deprecated and without effect.

Fine-tuning is billed on the sum of training-set tokens multiplied by the number of epochs plus validation-set tokens multiplied by the number of evaluations, with a minimum charge of 4.00 USD per job. For models up to 16 billion parameters the rates start at 0.48 USD per million tokens for supervised training and 1.20 USD for preference optimization. The price list shows two columns of numbers under each of those two headings, but the sub-column captions do not render without JavaScript, so I am not assigning them to either LoRA or full fine-tuning. For models called out by name the rates are higher: gpt-oss-120B is 5.00 USD for supervised LoRA training and 12.50 USD for preference optimization, with a minimum charge of 6.00 USD per job.

job = client.fine_tuning.create(
    training_file="file-abc123",
    model="openai/gpt-oss-120b",
    n_epochs=3,
    validation_file="file-def456",
    n_evals=3,
    learning_rate=1e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    batch_size="max",
    lora=True,
    lora_r=16,
    lora_alpha=32,
    lora_trainable_modules="all-linear",
    training_method="sft",
    suffix="support-bot",
    early_stopping_enabled=True,
    early_stopping_patience=2,
)

One detail from the unpacked package: in the synchronous version of the create method the lora parameter defaults to None, while in the asynchronous version it defaults to True. Set it explicitly rather than relying on the default.

Rate limits and prepaid billing

Together applies dynamic limits, set separately per organization and per model. The former account-tier labels, meaning Build Tier 1 through 5 plus Scale and Enterprise, have been retired and no longer appear in the account or in API responses. The limit grows with sustained, successful traffic, and sudden spikes far above recent usage are trimmed. A 429 response arrives with the error type dynamic_request_limited or dynamic_token_limited and with an x-ratelimit-reset header giving the suggested pause in seconds. A separate case is 503, returned when the request fit within the limit but the model was overloaded.

Here the documentation contradicts itself. The serverless rate-limits page states that successful requests come back without rate-limit headers and that x-ratelimit-reset appears only on a 429. The usage-limits page in the billing section states that every serverless inference request returns headers with current limits, usage, and reset timing. Both versions come from the provider and they exclude each other, so the code has to cope with the header being absent.

response = client.chat.completions.with_raw_response.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "ping"}],
)

reset = response.http_response.headers.get("x-ratelimit-reset")
if reset is None:
    print("status:", response.http_response.status_code, "(not rate limited)")
else:
    print("rate limited, retry in", reset, "seconds")

There is no free plan. The documentation states outright that Together does not currently offer free trials and that access to the platform requires a minimum 5 dollar credit purchase. Billing is fully prepaid: at a zero balance API access is suspended until you top up. Credits carry no expiry date, but credits bought after an invoice has been issued cannot settle earlier past-due balances. Auto-recharge works only when the default payment method is a card; setting a bank transfer as the default switches it off by itself.

Together next to Groq, OpenRouter, Replicate, and vLLM

Together sits between two extremes. On one side is vLLM, meaning running the weights on your own hardware with full control and the full maintenance cost. On the other is OpenRouter, which serves nothing itself and only routes requests to providers, taking a commission when you top up your account. Together serves the models itself, so the middleman layer drops away, but the catalog is single and closed. Against Groq the difference lies in hardware and scope: Groq bets on speed on its own chips, Together adds fine-tuning and cluster rental. Against Replicate the difference lies in what you can upload: Replicate lets you run your own image, Together confines you to its catalog and to models fine-tuned on its own platform.

A sensible choice looks like this. Take Together when you want one provider for both inference and fine-tuning, you accept rates above the cheapest on the market, and you value being able to put a fine-tuned model straight onto a dedicated endpoint. Do not take it when you are counting every cent at high volume, when you need a self-hosted variant for data reasons, or when your model is a single catalog entry whose disappearance would halt the product.

Common mistakes

Mixing up the package name. pip install together-ai will not work, and npm install together installs an HTML library from 2014. The direction is reversed in each of the two ecosystems.

Citing togethercomputer/together-python as the client repository. That is the 1.x line, frozen at v1.5.35 from 21 January 2026. The 2.31.0 code lives in togethercomputer/together-py.

Assuming Apache 2.0 on the client covers the model. It does not. The Qwen3.8-Max License and the Kimi K3 License carry revenue thresholds, the Llama 3.3 license has its own conditions, and Together does not show licenses in the catalog table.

Reading cached tokens from one place. Reasoning models return them in usage.prompt_tokens_details.cached_tokens, some of the rest flat in usage.cached_tokens. Read both places, because a missing value raises no error.

Spinning up a dedicated endpoint to try it and leaving it running. With min_replicas at 1 and no inactive_timeout, an H100 accrues 5.49 USD an hour without pause, close to 132 USD a day.

Budgeting on the batch discount. The price table shows the same rates as serverless for the listed models, even though the documentation speaks of discounts.

Counting on a fixed request limit. Limits are dynamic and grow with traffic; if you need a value known in advance, the documentation points you to dedicated inference.

FAQ

Does Together AI have a free plan?

No. The billing documentation states that free trials are not currently offered and that platform access requires a minimum 5 dollar credit purchase. The account is fully prepaid and at a zero balance the API is suspended.

Can I use the OpenAI client instead of the Together SDK?

Yes, by swapping the key and setting base_url to https://api.together.ai/v1. Outside that layer's reach are assistants, threads, runs, batches, and fine_tuning.jobs in the OpenAI shape, plus moderations.create, while files works partially.

Does calling a model through the API release me from its license?

No. The license on the weights applies regardless of who runs the server. For Qwen/Qwen3.8-2.4T-A95B a separate agreement with Qwen is required once you run a Model as a Service business and aggregate revenue exceeds 50 million dollars over consecutive twelve months; for moonshotai/Kimi-K3 the equivalent threshold is 20 million dollars.

How do I check a model's license before deployment?

The GET /v1/models endpoint returns an optional license field next to organization and link. The catalog table in the documentation has no license column, so the model card from the author of the weights is the more reliable source.

When is a dedicated endpoint cheaper than serverless?

An H100 hour costs 5.49 USD per GPU, meaning 3952.80 USD for thirty days of continuous running. On Llama 3.3 70B at 1.04 USD per million tokens in both directions, the same amount buys about 3.8 billion serverless tokens. Below that volume on a single model, dedicated hardware does not pay for itself.

Do rate limits depend on the account tier?

Not in the old sense. The Build Tier 1 through 5, Scale, and Enterprise labels have been retired. Limits are dynamic, set per organization and per model, and they grow with sustained successful traffic.