Replicate, machine learning models through an API

Replicate, machine learning models through an API

Replicate exposes machine learning models as ordinary HTTP calls. You do not configure CUDA drivers and you do not reserve graphics cards: you send JSON and receive an image, a piece of text or an audio file. The bill, however, is calculated in two completely different ways, and that is the single most important thing to understand before your first deployment.

What you are actually buying

The service sells three things at once, and confusing them is the most common source of invoice surprises.

The first is the catalogue of public models. Somebody else packaged a model, published it under an address shaped like owner/name, and you run it with a single call. You have no say over the hardware or the queue, because you share a machine pool with other customers. The second is private models. You package your own code into a container image with Cog, push it to Replicate, and get an identical interface but on hardware assigned only to you. The third is deployments, a layer over both of those, giving a stable address, a hardware choice and scaling control through the min_instances and max_instances fields.

The programming interface is shared. You create a prediction with POST /predictions, optionally supply a webhook address, and collect the result by polling, by webhook or as a stream. Request limits are 600 per minute for creating predictions and 3000 per minute for the remaining endpoints. Accounts granted credit without a payment method on file are limited to one request per second and a maximum of six per minute, which in practice rules out any load testing on a free account. Once you exceed a limit you get a 429 with a body like {"detail":"Request was throttled. Your rate limit resets in ~30s."}.

The Python library looks like this:

import replicate

output = replicate.run(
    "black-forest-labs/flux-schnell",
    input={"prompt": "harbour map in a technical drawing style"},
    wait=30,
    use_file_output=True,
)

The wait parameter accepts a boolean or a number of seconds from 1 to 60. With True the request is held open for up to 60 seconds and then the client falls back to polling. The remaining parameters accepted by replicate.run and by replicate.predictions.create are webhook, webhook_completed, webhook_events_filter, stream and file_encoding_strategy. The predictions.create method requires exactly one of model, version or deployment, otherwise it raises ValueError.

Two billing modes that produce different bills

This is the core of the article. Some public models are billed for hardware time and some per unit of output: an image, a second of video, a thousand tokens. For the same task the two methods produce bills that differ by an order of magnitude, and the choice between them is not yours, it follows from how a given model was published.

Here are the per-output rates from the provider's price list as of 22 August 2026:

The conversion column is not decoration. The price list quotes figures in mixed units on a single page: flux-schnell per thousand images, flux-dev per single image, and for language models the input is counted per million tokens while the output is counted per thousand. The claude-3.7-sonnet card shows USD 0.015 next to USD 3.00, which makes the output look two hundred times cheaper than the input. Reduced to a common unit it is the other way round: USD 15 per million output tokens against USD 3 per million input tokens, so the output costs five times more. The same trap applies to deepseek-r1: USD 10 per million out against USD 3.75 per million in.

The billing documentation gives a worked example on meta/meta-llama-3.1-405b-instruct: 8 input tokens cost USD 0.0000760, 43 output tokens cost USD 0.0004085, for a total of 51 tokens at USD 0.0004845. The arithmetic holds and both directions work out to USD 0.0000095 per token, that is USD 9.50 per million regardless of direction. This is the rare case where input and output carry the same price.

Hardware rates and hourly arithmetic

The second billing mode is easier to compute, because you multiply seconds by a rate. Below are the rates from the price list, checked for consistency between the per-second and the hourly figure:

Every pair multiplied by 3600 seconds matches to the cent, so the provider applies no hidden correction here. Two things in this table need comment. First, gpu-h200 costs exactly the same as gpu-h100 despite the larger card memory, but it is available only under a committed spend contract. The same applies to every multi-card variant above two units. Second, the price list gives no memory specification for the hardware in the additional section, so guessing is a bad idea.

Now let us compute both modes on one task. I am making an assumption the price list does not confirm, because it depends on the specific model: generating a single image takes three seconds on an H100. Under that assumption a thousand images is 3000 seconds of active time, that is USD 4.58. The same thousand images through the public flux-schnell costs USD 3.00, and through flux-1.1-pro already USD 40.00.

The bill looks different once idle time enters the picture. A gpu-h100 instance kept up around the clock is 86400 seconds times USD 0.001525, that is USD 131.76 a day. If it serves a hundred requests a day at three seconds each, active time is 300 seconds, that is USD 0.46, and you pay the rest for a machine waiting for traffic. Those same hundred images through the public flux-1.1-pro cost USD 4.00. The break-even point under my three-second assumption falls around 3294 images a day, because that many units at USD 0.04 add up to USD 131.76. Above that threshold your own instance starts to pay off, below it you overpay for idling.

Cold boots, idle time and keeping a model warm

The documentation splits an instance lifecycle into four states: offline, setting up, active and idle. Setup covers downloading the model weights and can take a few seconds. Idle is the few minutes after the last request during which the instance is not shut down, so it avoids setting up again.

Who pays for which state depends on the model type, and this is a rule to memorise literally. For public models you pay only for active time, while setup and idle are free. The price is paid another way: you share a queue with other customers, so cold boots and scaling limits happen depending on how other people's traffic is going. For private models and for deployments you pay for the entire time the instance is online, that is for setup, for idle and for work.

The exception is what the provider calls fast booting fine-tunes. These are tuned models run on a shared hardware pool alongside the base model, billed only for active time regardless of whether they are public or private. The documentation says such versions are labelled in the model's version list, but it does not define the conditions under which a model qualifies for this mode. Which means you cannot plan for it in advance.

Keeping a model warm is done with a deployment that sets min_instances. The structure is short:

{
  "name": "my-app-image-generator",
  "model": "some-org/some-model",
  "version": "da77bc59ee60423279fd632efb4795ab731d9e3ca9705ef3341091fb989b7eaf",
  "hardware": "gpu-t4",
  "min_instances": 1,
  "max_instances": 5
}

The min_instances value defaults to zero, so without this setting the model scales down to zero and the next request goes through a full setup. Setting it to one eliminates the cold boot but switches you to the daily billing described above. The max_instances field is the only mechanism for capping maximum spend that the provider exposes in the deployment configuration.

The billing documentation contains a contradiction at this point that is easy to miss. The section on failed runs opens with the sentence that for all models a failed run is not charged, and two sentences later states that for private models and deployments failed and cancelled runs are billed for the time the instances were active. The second sentence invalidates the first. For official models, cancelling can also be chargeable. If you are building a cost estimate, count according to the more expensive variant.

Cog, or your own model in a container

Cog is a tool for packaging a model into a container image with an HTTP server. The replicate/cog repository carries a LICENSE file with the full Apache 2.0 text, and release 0.22.0 landed on 14 August 2026, eight days before the date of this text. The previous release, 0.21.0, comes from 16 June 2026, so the gap was 59 days. The tool is being developed, unlike the client libraries described in the next section.

The answer to whether pushing a model here is a one-way street is: no. An image built by Cog is an ordinary Docker image with a server listening on port 5000 that exposes POST /predictions, PUT /predictions/<prediction_id> and a GET / returning the addresses of the other endpoints. The Prefer: respond-async header switches to asynchronous mode and Accept: text/event-stream to event streaming. The image field in cog.yaml points at r8.im, Replicate's registry, by default, but the documentation says outright that it can be any Docker registry. So you can build the image with Cog and run it on your own cluster.

The configuration file has three top-level keys: build, image and run.

build:
  gpu: true
  python_version: "3.13"
  python_requirements: requirements.txt
  system_packages:
    - "ffmpeg"
    - "git"
image: "r8.im/your-username/your-model"
run: "run.py:Runner"

The build key accepts, among others, cuda, gpu, python_version, python_packages, python_requirements, sdk_version, system_packages and its own run with shell commands. The python_packages and python_requirements fields are mutually exclusive. The naming change deserves attention: earlier projects used a predict key pointing at a Predictor class, and the 0.22.0 documentation marks it as deprecated in favour of run pointing at a Runner class. The old names still work, but Cog prints a warning when it loads the model.

The model code itself looks like this:

from cog import BaseRunner, Path, Input
import torch

class Runner(BaseRunner):
    def setup(self):
        self.model = torch.load("weights.pth")

    def run(
        self,
        image: Path = Input(description="Image to enlarge"),
        scale: float = Input(description="Factor to scale image by", default=1.5),
    ) -> Path:
        return self.model(image)

The setup method is optional and serves to load the weights, run is required. Since version 0.14.0 both may be asynchronous, and since 0.21.0 there is a @cog.concurrent(max=N) decorator that sets the maximum number of concurrent runs. Exceeding the limit returns a 409. The max value must be an integer literal because it is baked into the image at build time, and the COG_MAX_CONCURRENCY environment variable lets you override it at runtime. Input and output types include cog.Path, cog.Secret and custom classes based on BaseModel. Installation on macOS comes down to brew install replicate/tap/cog.

One thing in the dependencies is interesting. The cog package at version 0.22.0 requires coglet at version at least 0.22.0 and below 1.0, and coglet describes itself as a high-performance prediction server for Cog models written in Rust. The actual execution loop has therefore moved into a separate component.

Licences and the state of the client libraries

I checked the licence from three sources separately, because the most interesting defect in the whole set turned up here.

The LICENSE file in the replicate/cog, replicate/replicate-python and replicate/replicate-javascript repositories contains the full Apache 2.0 text, and there are no surprises there. The npm package replicate declares a registry license field of Apache-2.0, and the published 1.4.0 tarball contains a package/LICENSE file plus sixteen JavaScript files taking about 192 kilobytes, so there is real code in there rather than a stub. There are no runtime dependencies, and the engines field requires Node at version 18 or higher.

The discrepancy is on the PyPI side. The license field in the replicate package metadata does not hold a licence identifier but the entire Apache 2.0 text: 12921 characters stretched across the METADATA file from line six to line two hundred and eight, after which the Project-URL header finally begins. The license_expression field is empty and there is not a single classifier starting with License ::. The cog package at version 0.22.0 carries exactly the same bug. The practical effect is this: an automated licence scanner expecting a short SPDX identifier will receive two hundred lines of legal prose in that field and will either report the licence as unrecognised or write the first line of the text into its report. The licence file itself is fine, it sits under replicate-1.0.7.dist-info/licenses/LICENSE, and the package holds twenty-one Python modules totalling about 130 kilobytes. Curiously, the coglet package declares Apache-2.0 correctly, so this is an oversight in two projects' configuration rather than a deliberate decision.

A more serious problem concerns the release pace on the Python side. The last stable release of the replicate package is 1.0.7 from 27 May 2025. Since then only prereleases have appeared: 1.1.0b1 on 9 June 2025, 1.1.0b2 on 12 June 2025, 1.1.0b3 on 26 August 2025, and in parallel the whole 2.0.0 line from 2.0.0a1 on 10 June 2025 to 2.0.0b4 on 18 December 2025. Nothing has shipped since that day. Counting from today's date that is fifteen months without a stable release and eight months without any release at all. The JavaScript library is barely better off: 1.4.0 comes from 17 November 2025, and the alpha tag points at 2.0.0-alpha.74 from 18 December 2025, the same day as the last Python beta.

The service works, replicate.com answers with a 200, Cog gets releases every few weeks, but the two official SDKs are stalled. If you are building production on them, assume that patching client bugs will be on you, or write straight against the HTTP interface, which is documented and stable.

Replicate against Modal, Hugging Face and self-hosted vLLM

The difference against Modal is fundamental and does not reduce to pricing. In Modal you write your own Python code and annotate it with decorators, and the provider handles assembling the environment. Billing there is strictly per resource second, so there is no equivalent of a per-image rate. On Replicate the typical use starts with somebody else's catalogue model that you never wrote, and you add a private version only once the catalogue stops being enough. If your task is your own compute logic, Modal fits better. If it is running a well-known model without writing inference code, Replicate wins on time to deployment.

Hugging Face is above all the place you take weights and model code from. Replicate is the place where a model already runs behind an HTTP address. The two products often appear together, because a public model on Replicate usually pulls its weights from the Hub inside the setup method. If it matters to you that the weights stay in-house, running it yourself through vLLM or locally through Ollama gives full control and zero dependence on somebody else's price list, at the cost of driver, queue and scaling maintenance landing on you. For a comparison of language model API rates alone, check Groq and OpenAI, and if you need an isolated environment for running model-generated code, that is an entirely different category covered under E2B.

The vendor lock-in risk here is moderate but not zero. The Cog format and the container image are portable, and the Apache 2.0 licence permits it. What is not portable: the specific public model versions published by third parties, the deployment configuration and the whole scaling backend. Reproducing that in-house is feasible, but it is not an afternoon's work.

The closest relative to Replicate here is fal, which also offers a catalogue of ready models and also mixes two billing modes, but concentrates on image, video and audio rather than language models. The practical difference lies elsewhere: fal has a separate queue mode with a webhook, designed for jobs running for minutes, which for video generation is the only sensible route. Two drawbacks on that side: there is no free plan, because billing is prepaid, and the JavaScript client and the Python client carry two different licences, MIT and Apache 2.0, which complicates the licence inventory in a project using both.

Common mistakes

Comparing rates without reducing them to a common unit. The price list mixes thousands and millions within a single model card, so until you convert both numbers to the same base the comparison is meaningless.

Assuming a private model is cheaper than a public one because there is no margin on ready output. Under low and irregular traffic the opposite holds, because you pay for setup and idle time, which you do not pay for at all on a public model.

Setting min_instances to one reflexively to get rid of the cold boot. On an H100 card that is a USD 131.76 per day decision, whether a hundred requests arrive or none.

Relying on the documentation sentence that failed runs are not charged. That applies to public models, while private models and deployments follow the opposite rule, stated two sentences further down in the same paragraph.

Load testing on an account with granted credit and no payment method. A six requests per minute limit means you will measure the limit rather than the model.

Pinning a dependency to a prerelease version of the replicate package. The 2.0.0 line stopped at a beta in December 2025 and there is no sign it will be finished.

Trusting an automated licence scanner during a Python dependency audit. For the replicate and cog packages the license field carries the entire licence text instead of an identifier, so the report needs a manual correction.

FAQ

Is Replicate suitable for production?

The service works and is being developed, and the HTTP interface is stable and documented. The weak point is the official client libraries: the Python one has had no stable release since May 2025 and the JavaScript one since November 2025. For production use it is wiser to build on raw HTTP calls or to prepare for maintaining your own client patches.

Can you use Cog without Replicate?

Yes. Cog is licensed under Apache 2.0, it builds an ordinary Docker image with a server on port 5000, and the image field in cog.yaml can point at any registry, not only r8.im. You can run the image on your own cluster and call it through POST /predictions.

How do you avoid paying for a model sitting idle?

On public models you do not pay for idling at all, because only active time is billed. On private models and deployments you pay for every second the instance is online, so the only route is leaving min_instances at zero and accepting the cold boot, or checking whether your tuned model qualifies as a fast booting fine-tune.

What does the free plan cost?

The provider gives no number. The documentation only says that selected models can be run for free but that after a while you will be asked to set up billing, and that some features require billing to be enabled. Without a concrete limit you cannot base any plan on it.

Are the H100 and H200 rates really identical?

According to the price list of 22 August 2026, yes, both are USD 0.001525 per second and USD 5.49 per hour. The difference is availability: the H200 is offered only under a committed spend contract, not in self-service.

How does Replicate differ from Modal?

Modal is a platform for running your own Python code billed strictly per resource second. Replicate starts from a catalogue of ready models and allows two billing modes, including a per-unit-of-output rate. The Modal side is covered in a separate article.

Sources checked directly: the Replicate price list, the billing documentation, the Cog repository and the metadata of the replicate package on PyPI.