Modal, running code on GPUs without servers

Modal, running code on GPUs without servers

Modal is a paid platform where you mark a Python function with a decorator and the provider builds a container image, runs it on the GPU you name, and scales it from zero. The 1.5.4 client is open under the Apache 2.0 licence, but the platform itself is closed and there is no variant you can run on your own hardware.

What Modal actually does

The starting point is an ordinary Python file. In it you create a modal.App object, describe the container image using methods on the modal.Image class, and decorate the functions meant to run remotely with @app.function(). Once you run modal run, the client serialises your code, sends it to the service, which builds the image, starts a container on a machine with the accelerator you picked, and streams the result and the logs back to your terminal.

The set of objects the package exposes is fairly narrow, and that is an advantage. modal/__init__.py exports, among others, App, Image, Function, Cls, Volume, Secret, Dict, Queue, Sandbox, Tunnel, Proxy, Retries, Cron, Period, plus the decorators enter, exit, method, batched, concurrent, fastapi_endpoint, asgi_app, wsgi_app and web_server. Everything else is implementation detail.

There are three typical uses. The first is model inference, meaning an HTTP endpoint that loads weights once when the container starts and then serves requests. The second is batch processing, where Function.map fans thousands of inputs across hundreds of containers at once. The third is fine-tuning and training, where what matters is access to eight cards in a single machine without signing a reservation.

What Modal does not do matters just as much. It is not a database or a production queue, although it does have Dict and Queue for passing state between containers. It is not a platform for hosting ordinary web applications with a long-lived process, because the billing model punishes idleness. It does not support languages other than Python for defining jobs, although client packages for JavaScript and Go exist and are marked beta.

An open client and a closed platform

This distinction decides whether you should reach for Modal at all, so I will spell it out. Checking the licence against three independent sources gives a consistent answer, which in this collection happens less often than you might expect.

The first source is the LICENSE file in the modal-labs/modal-client repository. It is 176 lines long and it is the full text of Apache License 2.0. The second source is the metadata in the PyPI registry: the license_expression field reads Apache-2.0, while the old license field is empty, which matches the newer way of declaring licences in Python metadata. The third source, the one most often skipped, is the content of the published package. The modal-1.5.4-py3-none-any.whl wheel weighs 979 KB, contains 231 files including 173 .py files, and unpacks to 5.3 MB. The modal-1.5.4.dist-info/licenses/ directory holds a LICENSE file whose SHA-256 checksum matches the repository file character for character. There is no divergence here: the declaration, the repository and the package all say the same thing, and the package contains real code rather than a placeholder.

That is the end of the good news. Apache 2.0 covers the client library only, meaning the layer that packages your code and talks to the API over gRPC. The server, the scheduler, the image build system, the volume storage and the entire GPU infrastructure are closed and paid for. There is no self-hosted variant, no community edition, not even an emulation mode for testing without an account. The billing documentation states plainly that you must have a payment method on file in order to use Modal, so even the free account requires a card.

The practical consequence is that lock-in is total. A file containing @app.function(gpu="H100") and @modal.enter() will not run anywhere except Modal. Migration is not a matter of changing a URL, it means rewriting the runtime layer: decorators become a Dockerfile plus a deployment manifest, modal.Volume becomes the target platform's volume or an object bucket, modal.Secret becomes the target secret store, and autoscaling becomes another system's configuration. The computation itself, meaning loading a model and processing inputs, moves across unchanged, because it is plain Python.

You can limit that risk by design. Keep the domain logic in modules that import nothing from modal, and let the file with the decorators be a thin entry layer calling into those modules. Then a move costs one file per application rather than the whole repository.

The project itself is actively developed. Release 1.5.4 landed on 12 August 2026, and the repository publishes daily development builds, from 1.5.5.dev2 on 15 August through 1.5.5.dev9 on 22 August. The client requires Python between 3.10 and 3.14 inclusive and pulls in a dozen or so dependencies, including grpclib, protobuf, aiohttp, click, rich, watchfiles and synchronicity.

The first app and the programming model

Installing and authenticating takes two commands, and the rest of the CLI is predictable.

pip install modal

# writes a token to ~/.modal.toml, opens the browser
modal setup

# one-off run of a local file in the cloud
modal run app.py

# deploy under a name, endpoints get a stable address
modal deploy app.py

# live mode: editing the file rebuilds the container
modal serve app.py

# a shell inside a container built from your image
modal shell app.py::generate

# list deployed apps and running containers
modal app list
modal container list

# cost report, available on the Team and Enterprise plans
modal billing

The smallest useful application looks like this. Every field and parameter name comes from the signatures in modal/app.pyi in version 1.5.4.

import modal

image = (
    modal.Image.debian_slim(python_version="3.12")
    .pip_install("torch==2.5.1", "transformers==4.46.3")
    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
)

app = modal.App(name="inference-demo", image=image, tags={"team": "ml-platform"})

@app.function(
    gpu="L40S",
    cpu=4.0,
    memory=16384,
    timeout=600,
    startup_timeout=300,
    retries=modal.Retries(max_retries=2),
    secrets=[modal.Secret.from_name("hf-token", required_keys=["HF_TOKEN"])],
)
def embed(text: str) -> list[float]:
    import torch
    assert torch.cuda.is_available()
    return [0.0]

@app.local_entrypoint()
def main():
    print(embed.remote("hello"))

A few details in that file are worth understanding before the first bill surprises you. The timeout parameter defaults to 300 seconds and measures execution time only, in a range from 1 second to 24 hours. A separate startup_timeout, added in version 1.1.4, measures container startup time. If you do not set it, timeout covers both periods at once, which with a large model loaded inside @modal.enter() is a common cause of mysterious failures right after deployment. The cpu parameter counts physical cores, where one core equals two vCPUs, and the per-container minimum is 0.125 of a core. The memory parameter is given in mebibytes.

You call from your local machine with embed.remote(...), the async version is embed.remote.aio(...), embed.local(...) runs the code in place without sending it to the cloud, and embed.spawn(...) returns a FunctionCall you collect later.

Images, volumes and GPU classes

A plain function is rarely enough for inference, because model weights have to be loaded once rather than on every request. That is what @app.cls() together with the lifecycle methods is for.

import modal

app = modal.App("llm-serving")
weights = modal.Volume.from_name("model-weights", create_if_missing=True)

@app.cls(
    gpu="H100:2",
    volumes={"/weights": weights},
    min_containers=0,
    max_containers=20,
    buffer_containers=2,
    scaledown_window=300,
    enable_memory_snapshot=True,
)
@modal.concurrent(max_inputs=32, target_inputs=24)
class Generator:
    model_name: str = modal.parameter(default="Qwen3-8B")

    @modal.enter(snap=True)
    def load(self):
        self.model = load_from_disk(f"/weights/{self.model_name}")

    @modal.method()
    def generate(self, prompt: str) -> str:
        return self.model(prompt)

    @modal.fastapi_endpoint(method="POST", docs=True, requires_proxy_auth=True)
    def web(self, prompt: str) -> str:
        return self.generate.local(prompt)

    @modal.exit()
    def shutdown(self):
        self.model = None

min_containers keeps a given number of containers permanently warm, buffer_containers maintains a reserve above the current load, and scaledown_window says in seconds how long an idle container waits before being shut down. The same four fields are accepted by the Function.update_autoscaler method, which you can call at runtime, for instance ahead of an expected traffic peak. Settings applied through that method disappear on the next deployment of the app and the decorator values come back.

The @modal.concurrent decorator lets a single container handle several inputs at once: max_inputs is the hard ceiling, target_inputs is the threshold above which the autoscaler adds containers. For GPU-bound work you usually want the two values close together, because exceeding the card's memory capacity ends in an error rather than a queue.

Volumes are persistent file storage shared across containers. Volume.from_name accepts create_if_missing, environment_name, version and create_options. Loading weights from a volume instead of fetching them over the network on every start is the simplest way to shorten warm-up, because Modal's documentation states that for models in the tens of gigabytes this cuts boot from minutes to seconds.

Batch processing goes through the map family of methods. The Function.map signature accepts kwargs, order_outputs, return_exceptions and wrap_returned_exceptions.

@app.function(cpu=1.0, memory=2048, max_containers=500)
def transcode(path: str) -> str:
    return path.replace(".wav", ".flac")

@app.local_entrypoint()
def main():
    paths = [f"/data/{i}.wav" for i in range(100_000)]
    ok, failed = 0, 0
    for result in transcode.map(
        paths,
        order_outputs=False,
        return_exceptions=True,
    ):
        if isinstance(result, Exception):
            failed += 1
        else:
            ok += 1
    print(ok, failed)

Setting order_outputs=False returns results in completion order rather than input order and genuinely shortens the total run when the job has a long tail. return_exceptions=True pushes exceptions into the result stream instead of aborting everything, which at a hundred thousand inputs is the only sensible mode. The related methods are starmap, for_each and spawn_map.

Pricing worked through with real numbers

Modal bills three resources separately and all of them per second. The rates below come from the pricing page checked on 22 August 2026. Note how they are presented: the page source carries per-second prices only, and the toggle to the hourly view converts them in the browser, so I calculated the hourly figures below myself by multiplying by 3600.

# per-second rates from modal.com/pricing, as of 2026-08-22
GPU_PER_SEC = {
    "B300": 0.001972,           # 7.0992 USD/h
    "B200": 0.001736,           # 6.2496 USD/h
    "H200 SXM": 0.001261,       # 4.5396 USD/h
    "H100 SXM5": 0.001097,      # 3.9492 USD/h
    "RTX PRO 6000": 0.000842,   # 3.0312 USD/h
    "A100 80GB": 0.000694,      # 2.4984 USD/h
    "A100 40GB": 0.000583,      # 2.0988 USD/h
    "L40S": 0.000542,           # 1.9512 USD/h
    "A10": 0.000306,            # 1.1016 USD/h
    "L4": 0.000222,             # 0.7992 USD/h
    "T4": 0.000164,             # 0.5904 USD/h
}
CPU_PER_CORE_SEC = 0.0000131    # physical core, minimum 0.125 of a core
MEM_PER_GIB_SEC = 0.00000222
VOLUME_PER_GIB_MONTH = 0.09     # first 1 TiB per month at no charge

def cost(seconds, gpu=None, cores=0.125, gib=1.0):
    g = GPU_PER_SEC[gpu] * seconds if gpu else 0.0
    return g + cores * CPU_PER_CORE_SEC * seconds + gib * MEM_PER_GIB_SEC * seconds

# one hour of H100 in a container with 4 cores and 16 GiB of memory
print(cost(3600, "H100 SXM5", cores=4, gib=16))  # 4.265712

First conclusion: the card price is not the bill. An hour of H100 SXM5 is 3.9492 USD for the accelerator alone, but a container with four physical cores and 16 GiB of memory adds 0.18864 USD for the processor and 0.127872 USD for memory, giving 4.2657 USD in total. That is roughly 8 percent on top of the rate shown in the table.

The second conclusion concerns short functions. A thousand calls to a function running for 200 milliseconds, on one core with 1 GiB of memory, is 200 seconds of processor and memory time, which comes to 0.003064 USD. Three tenths of a cent. For workloads without a GPU the bill for execution time is practically irrelevant and the whole cost decision moves elsewhere, which I get to in a moment.

The third conclusion concerns multipliers. Region selection costs 1.5 to 1.75 times the base rate, and non-preemptible execution, meaning no risk of being evicted, costs 3 times the base rate. An hour of H100 with an eviction guarantee is therefore 11.8476 USD rather than 3.9492 USD. Sandboxes and Notebooks have their own price list: 0.00003942 USD per core per second and 0.00000667 USD per GiB per second, which is exactly triple the standard rates, at the same GPU price.

The subscription plans look like this.

The Starter plan allows up to 3 workspace seats, 200 deployed apps and 5 scheduled jobs. The Team plan removes the seat limit, raises the app count to 1000, and adds custom domains, a static outbound IP, deployment rollbacks across 3 versions and environment-level budgets. The arithmetic of the Team plan matters here: 250 USD of subscription minus 100 USD of credit leaves 150 USD of fixed cost per month before you run a single computation. That is the equivalent of roughly 38 hours of H100.

The 30 USD credit on the Starter plan works out to about 7.6 hours of H100, or about 37.5 hours of L4, or about 50 hours of T4 per month. Once the credit runs out you pay for usage from the card, with no hard cut-off, so a loop that accidentally starts a hundred A100 containers costs real money. The safeguard is workspace-level and environment-level budgets, and that is the first thing to configure after creating an account.

Volumes cost 0.09 USD per GiB per month with the first tebibyte included. A 200 GiB weights cache is therefore free, while 2 TiB costs 92.16 USD per month.

The comparison Modal puts on its pricing page is also worth reading. It sets 75 GPUs for 24 hours at 3 USD per hour, giving 5400 USD, against an average of 50 GPUs for 24 hours at 3.95 USD per hour, giving 4740 USD. Both totals are correct to the cent, but the comparison assumes Modal's rate is about 32 percent higher than the traditional cloud rate, and the saving comes entirely from the assumed drop in average utilisation from 75 cards to 50. Under an even load Modal comes out more expensive, and the provider does not hide this, it simply does not say it out loud.

Cold start as a line item on the bill

With per-second billing, container warm-up stops being an inconvenience and becomes a number on the invoice. Modal's documentation states that the container itself boots in about a second, but before it is considered ready it has to execute the code in the module's global scope and every method marked @modal.enter(). That stage takes, in the provider's own words, from seconds to minutes.

Let us do the sums. Take a function on H100 that runs for 2 seconds, in a container with 4 cores and 16 GiB, and a thousand calls. Execution time alone is 2000 seconds, which is 2.194 USD for the GPU, 0.1048 USD for the processor and 0.07104 USD for memory, 2.36984 USD in total. Now add a realistic 2 percent of calls hitting a cold container that loads weights for 40 seconds. That is 20 cold starts of 40 seconds each, meaning 800 seconds of extra container time with a GPU attached: 0.8776 USD for the card, 0.04192 USD for the processor and 0.028416 USD for memory. The bill rises from 2.37 USD to 3.32 USD, that is by 40 percent, for the same number of requests served.

That calculation assumes warm-up time lands on the invoice, and the provider confirms it. In the questions section of the pricing page, under "What counts as billable time?", it states plainly that Modal bills for application load time and for the time the container spends processing inputs. There is one further item that is easy to forget when budgeting: by default a container stays alive for 60 seconds after the last input and that time is billed too, although the length of that window is configurable. On an H100 card, sixty idle seconds cost about 6.6 cents per shutdown, so a workload spread across rare, isolated requests pays mostly for waiting. Only once you have scaled down to zero containers do the charges stop. The same pricing page promises in its headline that you "never pay for idle resources", so two statements in one document say different things, and for budgeting stick with the one from the questions section, because that is the one describing the mechanism. You can check your own consumption in the modal billing reports.

Modal gives you three tools for shortening warm-up. Weights kept in a modal.Volume rather than fetched over the network. Memory snapshots via enable_memory_snapshot=True together with @modal.enter(snap=True), which save the process state after initialisation and restore it on the next start. And min_containers plus buffer_containers, which simply means paying for readiness.

That third option has a price you can calculate, and it often settles the matter. A single H100 container kept alive around the clock costs 94.78 USD a day, roughly 2843 USD a month, before it serves its first request. If your traffic is even enough that you have to keep the card warm non-stop anyway, then Modal's billing model works against you and it makes more sense to rent a machine by the hour and put vLLM on it.

Modal against the alternatives

The split is fairly sharp. If your workload is spiky, needs a GPU and does nothing for most of the day, Modal wins, because scaling to zero is the default here rather than an add-on. If the load is even, your own machine with vLLM comes out cheaper. If you do not need a GPU at all, Modal is a needlessly exotic choice and a plain container on Railway or Fly.io will be simpler to maintain and easier to move.

One more approach is missing from that table. Replicate also runs models on someone else's cards, but instead of writing code with decorators you pick a ready model from a catalogue, and uploading your own goes through the Cog tool, which builds an ordinary container image. The difference that matters most to the invoice is another one: some public models bill not per second of card time but per unit of output, per image or per thousand tokens, so the same task can produce quite different amounts on the two services. Two drawbacks on that side: the Python client library has had no stable release since May 2025, and the billing documentation contradicts itself within one paragraph about whether a failed run is chargeable.

Common mistakes

The first mistake is confusing timeout with startup_timeout. The default 300 seconds covers both periods at once if you do not set the second one explicitly, so a model that takes 6 minutes to load will abort before it serves its first request, and it will do so in a way that looks random.

The second is importing heavy libraries in the global scope of the file you run locally. The client executes that file on your machine to build the app graph, so import torch at the top of the file slows down every modal run and forces a local install of dependencies you do not need locally. Heavy imports belong inside functions or in an image.imports() block.

The third is min_containers set "just in case". Every warm GPU container costs the full rate for its whole lifetime. Before you put a 1 there, calculate the monthly cost of readiness and compare it against the cost of the latency you want to remove.

The fourth is skipping the cpu and memory parameters on GPU jobs. The default minimum is 0.125 of a physical core, which for a pipeline that decodes images or tokenises text before it reaches the card means a processor bottleneck and a card sitting idle at the full rate.

The fifth is using Sandboxes where a plain function would do. Processor and memory rates there are three times higher, and a Sandbox makes sense when you are genuinely running code you do not control.

The sixth is having no budget. The account requires a card, there is no cut-off once the credit runs out, and a bug in a map loop can spin up hundreds of containers in a matter of seconds. Workspace-level and environment-level budgets are set once and there is no reason to postpone it.

The seventh is keeping domain logic in the same file as the decorators. That is not a technical error but a debt you will pay on the day Modal raises prices or changes its terms.

FAQ

Can Modal be run on your own hardware?

No. Only the client, meaning the modal package from PyPI, is released under Apache 2.0. The server, the scheduler, the image build system and the GPU infrastructure are closed. There is no community edition and no offline mode for testing.

What does an hour of H100 actually cost?

The accelerator alone is 3.9492 USD per hour, derived from the rate of 0.001097 USD per second. On top of that come the container's processor and memory, so a typical configuration with 4 cores and 16 GiB gives 4.2657 USD per hour. Non-preemptible execution multiplies that by three, and region selection by 1.5 to 1.75.

What exactly does the free plan include?

The Starter plan costs 0 USD in subscription and includes 30 USD of compute credit per month, up to 3 workspace seats, 100 containers, GPU concurrency of 10, 200 deployed apps, 5 scheduled jobs and 1 day of log retention. A payment card is required despite the zero subscription.

Is cold start really a problem?

It depends on the model. A container boots in about a second, but global-scope code and @modal.enter() methods can take minutes with large weights. With per-second billing that shows up on the bill, not only in the latency. Weights on a volume and memory snapshots shorten that time the most.

How hard is it to move an application off Modal?

The computation moves across unchanged, because it is plain Python. What has to be rewritten is the entire runtime layer: decorators into a Dockerfile and a manifest, volumes into the target storage, secrets, autoscaling and endpoints. Keeping that layer in a separate thin file limits the cost to one file per application.

Is Modal suitable for an ordinary web application?

Rarely. The @modal.fastapi_endpoint and @modal.asgi_app decorators work correctly, but the billing model and scaling from zero are designed for computation rather than for a long-lived process serving traffic around the clock. For that a container on a regular hosting platform will be simpler.