vLLM, a local model for more than one person

vLLM, a local model for more than one person

Tools for running models on your own computer work well for a single person. The problem appears when a hundred users must share one model: throughput drops, requests queue, and the graphics card sits idle between one and the next.

vLLM solves exactly that. It is an inference server built for handling many concurrent requests, and the difference from single user solutions is measured in multiples rather than percentages.

The two ideas it rests on

Understanding where that difference comes from also explains the limitations.

The first idea is continuous batching. The classic approach collects requests into a batch, processes them together, and waits for all to finish before accepting more. Since answers vary in length, the card waits for the longest one while the remaining slots sit empty.

Continuous batching adds a new request the moment one finishes, without waiting for the rest. The card then works constantly, and waiting time drops, since nobody queues behind a request that happens to be generating a long answer.

The second idea is paged attention. The context cache, growing with every generated token, is allocated in blocks rather than as a contiguous region, exactly as virtual memory works in an operating system. The classic approach reserves space for the maximum answer length up front and therefore wastes most of it, and that limitation decides how many requests fit at once.

Both point the same way: more concurrent conversations on the same card and less time spent waiting.

Starting up

pip install vllm

vllm serve mistralai/Mistral-Small-Instruct-2501 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90

The server exposes an interface compatible with a popular format, so code written against a cloud vendor works after changing the address.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-Instruct-2501",
    messages=[{"role": "user", "content": "Classify this ticket: ..."}],
)

That compatibility matters practically during migration. An application running on a cloud model moves to your own hardware with no logic changes, letting you compare both variants on the same data.

Two parameters in that command govern behaviour and deserve understanding from the start. The first caps context length, and lowering it frees memory for more concurrent requests. The second states what share of card memory the server may occupy.

Sizing memory

This is the commonest cause of a failed first start, and it deserves calculating before launch rather than after an out of memory message.

Card memory splits three ways. Model weights take the first part, sized by parameter count and precision. The context cache takes the second, growing with concurrent request count and context length. Server overhead takes the third.

A seven billion parameter model at eight bit quantisation therefore occupies around seven gigabytes, and the rest of card memory goes to context. On a twenty four gigabyte card that leaves a dozen or so gigabytes, translating into dozens of concurrent conversations at moderate context length.

The practical rule: lowering maximum context length is the cheapest way to raise concurrent request count. Setting one hundred twenty eight thousand tokens when your queries run to four thousand wastes memory nobody will use.

Quantisation and trade offs

Lowering weight precision is the simplest way to fit a larger model on the same card, and the server supports several formats.

The eight bit variant is a safe starting point. Quality loss is practically unmeasurable on typical tasks, and the memory gain is twofold against full precision.

The four bit variant halves memory again at a noticeable but usually acceptable cost. Measure that on your own cases, since differences depend on the task: slight on classification, larger on work requiring precise reasoning.

The third mechanism quantises the context cache itself. On long conversations it occupies more than the weights, so lowering its precision raises concurrent request count more than changing weight format does.

The practical order runs like this. Start with the eight bit variant and measure throughput. If it falls short, lower the maximum context length, since that costs nothing in quality. Only then reach for lower precision and check the result against your own case set.

Production deployment

A server started from a terminal command suits testing. A deployment serving traffic needs several things more.

The first is a container. An image with the server and the model pulled in advance removes the longest part of startup, namely downloading weights on every start. On a model weighing a dozen or more gigabytes that is the difference between a minute and a quarter of an hour.

The second is authentication. The server requires nothing by default, but it ships with an --api-key option and a matching VLLM_API_KEY environment variable. Set either and it accepts only requests carrying one of those keys in a header. The mechanism is a simple one: no roles, no per key limits, no usage accounting, so on a deployment serving several consumers the endpoint still belongs behind a reverse proxy or on a private network unreachable from outside.

The third is a queue in front of the server. On a traffic spike it is better for requests to wait than for the server to refuse them, since a refused request is an error visible to a user.

The fourth is a plan for card failure. One instance is one point of failure, so where something depends on the deployment, keep a second one or a fallback to a vendor interface. A routing layer, even a simple route in Hono, lets you switch that without touching the application.

What to measure

Three numbers suffice to know whether a deployment behaves, and they deserve collecting from day one.

Time to first token speaks to how it feels to a user. With streaming, that is what decides whether an interface feels fast, whatever the full answer's length.

Tokens per second across the whole server speaks to throughput. That value rises with concurrent request count until the card saturates, and that rise is precisely this solution's advantage.

Card memory occupancy tells you how close the wall is. A server that starts refusing requests usually does so without warning, so set an alert threshold early.

Test under load resembling reality. Measuring one request at a time shows this solution's worst result, since its entire advantage lies in handling many at once.

Prefix caching

A mechanism that in some applications delivers more than every other optimisation combined, and which gets skipped because it works only in a particular arrangement.

If many requests begin with the same fragment, a long system instruction or the same documentation for instance, the server can compute that fragment once and reuse the result for later requests.

The gain is proportional to the shared part's share. A two thousand token system instruction with two hundred token queries means ninety percent of the work on every request is repetition, and caching removes it.

There is one condition, the same as with caching at cloud vendors: the shared part must come first and match byte for byte. A date inserted into the system instruction or a session identifier at the start invalidates everything, and nothing reports it.

Current releases need no switch for this, since prefix caching is on by default. The work therefore sits in how the request is arranged rather than in server configuration, which is good news, because reordering fragments of a prompt costs less than any configuration change.

In an application answering questions about the same documentation, order the prompt so the documentation precedes the question. That is a reordering of two fragments, and throughput can rise several times over.

Matching the model to the task

The largest saving in a self hosted deployment lies not in server configuration but in choosing a model suited to the task.

Ticket classification, tagging, and extracting data into a structure are tasks where a three billion parameter model performs little worse than a twenty billion one, fits a card several times smaller, and serves many times more requests at once.

The reverse holds on tasks requiring reasoning or code work, where the size difference registers and a smaller model simply falls short.

The practical approach splits traffic. One instance with a smaller model handles bulk tasks, another with a larger one handles the hard ones, and a routing layer decides where a request goes. At volume that split delivers more than any single instance optimisation.

Measure it on your own cases before buying hardware. Thirty real queries run through two models tell you whether the smaller suffices, and that translates directly into the hardware required and the bill.

vLLM against the alternatives

Choosing between the first two rows comes down to concurrent user count. With one person the simpler tool suffices and saves configuration. With ten at once the throughput difference registers, and at a hundred it decides.

Price the third row honestly. A graphics card costs the same on a quiet day as at peak, so at a few hundred calls a day the vendor interface usually comes out cheaper. The break even for your own hardware typically sits around a few thousand calls a day, depending on query length and on whether you already own the hardware.

When it is the wrong choice

For single user work the configuration overhead does not repay itself. A model on a laptop run with a simpler tool gives the same result with no deployment work.

At low and uneven volume your own hardware sits idle most of the day while the bill stays the same. A vendor interface billed per token then follows the traffic.

Where the highest answer quality is required, open models runnable on one card still trail the closed leaders on hard reasoning. That argues for a mixed arrangement: bulk tasks locally, hard ones through a vendor interface.

Common mistakes

The first is leaving the default context length. A model declaring one hundred twenty eight thousand tokens reserves memory for it, cutting concurrent request count several times over.

The second is sizing memory by active parameter count on a mixture of experts architecture. Everything enters memory, so generation speed does not reduce the requirement.

The third is measuring performance at one request. That is the worst case for this solution and the result says nothing about behaviour under real traffic.

The fourth is no cap on answer length. One request generating endlessly occupies a batch slot and lowers throughput for everybody.

The fifth is exposing the server without authentication. An endpoint reachable from the network without a key is a graphics card working for somebody else on your electricity bill.

The sixth is not monitoring memory occupancy. A server refusing requests does so without warning, and the cause gets hunted in the application rather than in a saturated card.

FAQ

How does vLLM differ from Ollama?

Ollama is built for single user work and ease of starting; vLLM for handling many requests at once. With one user the difference is slight; with dozens concurrently vLLM's throughput is many times higher.

What hardware is needed?

A card with memory holding the model weights plus headroom for context. A seven billion parameter model at eight bit quantisation fits a twenty four gigabyte card with room for dozens of concurrent conversations. Larger models need a server class card or several cards together.

Is the interface compatible with the popular format?

Yes, so an application written against a cloud vendor moves by changing the address and model name. That lets you compare both variants on the same data without rewriting logic.

When does your own hardware come out cheaper?

At high and steady volume. A card costs the same regardless of traffic, so the break even typically sits around a few thousand calls a day. Below that, per token billing wins.

Can several models run at once?

On one card usually not, since memory is shared and weights occupy it permanently. Practical answers are separate instances on separate cards or one general model handling every task, and the second is often cheaper and sufficient.

Documentation sits on the project site, and the performance mechanisms are covered in a throughput guide.