Ollama, language models running locally

Ollama, language models running locally

Have you ever wanted to chat with AI but couldn't send your data to the cloud? Or wanted to experiment with different models without paying for tokens? Or maybe you just wanted full control over what happens with your data?

Ollama solves these problems in the simplest way possible. One command and you have a powerful language model running locally on your computer - no cloud, no subscription, no token limits.

What is Ollama?

Ollama is a free, open-source framework for downloading, running, and managing large language models (LLMs) on your local computer. It works like Docker for AI models - you type one command, and Ollama takes care of the rest: downloads the model, configures quantization, manages memory, and starts a local API server.

Under the hood, Ollama is built on the llama.cpp library, providing an abstraction layer that hides low-level details and makes working with local models as simple as ollama run llama3.2.

The project has over 178 thousand stars on GitHub and an MIT license, and the current release is numbered 0.32.6, dated 4 August 2026. Patch releases land here roughly weekly, so the third number in the version moves faster than anyone can write it down. The program itself is written in Go.

How much hardware this really needs

This is the question that decides whether local models make sense for you, and most write-ups skip it or answer it vaguely.

The working rule is roughly half a gigabyte of memory per billion parameters at four bit quantisation. A 7B model wants about 4 GB, a 13B model about 8 GB, a 70B model about 40 GB. Add a little on top for the context window, which grows with the length of the conversation.

The key distinction is where that memory sits. On a machine with a discrete graphics card the model loads into video memory and runs fast. Without one it lands in system memory and runs on the processor, which is several times slower but still usable for shorter answers.

Apple Silicon occupies a middle position that works out well here, because memory is shared between processor and graphics unit, so a laptop with 32 GB handles models that would need an expensive card on a desktop.

The practical advice: start with a model in the 8B range. It fits nearly everywhere, answers quickly enough to be worth using, and lets you find out whether the whole idea suits your work before you spend money on hardware.

Quantisation, or what you trade away

The term comes up in every model description and is worth understanding, because it directly governs what you can run.

At full precision a model takes about 2 GB per billion parameters. Quantisation reduces the precision of the weights, and four bit quantisation cuts that to roughly a quarter. That is why a model which would need 28 GB fits in 8 and runs on ordinary hardware.

Quality does drop, but not linearly. Between full precision and eight bit the difference is barely noticeable. Between eight and four it is visible in harder tasks and negligible in simple ones. Below four bits it becomes clear: the model starts making basic mistakes and losing the thread of longer instructions.

The rule most people arrive at through experience: a larger model quantised more aggressively usually beats a smaller model at higher precision, as long as you stay at four bits or above.

Why Ollama?

Data privacy

This is the main reason people reach for Ollama. Your data never leaves your computer. No prompts, no responses, no files are sent to any cloud. For companies working with sensitive data, source code, or internal documentation - it's a game changer.

Zero operational costs

After downloading a model, you don't pay for tokens, API calls, or subscriptions. The model runs on your hardware, and the only cost is electricity. If you use AI extensively at work, the savings can be significant.

Simplicity

Ollama does one thing and does it well. There's no complicated configuration, no dashboards, no setup wizards. Installation is one command, running a model is one command, integrating with applications is one URL setting.

Offline work

Once downloaded, a model works without internet. It's perfect on airplanes, in places without WiFi, or in environments with restricted network access.

Comparison with alternatives

Installation

macOS

brew install ollama

Or download the native installer from ollama.com.

Linux

curl -fsSL https://ollama.com/install.sh | sh

Windows

Download the installer from ollama.com or use WSL2:

curl -fsSL https://ollama.com/install.sh | sh

Docker

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

With GPU support (NVIDIA):

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Getting started

Running a model

After installation, just one command:

ollama run llama3.2

Ollama will automatically download the model (if not already downloaded) and start an interactive chat session. You can start typing and getting responses right away.

Basic commands

ollama run llama3.2         # Run a model and start chatting
ollama pull gemma3           # Download a model without running it
ollama list                  # Show installed models
ollama ps                    # Show currently running models
ollama rm llama3.2           # Remove a model
ollama show llama3.2         # Show model details
ollama serve                 # Start the API server (without GUI)

Working with a model

In interactive mode, you can simply chat with the model:

>>> Explain what TypeScript is in 3 sentences

TypeScript is a superset of JavaScript that adds static typing to the language.
It allows catching errors at compile time, before the code reaches the browser
or Node.js environment. It's developed by Microsoft and has become the standard
in large frontend and backend projects.

Multimodal mode

Models with vision support (e.g., Llama 3.2 Vision) allow image analysis:

ollama run llama3.2-vision
>>> What's in this image? /path/to/screenshot.png

Model library

Ollama offers over 230 models in its library at ollama.com/library. Here are the most important ones:

General-purpose models

Coding models

Vision models

Embedding models

How to choose a model?

• 8 GB RAM: Llama 3.2 3B, Gemma 3 1B, Phi-3 3.8B
• 16 GB RAM: Llama 3.1 8B, Gemma 3 4B, Mistral 7B
• 32 GB RAM: Gemma 3 12B-27B, DeepSeek-R1 14B
• 64+ GB RAM: Llama 3.1 70B, DeepSeek-R1 32B+

General rule: start with a smaller model (3B-8B) and scale up based on your needs and available hardware.

The tables above list proven, widely documented models rather than everything new. The library keeps up with releases, so it also carries later generations of the same families, llama4, gemma4, and qwen3 among them. Check the library page before choosing, since names and available sizes shift every few months.

REST API

Ollama runs a local API server on port 11434, compatible with the OpenAI format. This means many existing tools and libraries can integrate with it without code changes.

Generating responses

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain what React is",
  "stream": false
}'

Chat API

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "system", "content": "You are a helpful programming assistant."},
    {"role": "user", "content": "How do I create a hook in React?"}
  ],
  "stream": false
}'

Generating embeddings

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "Ollama is a framework for local LLMs"
}'

OpenAI-compatible endpoint

Ollama offers an endpoint compatible with the OpenAI API:

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "user", "content": "Hello!"}
  ]
}'

This means you just need to change the base_url in existing code:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "What is TypeScript?"}]
)
print(response.choices[0].message.content)

JSON mode

Forcing responses in JSON format:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "List 3 JavaScript frameworks as a JSON array",
  "format": "json",
  "stream": false
}'

Client libraries

Python

pip install ollama

import ollama

response = ollama.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "What is Rust?"}]
)
print(response["message"]["content"])

Streaming:

import ollama

stream = ollama.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Write a poem about programming"}],
    stream=True
)
for chunk in stream:
    print(chunk["message"]["content"], end="", flush=True)

JavaScript/TypeScript

npm install ollama

import { Ollama } from "ollama"

const ollama = new Ollama({ host: "http://localhost:11434" })

const response = await ollama.chat({
  model: "llama3.2",
  messages: [{ role: "user", content: "What is Next.js?" }],
})
console.log(response.message.content)

Modelfile - Creating custom models

A Modelfile is a configuration file (similar to a Dockerfile) that lets you create custom model variants. You can change the system prompt, generation parameters, and even combine a base model with your own weights.

Basic Modelfile

FROM llama3.2

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

SYSTEM """
You are an experienced TypeScript and React developer.
You respond concisely and provide practical code examples.
You always use the latest patterns and best practices.
"""

Creating and running

ollama create typescript-helper -f Modelfile
ollama run typescript-helper

Advanced Modelfile

FROM llama3.2

PARAMETER temperature 0.2
PARAMETER top_p 0.85
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 8192

TEMPLATE """
{{ if .System }}<|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>
{{ .Response }}<|eot_id|>
"""

SYSTEM """
You are a code review assistant. Analyze code for bugs, security issues,
and performance problems. Always suggest specific improvements with code examples.
"""

Importing GGUF models

If you have a model in GGUF format (e.g., from Hugging Face), you can import it:

FROM ./my-model-q4_K_M.gguf

PARAMETER temperature 0.8
SYSTEM "You are a helpful assistant."

ollama create my-custom-model -f Modelfile

GPU acceleration

Ollama automatically detects and uses available GPUs:

NVIDIA (CUDA)

Requirements: compute capability 5.0+ and driver version 550 or newer. Cards with compute capability 5.0 through 6.2 need driver 570 or newer.

ollama run llama3.2

Ollama automatically uses CUDA if available. For multiple GPUs:

CUDA_VISIBLE_DEVICES=0,1 ollama serve

Apple Silicon (Metal)

On Macs with M1/M2/M3/M4 chips, Ollama automatically uses Metal. Apple Silicon is particularly well supported - unified memory means even larger models run smoothly.

AMD (ROCm)

Supported AMD cards with ROCm. Flash attention is automatically enabled.

GPU performance

Approximate generation speed (tokens/s) for Llama 3.1 8B:

Docker and production deployment

Basic Docker Compose

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama-data:

With Open WebUI

A popular combination - Ollama as backend + Open WebUI as a graphical interface:

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui-data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama-data:
  open-webui-data:

Preloading models in Docker

FROM ollama/ollama

RUN ollama serve & sleep 5 && ollama pull llama3.2 && ollama pull nomic-embed-text

Configuration

Environment variables

Sharing on local network

By default, Ollama only listens on localhost. To share on the network:

OLLAMA_HOST=0.0.0.0:11434 ollama serve

Changing the model directory

Models can take up a lot of space. To move them to another drive:

OLLAMA_MODELS=/mnt/ssd/ollama/models ollama serve

Integrations

The Ollama ecosystem includes 50+ integrations:

Graphical interfaces

• Open WebUI - the most popular web interface, looks like ChatGPT
• LibreChat - multi-provider chat with multiple model support
• Chatbox - cross-platform desktop application
• Enchanted - native macOS app
• Lobe Chat - modern interface with plugins

AI frameworks

• LangChain - building RAG pipelines, agents, chains
• LlamaIndex - document indexing and semantic search
• CrewAI - orchestration of AI agent teams
• AutoGen - multi-agent conversations, the project went into maintenance mode and split into successors

Developer tools

• Continue - coding assistant in VS Code and JetBrains
• Aider - AI pair programming in the terminal
• Cody - Sourcegraph AI coding assistant
• GitHub Copilot (self-hosted backend) - via OpenAI-compatible API

RAG and search

• RAGFlow - RAG engine that reads document layout: tables, columns and scans
• AnythingLLM - all-in-one AI with RAG
• PrivateGPT - private Q&A over documents

Practical use cases

Coding assistant

ollama create code-assistant -f - <<EOF
FROM codellama:13b
PARAMETER temperature 0.3
SYSTEM "You are a senior software engineer. Write clean, well-tested code. Always include error handling."
EOF

ollama run code-assistant
>>> Write a TypeScript function that fetches data with retry logic

Source code analysis

cat src/components/App.tsx | ollama run llama3.2 "Review this React component for potential issues"

Documentation generation

cat src/utils/helpers.ts | ollama run llama3.2 "Generate JSDoc documentation for these functions"

Local translations

ollama run llama3.2 "Translate to Polish: The quick brown fox jumps over the lazy dog"

RAG with your own documents

Combining Ollama with LlamaIndex for searching your own documents:

from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

llm = Ollama(model="llama3.2", request_timeout=120)
embed = OllamaEmbedding(model_name="nomic-embed-text")

documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents, embed_model=embed)
query_engine = index.as_query_engine(llm=llm)

response = query_engine.query("How do I configure authentication?")
print(response)

Hardware requirements

Minimum

• RAM: 8 GB (1B-3B models)
• Disk: 5-10 GB per model
• CPU: Any modern x86_64 or ARM64 processor
• GPU: Optional (significantly speeds things up)

Recommended

• RAM: 16-32 GB (7B-13B models)
• Disk: SSD with at least 50 GB free space
• GPU: NVIDIA RTX 3060+ (12GB VRAM) or Apple Silicon M1+
• CPU: 8+ cores

For large models (70B+)

• RAM: 64-128 GB
• GPU: NVIDIA RTX 4090 (24GB) or 2x RTX 3090
• Disk: NVMe SSD

Tips and tricks

Quick pipe commands

echo "Explain Docker in one paragraph" | ollama run llama3.2

cat error.log | ollama run llama3.2 "What's causing these errors?"

git diff | ollama run llama3.2 "Summarize these code changes"

Keeping the model in memory

OLLAMA_KEEP_ALIVE=-1 ollama serve

A value of -1 keeps the model in memory indefinitely, eliminating loading time.

Parallel requests

OLLAMA_NUM_PARALLEL=4 ollama serve

Allows handling 4 requests simultaneously - useful when multiple users share one server.

Monitoring

ollama ps

Shows currently loaded models, memory usage, and last use time.

FAQ

Is Ollama free?

Yes, Ollama is completely free and open-source under the MIT license. There are no paid plans, limits, or hidden costs.

Which GPUs are supported?

NVIDIA with CUDA (compute capability 5.0+), Apple Silicon with Metal, AMD with ROCm, and Vulkan as a fallback.

Can I run a model without a GPU?

Yes, but it will be significantly slower. CPU inference works, especially for smaller models (1B-7B).

How much disk space do I need?

Depends on the model. Llama 3.2 3B is ~2 GB, Llama 3.1 8B is ~4.9 GB, Llama 3.1 70B is ~43 GB.

Can I use models from Hugging Face?

Yes, Ollama supports importing GGUF format models via Modelfile. Many Hugging Face models are also available directly in the Ollama library.

How do I share Ollama with others on the network?

Set OLLAMA_HOST=0.0.0.0:11434 and make sure the port is open in your firewall.

Does Ollama support function calling?

Yes, newer versions of Ollama support tool calling, allowing models to invoke external functions.

How does it differ from ChatGPT/Claude?

ChatGPT and Claude are cloud services - your data goes to OpenAI/Anthropic servers. Ollama runs models locally - data never leaves your computer. Local models are typically smaller and less capable than top cloud models, but they provide complete privacy.

Can I run multiple models simultaneously?

Yes, and by default nothing needs setting: the server keeps up to three models per GPU loaded at once, and three when running on CPU. The OLLAMA_MAX_LOADED_MODELS variable raises or lowers that number. Each model uses separate RAM or VRAM, and if the next one does not fit, requests queue until an earlier model is unloaded.

The cloud layer, or the end of pure locality

One change worth noting, which older material does not account for and which complicates the simple picture of "Ollama means a model on your own computer".

The project now also offers running models in the cloud, billed by graphics hardware usage rather than per token. The point is practical: the same commands and the same API serve both a local and a remote model, so you can develop against a small model on your machine and reach for external resources when a task needs a bigger one, without changing code.

I deliberately give no specific rates here, because the summaries circulating online disagree on the amounts for individual plans. Check current values directly with the provider.

One thing is worth noticing, though, because it bears on whether this tool fits. If your reason for reaching for local models is keeping data on your own machine, then using a cloud model voids that reason. This is not a criticism of the project, since both options are stated plainly, but when rolling this out across an organisation it is worth deciding explicitly which of them the team may use, before someone reflexively fires a remote model at data that should never leave.

When a local model makes sense and when it does not

The honest answer is more measured than the enthusiastic narrative that surrounds this topic.

It makes sense wherever data cannot leave the machine or the organisation. No external service beats that argument, and it is the most common real reason for choosing this route.

It makes sense for repetitive high volume work of low difficulty: classification, tagging, pulling data out of text, generating embeddings. There the quality gap between a local model and the best commercial one is small, while the cost gap across hundreds of thousands of calls is enormous.

It makes sense for learning and prototyping, because the absence of a running meter changes how you work. You can experiment without weighing what a failed attempt cost.

It does not make sense where reasoning quality on hard tasks matters. Models that fit on typical hardware are a class below the best commercial ones such as Claude, and in code writing, long document analysis, or multi step tasks that difference is felt rather than cosmetic.

Nor does it make sense as a way to save money at small scale. Electricity, hardware, and the time spent on setup cost more at a few hundred calls a month than an API bill does. The saving starts at the volume where an external service invoice becomes visible in the budget.

Wiring it into existing code

The biggest practical advantage here is an API compatible with the interface known from commercial services. That means a client library written for another service works after changing the base address and the key, with no logic rewritten.

The same mechanism means orchestration tools, including LangChain and LlamaIndex, treat a local model like any other provider. When building retrieval augmented generation you can therefore keep the whole thing in house: the embedding model, the vector database, and the model generating the answer.

Two differences from commercial services are worth keeping in mind, and they only surface in code. The first is the absence of queueing on your behalf: the server handles a limited number of concurrent requests and starts queueing them under load, which shows up as growing response time rather than an error. With one user that is a detail; with a dozen it becomes a wall, and the right tool then is vLLM, which batches requests on the fly and holds throughput instead of lengthening the queue, at the cost of a harder setup and a card that fits the whole model. The second is the first call after a longer pause, where the model loads into memory and the answer arrives with a noticeable delay.

Source code and releases live in the Ollama repository, and the model catalogue sits on the project site.