How to Run an LLM Locally with Ollama (Windows, macOS, Linux)

To run a language model on your own computer, install Ollama and run ollama run gemma4: it downloads the model on first use and opens a chat prompt in your terminal. Pick a model tag whose download size fits comfortably in your GPU memory, and other programs can use the same model through the local API at http://localhost:11434.

Commands below are from Ollama's official README and docs as of October 2026. Model names change often; gemma4 is the example Ollama's own docs use.

Install Ollama

Windows

Download and run OllamaSetup.exe from ollama.com, or install from PowerShell:

irm https://ollama.com/install.ps1 | iex

Ollama's Windows docs list the requirements as Windows 10 22H2 or newer (Home or Pro), plus NVIDIA driver 551.61 or newer for NVIDIA cards, or a ROCm v7 or Vulkan-capable driver for AMD Radeon cards. The installer does not need administrator rights and installs into your home directory. Logs are in %LOCALAPPDATA%\Ollama (server.log for the server).

macOS

Download ollama.dmg and drag Ollama into Applications, or use the install script:

curl -fsSL https://ollama.com/install.sh | sh

The macOS docs require macOS Sonoma (14) or newer. Apple M-series Macs get CPU and GPU support; Intel Macs run on the CPU only. On first start the app offers to link the ollama command into /usr/local/bin if it is not already on your PATH.

Linux

curl -fsSL https://ollama.com/install.sh | sh

The Linux docs describe running Ollama as a systemd service under a dedicated ollama user. Check the service and its logs with:

sudo systemctl status ollama
journalctl -e -u ollama

To update on Linux, rerun the install script. On Windows and macOS updates download automatically and are applied from the tray or menu bar icon.

Check that it works

ollama -v

If it warns that it could not connect to a running Ollama, start the server with ollama serve in another terminal (or start the desktop app) and try again.

Run your first model

ollama run gemma4

The first run downloads the model, then shows a >>> prompt. Type a message and press Enter. For multi-line input, wrap the text in triple quotes:

>>> """Summarise this:
... Ollama runs language models locally.
... """

You can also pass a single prompt as an argument, get one answer and return to the shell:

ollama run gemma4 "Explain what a context window is in two sentences."

The CLI docs also show piping text into ollama run, for example echo "Hello world" | ollama run nomic-embed-text.

Add --verbose to ollama run to print timings after each response, which is a quick way to see how fast a model runs on your hardware.

Manage models: pull, list, ps, stop, rm

A model stays loaded for 5 minutes after its last use by default, then is unloaded to free memory. Pass --keepalive 30m to ollama run, or keep_alive in an API request, to change that.

Choose a model size for your RAM and VRAM

Most models come in several sizes, selected with a tag after a colon. As an example, Ollama's library page for gemma4 listed these local tags in October 2026:

A workable rule: the download size is roughly the memory the weights need, and you need extra on top for the context (KV cache) and runtime buffers. If the whole thing fits in VRAM, the model runs on the GPU. If not, Ollama puts part of it in system RAM and runs that part on the CPU, which is much slower. How much VRAM you need to run an LLM locally has the formula and worked examples, and what quantization and GGUF are explains why the same model comes in different file sizes.

After loading a model, check where it landed:

ollama ps

Read the PROCESSOR column. Ollama's FAQ says 100% GPU means the model is entirely in GPU memory, while something like 48%/52% CPU/GPU means part of it is in system RAM. If you see a split and speed matters, try a smaller tag or a shorter context.

Set the context length

Ollama chooses a default context length from your VRAM: 4k tokens below 24 GiB, 32k for 24 to 48 GiB, and 256k at 48 GiB or more. A longer context needs more memory. To change it, use the context length slider in the app's settings, or start the server with an environment variable:

OLLAMA_CONTEXT_LENGTH=32000 ollama serve

That is Linux and macOS shell syntax. In PowerShell, run $env:OLLAMA_CONTEXT_LENGTH = "32000" first, then ollama serve. Then check the CONTEXT column in ollama ps. Background on what this limit means: what a context window is.

Use the local API

While Ollama is running it serves an HTTP API at http://localhost:11434/api. Generate a completion:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Responses stream by default ("stream": true), one JSON object per chunk; "stream": false returns a single object. Example response (trimmed):

{
  "model": "gemma4",
  "response": "...",
  "done": true,
  "done_reason": "stop",
  "prompt_eval_count": 11,
  "eval_count": 18
}

prompt_eval_count and eval_count are the input and output token counts. For multi-turn chat, use /api/chat with a messages array:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": false
}'

Ollama also lists an OpenAI-compatible base URL, http://localhost:11434/v1, so many tools built for the OpenAI API can point at a local model by changing the base URL. For JSON output, the generate endpoint accepts "format": "json" or a JSON schema.

If something goes wrong

• Very slow responses: run ollama ps. A CPU/GPU split means the model or context does not fit in VRAM. Use a smaller tag or a shorter context.
• Cannot connect to localhost:11434: the server is not running. Start the app, or run ollama serve, or on Linux check sudo systemctl status ollama.
• GPU not used at all on Windows: check the driver versions above, then look in server.log under %LOCALAPPDATA%\Ollama.
• Disk filling up: ollama list shows what is downloaded; ollama rm <model> deletes what you no longer need.

Frequently asked questions

Does Ollama work offline?

Yes, once a model has been pulled it runs on your own machine, so prompts are processed locally. You need an internet connection to download Ollama and each model. The model library also lists some cloud-hosted tags, such as tags ending in -cloud, which run on Ollama's servers rather than locally.

Where does Ollama store downloaded models?

According to Ollama's FAQ: ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux, and C:\Users\%username%\.ollama\models on Windows. Models are several gigabytes each, so check free disk space before pulling large ones, and remove unused models with ollama rm.

Can other computers on my network use my Ollama server?

Not by default. Ollama binds to 127.0.0.1 port 11434, so only the local machine can connect. The FAQ explains changing the bind address with the OLLAMA_HOST environment variable. Only do that on a network you trust, because the API has no authentication of its own when run locally.

Why does a model forget the start of a long conversation?

Its context window is full. Ollama sets a default context length from your VRAM, 4k tokens below 24 GiB, so long chats and documents can exceed it. Raise it with the app's context length slider or by starting the server with OLLAMA_CONTEXT_LENGTH set, for example OLLAMA_CONTEXT_LENGTH=32000 ollama serve, if you have the memory for it.

Sources

• Ollama on GitHub: README
• Ollama docs: CLI reference
• Ollama docs: API reference
• Ollama docs: Generate a response (API)
• Ollama docs: Windows
• Ollama docs: macOS
• Ollama docs: Linux
• Ollama docs: FAQ
• Ollama docs: Context length
• Ollama source: cmd/cmd.go (command definitions)