74. Ollama

The lesson

Ollama is a free, open-source tool that makes it easy to run large language models on your own computer. It handles the complex steps of downloading model weights, configuring the runtime, and exposing a chat interface or API, so you can focus on using the model instead of managing infrastructure.

The main reason people use Ollama is local execution. When a model runs on your machine, your prompts and documents never leave your network. This is important for privacy, intellectual-property protection, and working offline. There are also no per-token API fees, although you do need capable hardware.

Installing Ollama gives you a command-line program. To start chatting, you type a command such as `ollama run llama3` or `ollama run qwen2`. Ollama downloads the model if you do not already have it, then opens an interactive chat session where you can type prompts and receive answers.

Ollama maintains a large model library. You can find general chat models such as Llama 3, Mistral, and Qwen; coding models such as CodeLlama and DeepSeek-Coder; and smaller models such as Phi 3 and Gemma that fit on laptops with limited memory. The library is constantly growing as the open-source community releases new models.

You can customize models with a Modelfile, which is like a recipe. In a Modelfile you set the base model, the system prompt, parameters such as temperature and context length, and any adapter files. This lets you build specialized agents or enforce a consistent personality across conversations.

Ollama exposes a local REST API and an OpenAI-compatible endpoint. That means tools such as LangChain, OpenWebUI, Continue, and many IDE extensions can talk to Ollama as if it were a cloud provider. This makes it easy to integrate local models into your existing workflow.

Hardware matters. Running a 70-billion-parameter model at full precision requires a powerful GPU and a lot of memory. Most users run quantized versions, which use less memory and run faster at the cost of a small quality drop. Ollama handles quantization automatically when you pull a model.

Best practices: keep Ollama and your models updated, choose a model size that fits your hardware, test outputs before using them in production, and remember that local models can still produce incorrect or biased answers.

Check yourself