Run AI Language Models on Your VPS with Ollama

Ollama is a tool that makes it simple for system administrators and developers to download and run open-source Large Language Models (LLMs) on their own servers. Unlike cloud-based APIs from OpenAI or Anthropic where you pay per token, Ollama lets you host models locally—whether on a personal machine or a VPS—giving you complete control and no API billing.

What Is Ollama and Why Use It?

Ollama is an open-source project that simplifies downloading and running LLMs on your own hardware—laptops, desktops, or servers. It handles model downloads, memory management, and model loading/execution automatically through an intuitive command-line interface.

Developers and operations teams typically self-host LLMs for three main reasons:

• Privacy: All prompts and data stay on your VPS. Nothing is sent to external API providers, protecting sensitive information and confidential workflows.
• Cost Savings: No per-token billing. If you use LLMs heavily, third-party APIs can become prohibitively expensive. Self-hosting costs only your server resources.
• Offline Operation: Your LLM runs entirely locally. Even without internet connectivity to external services, the model continues to work.

Hardware Reality: What Your VPS Can Actually Handle

Large Language Models are resource hogs. They demand massive amounts of RAM and CPU (or GPU) compute. It is critical to understand this before expecting smooth performance.

Different model sizes have different requirements:

Model Size	Quantized Size (Q4)	Minimum RAM	Speed (CPU)	Recommendation
7B parameters	~4–5 GB	8 GB	Slow (5–10 tokens/sec)	Usable with patience
13B parameters	~8–9 GB	16 GB	Very slow (1–2 tokens/sec)	Not recommended for CPU
70B parameters	40+ GB	64+ GB	Unusable without GPU	Requires dedicated GPU machine

For a CPU-only VPS, we recommend using 7B-parameter models (like Llama 3.1 7B, Mistral 7B, or Phi-3) quantized at Q4 level. These use about 4–5 GB of disk space and 8–16 GB of total RAM—achievable on most moderately-sized VPS plans.

Prerequisites

• Server: Ubuntu 22.04 LTS or newer with root access
• RAM: Minimum 8 GB (16 GB recommended)
• Storage: At least 20 GB free (a 7B model is ~5 GB, but you may want multiple models)
• Bandwidth: Good internet speed to download models (a 7B model is roughly 4–5 GB)
• Knowledge: Basic Linux command-line and systemd familiarity

Installing Ollama on Ubuntu

Ollama provides an official installation script. Follow these steps:

1. SSH into your VPS:

ssh root@your-vps-ip

2. Download and run the official install script:

curl -fsSL https://ollama.ai/install.sh | sh

This script installs the Ollama binary, required dependencies, and a systemd service automatically.

3. Verify Ollama is running:

systemctl status ollama

Look for "active (running)". If you see that, installation succeeded. Ollama binds to localhost:11434 by default.

4. Test connectivity:

ollama list

This shows installed models (will be empty on a fresh install).

Downloading and Running Your First Model

Let's download Llama 3 7B (stable and well-regarded):

1. Pull the model:

ollama pull llama3

This downloads the Llama 3 8B model (~4.7 GB) from Ollama's repository. Be patient—do not interrupt the download.

2. Run the model interactively:

ollama run llama3

When you see the > prompt, try typing a question:

> What is machine learning?

The model will generate a response, which may take a moment on CPU. That's normal.

3. Exit the chat:

> /bye

Or press Ctrl+D.

Other popular models to try:

• ollama pull mistral — Mistral 7B (faster than Llama, still good quality)
• ollama pull phi — Phi-3 (3.8B, from Microsoft, lowest RAM usage)
• ollama pull neural-chat — Neural Chat 7B (optimized for conversation)

Using the REST API

Ollama exposes a REST API on localhost:11434, letting you send requests programmatically instead of using the interactive shell.

Here are basic curl examples:

1. Simple generation request:

curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "prompt": "Explain quantum computing in simple terms",
    "stream": false
  }'

Setting "stream": false tells the API to wait for the full response before returning it.

2. Using streaming (progressive output):

curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "prompt": "What is the capital of Thailand?",
    "stream": true
  }'

With "stream": true, the API sends back the response word-by-word as the model generates it, like ChatGPT's streaming interface.

The response is JSON:

{
  "model": "llama3",
  "created_at": "2024-01-15T10:30:00Z",
  "response": "The capital of Thailand is Bangkok...",
  "done": true
}

Building a Simple Chat Script

Here's a minimal Python script to interact with Ollama:

#!/usr/bin/env python3

import requests
import json

def chat_with_ollama(prompt, model="llama3"):
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": model,
        "prompt": prompt,
        "stream": False
    }
    response = requests.post(url, json=payload)
    data = response.json()
    return data.get("response", "No response")

if __name__ == "__main__":
    user_input = input("You: ")
    response = chat_with_ollama(user_input)
    print(f"Ollama: {response}")

Usage:

python3 chat.py
You: How do I configure a firewall on Linux?
Ollama: To configure a firewall on Linux...

Security Considerations

⚠️ Critical: By default, Ollama listens only on localhost:11434, meaning only processes on the same server can reach it. This is secure and appropriate for local use.

Never expose port 11434 to the public internet without additional security layers. Doing so opens your server to:

• Resource theft: Anyone can use your CPU/GPU to run their own prompts, costing you in electricity and compute cycles.
• Data exposure: Prompts and responses travel over unencrypted HTTP (without an HTTPS reverse proxy), risking interception.
• No authentication: Ollama has no built-in user authentication or rate limiting.

If you need to expose Ollama to other servers, use an HTTPS reverse proxy (like Nginx) with authentication:

server {
    listen 443 ssl http2;
    server_name ollama.example.com;
    
    ssl_certificate /etc/ssl/certs/your-cert.crt;
    ssl_certificate_key /etc/ssl/private/your-key.key;
    
    auth_basic "Ollama API";
    auth_basic_user_file /etc/nginx/.ollama_auth;
    
    location / {
        proxy_pass http://localhost:11434;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_buffering off;
    }
}

This approach: 1) closes port 11434 from the internet, 2) opens port 443 (HTTPS) with an SSL certificate, and 3) adds HTTP Basic Auth so only authorized users can access the API.

Running Ollama as a Persistent Service

The installation script already created a systemd unit for Ollama. Here's how to manage it:

1. View the systemd unit file:

cat /etc/systemd/system/ollama.service

You should see something like:

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
User=ollama
Group=ollama
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target

2. Enable Ollama to start automatically at boot:

systemctl enable ollama

3. Restart the Ollama service:

systemctl restart ollama

4. Monitor logs in real-time:

journalctl -u ollama -f

The -f flag streams logs as they appear.

Summary

Ollama dramatically simplifies running open-source LLMs on your own infrastructure. You don't need expensive GPUs (though they help) or reliance on third-party APIs.

Key takeaways:

• Use 7B–8B-parameter quantized models (Q4) on typical CPU VPS.
• Provision at least 8–16 GB of RAM.
• Secure the API with a reverse proxy if exposing to the internet.
• Install via the official script and use systemd for continuous operation.
• Start with smaller models (Phi-3, Mistral) and scale up as needed.

Frequently Asked Questions

Should I use Ollama or LM Studio or something else?

Ollama comes with a built-in REST API server, making automation straightforward, and has a large model library. LM Studio is better for GUI-based desktop use. For a VPS, Ollama is the superior choice.

Which model is fastest on CPU?

Phi-3 (3.8B) is fastest, followed by Mistral 7B. Llama 3 7B is slightly slower but still usable. For 70B models, you absolutely need a GPU.

Can I run multiple models at once?

You can, but Ollama loads models into RAM sequentially. Running multiple large models simultaneously will exhaust RAM quickly. A model switching controller is a better approach for production.

What VPS specs do I need?

A minimum of 16 GB RAM and 4–8 vCPUs is ideal, costing roughly $10–30/month. While 8 GB might work, 16 GB offers a good balance of affordability and performance.