Install Headscale VPN Server on VPS: Self-Hosted Tailscale Control

What is Headscale?

Headscale is an open-source implementation of the Tailscale control server that you can run on your own VPS instead of relying on Tailscale Inc.'s SaaS platform. Tailscale is built on WireGuard, a secure and high-performance VPN protocol, and uses a mesh network architecture that allows devices to connect directly to each other rather than through a central gateway. However, device discovery and coordination traditionally relies on Tailscale's control server in the cloud. Headscale lets you run this control server yourself, offering significant benefits: enhanced privacy (your data stays on your server), unlimited devices (no 3-device free-tier limit), full ACL control, and compliance with data residency requirements if your organization demands all data remain within your country.

Why Use Headscale Instead of Standard Tailscale

Several reasons drive organizations and privacy-conscious users toward Headscale. First, Tailscale's free tier caps you at 3 devices; Headscale has no such limit. Second, all connection metadata and IP addresses remain on your server—nothing leaves to Tailscale Inc.—crucial for privacy and compliance regimes like GDPR. Third, you gain complete control over ACL policies and can configure MagicDNS, server names, and behavior exactly as you need. Fourth, if your organization mandates that all data remain on domestic servers, Headscale enables compliance without monthly SaaS bills. Fifth, you can audit the source code (Headscale is open-source on GitHub) and customize behavior far beyond what Tailscale's closed-source control server allows.

Headscale vs. Tailscale vs. Plain WireGuard

Understanding the distinctions matters. WireGuard is a low-level VPN protocol: fast, secure, but requires manual key generation, interface configuration, and route management. Tailscale wraps WireGuard in a convenient application layer, adding automatic peer discovery, MagicDNS (resolve peers by hostname instead of IP), intelligent NAT traversal, and a web UI—but the control server is closed-source and hosted by Tailscale Inc. Headscale mimics the Tailscale control server on your own infrastructure, so you get all Tailscale's convenience (official client apps, MagicDNS, mesh routing) while the server remains under your control. Think of it as "Tailscale's features with your own infrastructure."

Prerequisites and Planning

Before starting, ensure you have: (1) a VPS running Ubuntu 22.04 LTS (or similar Debian-based distribution) with root access; (2) a public IP address (not behind CGNAT—Carrier Grade NAT—since Headscale needs a stable public IP for peer discovery); (3) a DNS domain name pointing to your VPS IP (e.g., headscale.example.com) required for HTTPS certificates; (4) Docker and Docker Compose installed, or the ability to install Headscale natively; and (5) basic comfort with Linux commands, Nginx config, and firewall rules. If you don't have a DNS A record set up yet, configure it now—point your domain to your VPS's public IP before proceeding further.

Install Headscale: Docker Compose (Easiest)

Docker Compose offers the fastest setup. Verify Docker is installed:

docker --version && docker-compose --version

Create a directory for Headscale:

mkdir -p /opt/headscale/data /opt/headscale/config
cd /opt/headscale

Create docker-compose.yml:

version: '3.8'
services:
  headscale:
    image: headscale/headscale:latest
    container_name: headscale
    hostname: headscale
    ports:
      - "8080:8080"
      - "51820:51820/udp"
    volumes:
      - ./data:/var/lib/headscale
      - ./config.yaml:/etc/headscale/config.yaml
    environment:
      - TZ=Asia/Bangkok
    restart: unless-stopped
    networks:
      - headscale

networks:
  headscale:
    driver: bridge

Download the Headscale example config:

docker run --rm headscale/headscale:latest headscale generate config > /opt/headscale/config.yaml

Start the container:

cd /opt/headscale
docker-compose up -d
docker-compose logs -f

Verify status:

docker exec headscale headscale nodes list

Install Headscale: Native Binary (Advanced)

For more control or to avoid Docker, install the Headscale binary directly. Download the latest release (check GitHub releases for version):

cd /tmp
HEADSCALE_VERSION=0.22.0
wget https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64 -O headscale
chmod +x headscale
sudo mv headscale /usr/local/bin/

Create system user and directories:

sudo useradd -r -s /bin/false headscale
sudo mkdir -p /etc/headscale /var/lib/headscale
sudo chown -R headscale:headscale /var/lib/headscale /etc/headscale

Create systemd service file:

sudo tee /etc/systemd/system/headscale.service > /dev/null <<'EOF'
[Unit]
Description=Headscale VPN Control Server
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=headscale
Group=headscale
WorkingDirectory=/var/lib/headscale
ExecStart=/usr/local/bin/headscale serve -config /etc/headscale/config.yaml
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable headscale

Configuring config.yaml

The generated config.yaml contains many settings. Here are the critical ones to modify. Edit the file:

server:
  listen_addr: 0.0.0.0
  listen_port: 8080
  
server_url: https://headscale.example.com

db:
  type: sqlite3
  sqlite3:
    path: /var/lib/headscale/db.sqlite3

derp:
  server:
    enabled: true
    region_id: 900
    region_code: "headscale"
    region_name: "Headscale DERP"
    stun_listen_addr: "0.0.0.0:3478"

dns:
  base_domain: example.com
  magic_dns: true
  domains:
    - example.com
    - headscale.example.com

policy:
  mode: file
  path: /etc/headscale/acl.yaml

log:
  level: info

Change example.com and headscale.example.com to your actual domain. The server_url is critical—Tailscale clients use this to locate your control server. Never use HTTP; always HTTPS. SQLite is configured by default (sufficient for most deployments). DERP (Detour Encrypted Relay Protocol) provides relay servers when direct peer-to-peer connection fails. MagicDNS allows devices to discover each other by hostname instead of IP.

Setting Up HTTPS with Nginx and Let's Encrypt

Tailscale clients enforce HTTPS on the server_url—unencrypted connections are rejected. Use Nginx as a reverse proxy in front of Headscale and Certbot to provision Let's Encrypt certificates. Install Nginx:

sudo apt update && sudo apt install -y nginx certbot python3-certbot-nginx curl

Create Nginx config for Headscale:

sudo tee /etc/nginx/sites-available/headscale > /dev/null <<'EOF'
upstream headscale_backend {
    server localhost:8080;
}

server {
    listen 80;
    listen [::]:80;
    server_name headscale.example.com;

    location / {
        proxy_pass http://headscale_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        proxy_request_buffering off;
    }
}
EOF

Enable the site:

sudo ln -s /etc/nginx/sites-available/headscale /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl restart nginx

Run Certbot to generate HTTPS certificates:

sudo certbot --nginx -d headscale.example.com

Certbot's interactive prompts will ask whether to redirect HTTP to HTTPS; choose "2" to let Certbot handle everything. It will generate a certificate and update your Nginx config automatically.

Setting Up Namespaces, Users, and Pre-Auth Keys

Once Headscale is running, create a namespace (think of it as a "workspace" or "team") to organize devices. For Docker:

docker exec headscale headscale namespaces create mycompany
docker exec headscale headscale users create admin

For native binary:

sudo -u headscale /usr/local/bin/headscale namespaces create mycompany
sudo -u headscale /usr/local/bin/headscale users create admin

Create a pre-auth key to allow devices to join:

docker exec headscale headscale preauthkeys create --namespace mycompany --reusable --expiration 24h

This outputs a string—save it for use on client devices. The --reusable flag allows the key to be used multiple times; --expiration sets when it expires.

Connecting Tailscale Clients

On any client device (macOS, Linux, Windows, iOS, Android), install the official Tailscale client from tailscale.com. Then, from a terminal or command prompt, run:

tailscale up --login-server=https://headscale.example.com

The --login-server flag directs the client to your Headscale server instead of the official Tailscale SaaS. The client will display a URL and ask you to paste the pre-auth key. After authentication, verify that nodes appear:

docker exec headscale headscale nodes list

Your devices should now appear with Tailscale IPs (100.x.x.x range). They can ping and SSH each other by name (if MagicDNS is enabled) or by IP address.

Managing ACLs (Access Control Lists)

The ACL file at /etc/headscale/acl.yaml (or mounted in Docker) defines which devices can communicate with which. Here's a basic example:

groups:
  "group:admin":
    - admin
  "group:developers":
    - dev1
    - dev2
    - dev3

acls:
  - action: accept
    src:
      - group:admin
    dst:
      - "*:*"
  
  - action: accept
    src:
      - group:developers
    dst:
      - "10.0.0.0/24:22"
      - "10.0.1.0/24:443"
  
  - action: deny
    src:
      - "*"
    dst:
      - "*:*"

This allows the admin group access to all devices on all ports; developers can SSH (port 22) to 10.0.0.0/24 and use HTTPS (port 443) to 10.0.1.0/24; everything else is denied. After editing the ACL file, reload Headscale:

docker exec headscale headscale serve --config /etc/headscale/config.yaml

Or for the native binary:

sudo systemctl restart headscale

Security and Maintenance

Once Headscale is operational, follow these security practices. First, firewall your VPS to allow only port 443 (HTTPS) and UDP 51820 (WireGuard) from the internet; restrict port 8080 (Headscale internal) to localhost. Second, if you generate API keys for integrations, store them in a secrets manager, not in scripts. Third, update Headscale regularly. For Docker: `docker pull headscale/headscale:latest && docker-compose up -d`. For the binary, download the latest release from GitHub. Fourth, back up the SQLite database regularly (/var/lib/headscale/db.sqlite3) via cron or your VPS backup service. Fifth, monitor logs for anomalies: for Docker, `docker logs headscale`; for the binary, `journalctl -u headscale -n 50`. Sixth, store pre-auth keys with expiration dates to limit their window of validity; rotate them periodically.

Summary

You now have a fully operational Headscale VPN server with privacy by default, unrestricted devices, and granular ACL control. By securing it with HTTPS, Nginx reverse proxy, and Let's Encrypt, your clients connect safely. Headscale strikes an excellent balance between WireGuard's security and Tailscale's convenience. AsiaGB's VPS infrastructure—with fast CPUs and SSD storage—is ideal for running a stable, responsive Headscale server. Connect your devices, configure ACLs to your security policy, and enjoy a private, self-managed mesh VPN network.

Frequently Asked Questions

Does Headscale require PostgreSQL or MySQL, or is SQLite sufficient?

SQLite is sufficient for small-to-medium deployments (hundreds to thousands of devices). Headscale supports SQLite, PostgreSQL, and MySQL via the config file. For very large deployments, PostgreSQL or MySQL may offer better performance, but migration is straightforward.

I lost my pre-auth key. Can I create a new one?

Yes. Generate a fresh pre-auth key with: `headscale preauthkeys create --namespace mycompany --reusable --expiration 24h`. Use this new key on the device instead.

Does Headscale support two-factor authentication (2FA)?

Headscale is simpler than official Tailscale and does not include built-in 2FA. However, you can restrict API keys and pre-auth keys by setting expiration dates. For enhanced security, use OS-level or VPS firewall rules to limit who can access the Headscale control server port.

If I shut down my Headscale control server, will clients stop working?

Tailscale devices will continue routing through cached peer routes, so existing connections remain functional. However, they cannot discover new devices, accept new ACL changes, or adjust settings. The control server orchestrates—it does not relay traffic—so downtime affects configuration, not active data transfer.