Immutable Infrastructure & Cloud-Init

1. Concept Overview & Systemic Problem

Traditional manual administration via SSH access inevitably leads to chaos:

• An engineer logs into a server on Friday at 10 PM, urgently edits one line in the Nginx configuration, forgets to document it, and goes home.
• Six months later, the server needs to be migrated to another host or scaled.
• A new server is deployed using the old guide — and nothing works! No one remembers which packages were installed manually and what permissions were granted in /etc.

Immutable Infrastructure permanently prohibits the mutation of live servers: if any configuration needs to change, you do not modify the old server — you deploy a new ready instance, switch the traffic, and destroy the old one.

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│                 IMMUTABLE PROVISIONING CYCLE                │
├─────────────────────────────────────────────────────────────┤
│ 1. INFRASTRUCTURE AS CODE (Git Repository):                 │
│    • `cloud-init.yaml`: Security hardening, UFW, Docker      │
│    • `docker-compose.prod.yml`: Application stack           │
├─────────────────────────────────────────────────────────────┤
│                          │                                  │
│                          ▼ Trigger: Provision New VPS (API) │
├─────────────────────────────────────────────────────────────┤
│ 2. CLOUD-INIT BOOTSTRAP (< 90 seconds on Hetzner):          │
│    • Disable root login & password authentication           │
│    • Provision non-root user `deployer` with SSH keys       │
│    • Lock down UFW (allow ONLY ports 80, 443, WireGuard)    │
│    • Install Docker Engine & Tailscale daemon               │
├─────────────────────────────────────────────────────────────┤
│ 3. TRAFFIC SWAP & ZERO DOWNTIME CUTOVER:                    │
│    • Healthcheck PASS ➔ Point DNS / Floating IP to NEW VPS  │
│    • Terminate OLD VPS ➔ Zero leftovers, zero drift!        │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

01. Reference Cloud-Init Script for Ordering a Secure VPS

Passed in the User Data field when creating the server:

#cloud-config
users:
  - name: orlov
    groups: sudo
    shell: /bin/bash
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...
package_update: true
packages:
  - ufw
  - fail2ban
  - docker.io
runcmd:
  - ufw default deny incoming
  - ufw default allow outgoing
  - ufw allow 22/tcp
  - ufw allow 80/tcp
  - ufw allow 443/tcp
  - ufw --force enable
  - systemctl enable --now docker

4. Production Engineering Scenarios

01. Managing Configuration Drift

When an application stores user-uploaded files directly on the server's local disk, those files will be lost during instance replacement. All user state must be stored in dedicated storage (S3, Cloudflare R2) or databases.

02. Initialization Script Errors

If there is a syntax error in the SSH key within the cloud-init.yaml file, the server will boot without access. Always test initialization configurations on inexpensive test machines.

03. Local State Preservation Attempts

Attempting to preserve local state on an immutable server can lead to data loss. Ensure that all stateful data is managed externally to maintain consistency across deployments.

5. Pitfalls, Common Mistakes & Security

Immutable infrastructure liberates engineers from the fear of server failures. When any node in your system can be destroyed and rebuilt in under a minute with a single keystroke, infrastructure becomes predictable, easily scalable, and resilient to human error.