The Midnight Gauntlet: Deploying nyxcore to Production on Hetzner ARM64

The clock had ticked past 2 AM UTC on March 2nd, 2026. My mission: take nyxcore, a project I'd been nurturing, from local development to a live, production-ready service on a fresh Hetzner ARM64 instance. The goal was clear: nyxcore.cloud needed to be up, running, secure, and populated with its initial set of Greek-named personas.

What followed was a classic late-night deployment session – a blend of methodical setup, unexpected roadblocks, head-scratching debugging, and ultimately, the sweet taste of success. This isn't just a list of tasks; it's the story of getting nyxcore production-ready, complete with the "gotchas" and the "aha!" moments.

The Foundation: Setting Up Shop on Hetzner

Starting with a clean Ubuntu 24.04 ARM64 instance on Hetzner (46.225.232.35), the first steps were foundational:

1. Docker & Compose: Installing Docker CE 29.2.1 and Compose 5.1.0 was straightforward. This is the bedrock for our containerized application.
2. Code Access: A dedicated SSH deploy key was set up for secure access to the private nyxcore repository.
3. The Stack: My docker-compose.production.yml spun up the core services:
   • app: The Next.js application.
   • postgres: PostgreSQL 16 with the pgvector extension (critical for future AI features).
   • redis: For caching and session management.
   • nginx: As the reverse proxy, handling SSL termination and routing.

A minor but critical fix emerged early: the Dockerfile for the Next.js app needed apk add --no-cache openssl. Without it, the Prisma engine would silently fail, leading to cryptic database connection errors later on. Trust me, you don't want to debug that one in the dark.

The Gauntlet: Lessons Learned in the Trenches

No deployment is ever truly smooth. The real learning happens when things don't work as expected. Here’s a dive into the "pain log" and the valuable lessons it yielded:

1. The Elusive Prisma CLI: Seeding the Database

The Problem: After getting the postgres container up and running with pgvector enabled, it was time to push the schema and seed the database. My initial thought was to docker exec into the running app container and run prisma db push and prisma db seed.

The Attempt:

docker exec -it nyxcore_app_1 npx prisma db push --accept-data-loss

The Failure: npx: command not found or prisma: command not found. The standalone Next.js build, optimized for production, strips node_modules and thus the Prisma CLI. It's lean, but sometimes too lean for administrative tasks.

The Workaround & Lesson Learned: The solution was to create a temporary, dedicated container for database operations. This container would mount the application's source code and connect to the existing Docker network.

# Assuming your app's root is at /opt/nyxcore
docker run --rm \
  -v /opt/nyxcore:/app \
  --network nyxcore_nyxcore-net \
  -w /app \
  node:20-alpine \
  sh -c "npm install && npx prisma db push --accept-data-loss && npx prisma db seed"

Lesson: For tasks requiring development dependencies or CLIs not part of your optimized production build, create specialized temporary containers. They offer the necessary environment without bloating your main application containers.

2. The --ignore-scripts Trap: Faster Seeds, Broken Builds

The Problem: To speed up the temporary seed container setup, I tried using npm ci --ignore-scripts.

The Attempt:

npm ci --ignore-scripts && npx prisma db push ...

The Failure: The prisma db seed script, which relies on tsx and esbuild, crashed. esbuild couldn't find its platform-specific binary.

The Workaround & Lesson Learned: Removing --ignore-scripts fixed it. esbuild needs its post-install scripts to fetch the correct binary for the node:20-alpine environment.

Lesson: Be extremely cautious with npm ci --ignore-scripts. While it can speed up installs, many packages (especially build tools and native modules like esbuild or sharp) rely on post-install scripts to function correctly in their target environment. If in doubt, run the full install.

3. The SSH Connection Drops: MaxStartups Strikes Again

The Problem: During rapid iteration and debugging, I found my SSH connections to the Hetzner server dropping after just 2-3 attempts. I had already removed fail2ban, so that wasn't the culprit.

The Failure: The server would silently refuse new connections for a period, leading to frustration and lost time.

The Workaround & Lesson Learned: This is a classic sshd configuration issue, specifically MaxStartups. While I plan to configure it properly later, the immediate workaround was to slow down:

• Add sleep 30 between SSH sessions for intensive work.
• Use nohup for commands that might take a long time, so they continue running even if the SSH session drops.
  bash

  nohup ./long-running-script.sh &
  

Lesson: When facing intermittent SSH issues, especially after multiple rapid connections, check your sshd_config for MaxStartups. Also, nohup and screen/tmux are your best friends for robust remote execution.

4. Docker Compose Environment Variable Interpolation Woes

The Problem: I wanted to define my DATABASE_URL in .env.production but also use individual components like POSTGRES_USER within the environment: block of my docker-compose.production.yml.

The Attempt:

# docker-compose.production.yml
services:
  postgres:
    environment:
      - POSTGRES_USER=${POSTGRES_USER} # This was the problem
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      # ...

And in .env.production:

POSTGRES_USER=nyxcore
POSTGRES_PASSWORD=supersecret
DATABASE_URL="postgresql://nyxcore:supersecret@postgres:5432/nyxcore"

The Failure: The POSTGRES_USER variable inside the environment: block was empty. Docker Compose was reading the .env.production file after it tried to interpolate variables within the environment section itself.

The Workaround & Lesson Learned: The simplest solution was to remove the compose-level override for the DATABASE_URL components and let the DATABASE_URL defined directly in .env.production be the source of truth for the application. For the postgres service itself, it uses its own internal variables like POSTGRES_USER which are typically set directly or from a separate env_file for the database service.

Lesson: Understand Docker Compose's variable processing order. Environment variables defined in an env_file are usually made available to services, but interpolation within the docker-compose.yml file itself happens before those env_file variables are fully available to the Compose parser for its own configuration. For sensitive or complex URLs, defining the full DATABASE_URL in the .env file and passing it directly to the app container is often cleaner.

5. Nginx Dynamic Upstream Resolution for Docker Services

The Problem: Nginx needed to proxy requests to the app service within the Docker network. My initial thought was to use a simple upstream block.

The Attempt:

# In nginx.conf
upstream app_server {
    server app:3000; # 'app' is the Docker service name
}

server {
    listen 443 ssl;
    server_name nyxcore.cloud;
    # ...
    location / {
        proxy_pass http://app_server;
    }
}

The Failure: host not found in upstream "app:3000". Nginx, by default, resolves upstream hostnames at startup. Since the Docker DNS resolver (and the app service itself) might not be fully ready or known to Nginx at its startup time, it couldn't resolve app.

The Workaround & Lesson Learned: The solution involves dynamic DNS resolution within Nginx using Docker's internal DNS resolver (127.0.0.11).

# In nginx.conf
# This tells Nginx to use Docker's internal DNS resolver
resolver 127.0.0.11 valid=10s;

server {
    listen 443 ssl;
    server_name nyxcore.cloud;
    # ...
    location / {
        # Define a variable for the upstream host
        set $upstream http://app:3000;
        # Use the variable in proxy_pass
        proxy_pass $upstream;
        # ... other proxy headers
    }
}

Lesson: When Nginx needs to proxy to services within a dynamic environment like Docker, always use dynamic upstream resolution with resolver and a set $variable; proxy_pass $variable; pattern. This ensures Nginx re-resolves the hostname at runtime, adapting to changes in the Docker network.

The Triumphs: nyxcore is Live!

Despite the hurdles, the mission was accomplished. By the time the session wrapped, nyxcore was fully deployed and running at https://nyxcore.cloud.

Here's a recap of the successful outcomes:

• Full Docker Stack: App, Postgres (with pgvector), Redis, and Nginx are all humming along.
• HTTPS Enabled: Let's Encrypt provides a secure connection to nyxcore.cloud. (Expires 2026-05-31 – time to set up auto-renewal!)
• Database Seeded: 12 Greek-named personas (Athena, Nemesis, Harmonia, Clotho, Hermes, Tyche, Themis, Prometheus, Aletheia, Cael, and two others) are ready for action.
• Clean Environment: wstunnel-server (which was blocking port 443) and fail2ban (which was overzealously rate-limiting SSH) were disabled, ensuring smooth operation.
• Production-Ready Config: Environment variables, SSH deploy keys, and certificate paths are all configured correctly.

The entire journey from 3bdba26 to 53a1271 in the commit history represents the iterative process of getting this production environment just right.

What's Next for nyxcore.cloud?

The deployment is a huge milestone, but the work isn't over. Immediate next steps include:

1. SSH Hardening: Configure sshd MaxStartups to prevent future connection drops.
2. Certbot Automation: Set up a cron job for Let's Encrypt certificate auto-renewal.
3. CI/CD Pipeline: Implement GitHub Actions for automated deployments.
4. Security Policies: Apply Row-Level Security (RLS) policies on the production PostgreSQL database.
5. Smoke Testing: Conduct thorough E2E smoke tests against the live site.
6. Feature Development: Continue with the planned dual-provider implementation.

This late-night deployment was a stark reminder that even with mature tools like Docker and Nginx, the devil is often in the details. Each "pain" point was a learning opportunity, strengthening my understanding of the underlying systems. Here's to many more successful (and hopefully less late-night) deployments!

{"thingsDone":[
    "Deployed full Docker stack (app, postgres, redis, nginx) to Hetzner ARM64.",
    "Installed Docker CE 29.2.1 + Compose 5.1.0 on Ubuntu 24.04.",
    "Set up SSH deploy key for private repository access.",
    "Fixed Dockerfile to include OpenSSL for Prisma engine.",
    "Configured nginx for reverse proxy with dynamic Docker DNS resolution.",
    "Disabled conflicting `wstunnel-server` and `fail2ban`.",
    "Installed Let's Encrypt certificate for `nyxcore.cloud`.",
    "Enabled `pgvector` extension on production PostgreSQL.",
    "Performed Prisma schema push and database seeding with 12 Greek-named personas.",
    "Ensured production environment variables and security keys are in place."
],"pains":[
    "Prisma CLI missing in standalone Next.js container for `db push`.",
    "`npm ci --ignore-scripts` causing `esbuild` platform binary issues during seed.",
    "Rapid SSH connections dropping due to `sshd MaxStartups` (even after `fail2ban` removal).",
    "Docker Compose environment variable interpolation order issues with `env_file` and `environment:` block.",
    "Nginx `upstream` block failing to resolve Docker service names at startup."
],"successes":[
    "Used a temporary `node:20-alpine` container with volume mount for Prisma CLI operations.",
    "Used full `npm ci` (without `--ignore-scripts`) for reliable dependency installation in seed container.",
    "Implemented `sleep` between SSH sessions and `nohup` for long-running commands.",
    "Simplified Docker Compose environment configuration by relying on direct `DATABASE_URL` from `.env.production`.",
    "Configured Nginx with `resolver 127.0.0.11` and dynamic `set $upstream; proxy_pass $upstream;` for runtime Docker DNS resolution."
],"techStack":[
    "Next.js",
    "Docker",
    "Docker Compose",
    "Nginx",
    "PostgreSQL 16",
    "pgvector",
    "Redis",
    "Prisma",
    "TypeScript",
    "Hetzner Cloud",
    "Ubuntu 24.04 ARM64",
    "Let's Encrypt"
]}