Stop Using Self-Signed Certificates for Internal Services

Internal developer tools, staging environments, and admin dashboards have historically been the Wild West of transport layer security. For years, the standard operating procedure for securing an internal service, like a self-hosted Grafana instance or a staging API, was to generate a self-signed certificate, spin up a private Certificate Authority (CA), or simply bypass HTTPS altogether.

But this approach is no longer viable. Training developers and internal users to "click through" browser warnings destroys security culture. When users are conditioned to ignore certificate warnings on internal tools, they become highly vulnerable to real man-in-the-middle attacks on public networks. Furthermore, modern development workflows, mobile clients, HTTP/2, and WebSockets flatly reject untrusted certificates.

The right way to handle internal TLS today is to use publicly trusted certificates mapped to public domains, resolved via split-horizon DNS or overlay networks, and strictly bound to private interfaces.

The Failure of Legacy Internal TLS

Historically, teams chose one of three paths for internal HTTPS, each with severe drawbacks:

1. Self-Signed Certificates: This is the easiest to set up but the hardest to maintain. Every client machine, browser, and command-line tool (like curl or language runtimes) must be manually configured to trust the certificate. Forcing users to bypass browser warnings is a massive training failure.
2. Private CAs: Setting up an internal CA using tools like OpenSSL or Active Directory Certificate Services works well for managed corporate laptops. However, it fails completely for bring-your-own-device (BYOD) setups, mobile devices, and modern containerized environments where root certificates must be injected into every Docker image.
3. Non-Unique Domains: Many legacy setups used domains like .local or .internal. However, public CAs are prohibited from issuing certificates for non-unique internal names or private IP addresses. This restriction has been in place since 2015. Additionally, using .local can break multicast DNS (mDNS) routing on local networks.

If you cannot use a private CA easily and cannot get a public certificate for a .local domain, you must use a public fully qualified domain name (FQDN) that you own, even for services that live entirely behind a firewall or VPN.

The Modern Blueprint: Split-Horizon DNS and Public ACME

To get a valid, publicly trusted TLS certificate from a CA like Let's Encrypt, you must prove ownership of a public domain. However, this does not mean your service must be exposed to the public internet.

The solution relies on split-horizon DNS. For public DNS resolvers, your internal domain (e.g., grafana.company.dev) either does not exist or resolves to a public gateway. But for clients connected to your internal network or VPN, the domain resolves directly to a private IP address (such as 10.0.1.10).

Browsers do not care if a publicly trusted certificate resolves to a private IP address. As long as the domain name matches the certificate and the certificate is signed by a trusted root CA, the browser will establish a secure, warning-free HTTPS connection.

To automate this, you need an ACME client and a validation method. There are two primary ways to handle this:

The DNS-01 Challenge (The Cleanest Method)

The DNS-01 challenge is the gold standard for internal services. Instead of requiring the CA to reach your server over port 80, your ACME client (like acme.sh or Certbot) uses an API key to write a temporary TXT record to your public DNS provider. The CA verifies this record and issues the certificate. Your internal server never has to open any ports to the public internet.

Modern overlay networks have productized this pattern. For instance, Tailscale automatically provisions Let's Encrypt certificates for nodes on a private network by handling the DNS-01 challenge through its control plane.

The HTTP-01 Challenge with Split-Horizon DNS

If you cannot use DNS-01 (perhaps due to restricted DNS API access), you can use the HTTP-01 challenge. This requires a split-horizon setup where your public DNS temporarily points to the server's public IP during renewal, allowing the ACME server to reach port 80. Once the certificate is issued, internal clients continue to resolve the domain to the private IP.

The Developer Angle: Implementation and Hardening

Let's look at how to implement this pattern using Nginx as a reverse proxy, acme.sh as the ACME client, and a private network interface.

A common security failure when setting up internal reverse proxies is binding to 0.0.0.0:443. If your firewall rules are misconfigured or temporarily flushed during an update, your internal service is suddenly exposed to the public internet.

To prevent this, you must bind your reverse proxy strictly to the private network interface or VPN IP address. If you are using an overlay network like NetBird or Tailscale, bind directly to that interface's IP.

Here is a hardened Nginx configuration for an internal Grafana instance:

upstream grafana {
    server localhost:3000;
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    # Bind strictly to the private VPN IP address
    listen 10.0.1.10:443 ssl;
    server_name grafana.company.dev;
    http2 on;

    ssl_certificate /etc/ssl/certs/grafana.company.dev.crt;
    ssl_certificate_key /etc/ssl/private/grafana.company.dev.key;

    access_log /var/log/nginx/grafana.access.log main;
    error_log /var/log/nginx/grafana.error.log warn;

    location / {
        proxy_pass http://grafana;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # Proxy Live WebSocket connections safely
    location /api/live/ {
        proxy_pass http://grafana;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
    }
}

By binding to 10.0.1.10:443, Nginx will reject any traffic that does not originate from or pass through the private network interface. This acts as an application-level firewall, ensuring that even if public DNS points to this server, external clients cannot establish a connection.

Automating Renewals Safely

Because Let's Encrypt certificates expire every 90 days, automation is mandatory. If you are using acme.sh in standalone mode for HTTP-01 validation, you can run a cron job that temporarily spins up a web server on port 80 to complete the challenge, copies the certificates, and reloads Nginx.

Here is a shell script designed for a daily cron job:

#!/usr/bin/env bash
set -euo pipefail

SRC_KEY="/home/acmesh/.acme.sh/grafana.company.dev/grafana.company.dev.key"
SRC_CERT="/home/acmesh/.acme.sh/grafana.company.dev/fullchain.cer"
DST_KEY="/etc/ssl/private/grafana.company.dev.key"
DST_CERT="/etc/ssl/certs/grafana.company.dev.crt"

# Allow acme.sh to bind to port 80 without running as root
setcap CAP_NET_BIND_SERVICE=+ep /usr/bin/socat 2>/dev/null || true

# Run renewal check
sudo -u acmesh /home/acmesh/.acme.sh/acme.sh --cron --home /home/acmesh/.acme.sh

# If certificates changed, copy and reload
if ! cmp -s "$SRC_KEY" "$DST_KEY"; then
    cp "$SRC_KEY" "$DST_KEY"
    cp "$SRC_CERT" "$DST_CERT"
    chmod 600 "$DST_KEY"
    chown www-data:www-data "$DST_KEY" "$DST_CERT"
    
    # Test and reload Nginx
    nginx -t && systemctl reload nginx
fi

Weighing the Trade-offs

While this approach is highly secure, it introduces a few trade-offs that you must evaluate:

• Subdomain Leakage: Every certificate issued by a public CA is recorded in public Certificate Transparency (CT) logs. If you issue a certificate for super-secret-stage.company.dev, an attacker monitoring CT logs will learn that this subdomain exists. If host discovery is a concern, you should use a wildcard certificate (e.g., *.internal.company.dev) instead of individual host certificates.
• DNS API Security: If you use the DNS-01 challenge, the API credentials for your DNS provider must live on your internal servers or deployment runners. To minimize risk, use scoped API tokens restricted to editing only the specific _acme-challenge TXT records rather than global admin keys.
• VPN Dependency: If your VPN or internal DNS resolver goes down, clients will lose the ability to resolve the domain to the private IP, breaking access even if they are physically on the same local network.

The Verdict

The days of tolerating self-signed certificate warnings are gone. By using split-horizon DNS, binding reverse proxies strictly to private interfaces, and automating public CA challenges, you can achieve frictionless, production-grade HTTPS for all internal services. It is a minor upfront configuration investment that pays massive dividends in developer velocity and organizational security posture.