Certificate Management¶

Intermediate35 minPrereqs: tls-ssl-fundamentalssecurity tls ssl certificates

Learning outcomes

Obtain and renew certificates with Let's Encrypt and Certbot using ACME challenges
Build and manage an internal certificate authority for private infrastructure
Troubleshoot certificate expiry, chain issues, and common TLS configuration failures

Managing TLS certificates is one of the most operationally critical tasks in systems administration. An expired or misconfigured certificate takes down your site, breaks API integrations, and erodes user trust. This guide covers the full lifecycle: obtaining certificates, automating renewal, building internal CAs, and troubleshooting the most common failures.

The ACME Protocol¶

The Automatic Certificate Management Environment (ACME) protocol is how clients like Certbot communicate with Let's Encrypt to automate certificate issuance and renewal. Understanding the protocol helps you debug problems when automation fails.

The flow:

The client generates a key pair and registers an account with the CA.
The client requests a certificate for one or more domains.
The CA issues challenges to prove the client controls the domain(s).
The client completes the challenges and notifies the CA.
The CA verifies the challenges and issues the certificate.
The client downloads and installs the certificate.

Challenge Types¶

Challenge	Mechanism	Port Required	Use Case
HTTP-01	Place a file at `/.well-known/acme-challenge/`	80	Standard web servers
DNS-01	Add a TXT record to `_acme-challenge.domain`	None	Wildcards, internal servers, CDN-fronted sites
TLS-ALPN-01	Present a special self-signed cert on port 443	443	When port 80 is unavailable

HTTP-01 is the simplest - Certbot places a token file on your web server and Let's Encrypt fetches it. DNS-01 is required for wildcard certificates (*.example.com) because it proves control over the entire DNS zone, not just a single server.

Let's Encrypt with Certbot¶

Installation¶

# Ubuntu/Debian (snap is the recommended method)
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

# Alternative: pip (useful in containers or minimal environments)
pip install certbot certbot-nginx

Obtaining a Certificate¶

# Automatic Nginx configuration (obtains cert + modifies nginx config)
sudo certbot --nginx -d example.com -d www.example.com

# Certificate only (does not modify server config)
sudo certbot certonly --nginx -d example.com -d www.example.com

# Standalone mode (Certbot runs its own temporary web server on port 80)
sudo certbot certonly --standalone -d example.com

# DNS challenge for wildcard certificates
sudo certbot certonly --manual --preferred-challenges dns -d "*.example.com" -d example.com

fullchain.pem vs cert.pem

Certbot stores files in /etc/letsencrypt/live/example.com/. Always use fullchain.pem in your server configuration, not cert.pem. The fullchain includes both your certificate and the intermediate CA certificate. Using just cert.pem causes trust errors on clients that don't have the intermediate cached.

Certificate Files¶

After a successful issuance, Certbot creates these files:

File	Contents
`privkey.pem`	Your private key. Never share this.
`cert.pem`	Your certificate only (end-entity).
`chain.pem`	Intermediate CA certificate(s).
`fullchain.pem`	`cert.pem` + `chain.pem` combined. Use this in server configs.

Automated Renewal¶

Let's Encrypt certificates are valid for 90 days. Short lifetimes reduce the window of exposure if a key is compromised and encourage automation over manual processes.

How Certbot Renews¶

Certbot installs a systemd timer (or cron job) that runs certbot renew twice daily. It only renews certificates within 30 days of expiration.

# Check the renewal timer status
sudo systemctl status certbot.timer

# View all managed certificates and their expiration dates
sudo certbot certificates

# Test renewal without actually renewing (dry run)
sudo certbot renew --dry-run

# Force renewal of a specific certificate
sudo certbot renew --cert-name example.com --force-renewal

Post-Renewal Hooks¶

After renewing a certificate, you need to reload the web server so it picks up the new files. Certbot supports hooks for this:

# Reload Nginx after any successful renewal
sudo certbot renew --deploy-hook "systemctl reload nginx"

For a permanent hook, create a file in /etc/letsencrypt/renewal-hooks/deploy/:

#!/bin/bash
# /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
systemctl reload nginx

sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

Monitoring Certificate Expiration¶

Automated renewal should handle everything, but monitoring catches the cases where it doesn't - DNS changes that break challenges, Certbot bugs, or servers that were never set up for automation.

Script-Based Monitoring¶

#!/bin/bash
# check-cert-expiry.sh - Alert if certificate expires within 14 days
DOMAIN="example.com"
DAYS_WARNING=14

EXPIRY=$(echo | openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" 2>/dev/null \
  | openssl x509 -noout -enddate | cut -d= -f2)

EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
NOW_EPOCH=$(date +%s)
DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

if [ "$DAYS_LEFT" -lt "$DAYS_WARNING" ]; then
    echo "WARNING: $DOMAIN certificate expires in $DAYS_LEFT days ($EXPIRY)"
    # Send alert: email, Slack webhook, PagerDuty, etc.
fi

External Monitoring Services¶

For production systems, supplement local scripts with external monitoring that checks from outside your network:

Uptime Robot / Better Uptime: Monitor HTTPS endpoints and alert on certificate errors.
SSL Labs: Periodic deep scans of your TLS configuration.
Certbot certificates command: Quick local check of managed certificate status.

Never expose or commit private keys

Private keys (privkey.pem, .key files) must never appear in version control, log files, error messages, or backup archives without encryption. If a private key is compromised, you must revoke the certificate immediately (certbot revoke --cert-path /etc/letsencrypt/live/example.com/cert.pem) and obtain a new one.

Building an Internal CA¶

For internal services (microservices, development environments, VPN authentication), running your own CA avoids paying for commercial certificates and lets you issue certificates for internal hostnames that aren't publicly resolvable.

Create a Root CA¶

# Generate the CA private key (keep this offline/secure after setup)
openssl genrsa -aes256 -out ca.key 4096

# Create the self-signed root certificate (valid 10 years)
openssl req -new -x509 -key ca.key -sha256 -days 3650 \
  -out ca.crt -subj "/CN=Internal Root CA/O=My Company"

Sign a Server Certificate¶

# Generate the server's private key
openssl genrsa -out server.key 2048

# Create a CSR with SANs
openssl req -new -key server.key -out server.csr \
  -subj "/CN=api.internal" \
  -addext "subjectAltName=DNS:api.internal,DNS:api.staging.internal"

# Sign it with your CA (valid 1 year)
openssl x509 -req -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt -days 365 -sha256 \
  -copy_extensions copyall

Distributing the CA Certificate¶

For clients to trust certificates signed by your internal CA, they need your ca.crt installed as a trusted root:

# Ubuntu/Debian
sudo cp ca.crt /usr/local/share/ca-certificates/internal-ca.crt
sudo update-ca-certificates

# RHEL/CentOS
sudo cp ca.crt /etc/pki/ca-trust/source/anchors/internal-ca.crt
sudo update-ca-trust

# macOS
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ca.crt

Format Conversion¶

Different platforms require different certificate formats. These conversions come up frequently.

# PEM to PKCS#12 (for Windows/Java - bundles cert + key + chain)
openssl pkcs12 -export \
  -in server.crt -inkey server.key -certfile ca.crt \
  -out server.p12

# PKCS#12 to PEM (extract from Windows/Java format)
openssl pkcs12 -in server.p12 -out server.pem -nodes

# PEM to DER (binary format)
openssl x509 -in server.crt -outform DER -out server.der

# DER to PEM
openssl x509 -in server.der -inform DER -out server.pem

# Import PKCS#12 into a Java keystore
keytool -importkeystore \
  -srckeystore server.p12 -srcstoretype pkcs12 \
  -destkeystore server.jks -deststoretype jks

Troubleshooting¶

Certificate/Key Mismatch¶

The modulus of the certificate and key must match. If they don't, the server will fail to start TLS.

# These two hashes must be identical
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in server.key | openssl md5

Common cause: Renewed the certificate but forgot to update the key path in the server config, or mixed up files from different domains.

Incomplete Certificate Chain¶

Symptom: Desktop browsers work (they cache intermediates), but mobile devices, API clients, or curl report trust errors.

# Test the chain
openssl s_client -connect example.com:443 -servername example.com

# Look for this in the output:
# Verify return code: 21 (unable to verify the first certificate)
# This means the intermediate certificate is missing.

Fix: Use fullchain.pem instead of cert.pem in your server configuration.

Expired Certificate¶

# Check from the command line
echo | openssl s_client -connect example.com:443 2>/dev/null \
  | openssl x509 -noout -dates

Common cause: Certbot renewal failed silently. Check sudo certbot certificates and /var/log/letsencrypt/letsencrypt.log.

SNI Issues¶

If the wrong certificate is served, the server may not support Server Name Indication (SNI) or the server_name directive doesn't match.

# Explicitly test with SNI
openssl s_client -connect 1.2.3.4:443 -servername example.com