Securely Connecting Watchtower to Docker
Overview
Watchtower supports secure TLS connections to Docker hosts through its usage of Docker’s Go SDK to create a Docker client.
It is highly recommended to review Docker’s documentation:
- https://docs.docker.com/engine/daemon/remote-access/
- https://docs.docker.com/engine/security/protect-access
- https://docs.docker.com/reference/cli/dockerd/#daemon-socket-option
Configuration
TLS Verification
Argument: --tlsverify
Environment Variable: DOCKER_TLS_VERIFY
Type: Boolean
Default: falsehttp:// and unix:// schemes are incompatible with TLS verification.When TLS verification is enabled
- The use of
tcp://orhttps://schemes for the Docker host URL is required. tcp://is converted tohttps://when TLS verification is enabled.
TLS Certificate Path
Argument: --cert-path
Environment Variable: DOCKER_CERT_PATH
Type: String
Default: /etc/ssl/docker- This specifies the directory where Watchtower’s Docker client should find the certificate files within the Watchtower container.
- Docker expects the following filenames:
ca.pemcert.pemkey.pem
Docker Host URL
Argument: --host
Environment Variable: DOCKER_HOST
Type: String
Default: unix:///var/run/docker.sock/var/run/docker.sock) and a remote host is not supported.- This is required for connections to any Docker host.
- The use of
tcp://orhttps://schemes is required when using a TLS connection. tcp://is internally converted tohttps://when TLS verification is enabled.
Docker API Version
Argument: --api-version
Environment Variable: DOCKER_API_VERSION
Type: String
Default: <Auto-negotiated>- This provides the ability to manually specify the Docker API version.
- The default version autonegotiation should be sufficient for normal use cases.
Examples
remote-host with your actual Docker host address and /path/to/certs with the path to your certificate directory.docker run -d \
--name watchtower \
-v /path/to/certs:/etc/ssl/docker:ro \
nickfedor/watchtower --host tcp://remote-host:2376 --cert-path /etc/ssl/docker --tlsverify| Parameter | Description |
|---|---|
--name watchtower |
Assigns the name “watchtower” to the container for easy identification and management. |
-v /path/to/certs:/etc/ssl/docker:ro |
Mounts the local certificate directory to the container’s SSL directory as read-only. |
nickfedor/watchtower |
Specifies the Docker image to run, which is the Watchtower container image. |
--host tcp://remote-host:2376 |
Sets the Docker host to connect to via TCP on port 2376. |
--cert-path /etc/ssl/docker |
Defines the path inside the container where TLS certificates are located. |
--tlsverify |
Enables TLS verification for secure connections to the Docker host. |
-e flags to pass environment variables, then remember to place them before the nickfedor/watchtower image reference.Basic Tutorial
This is not a comprehensive guide and is merely a simple tutorial to illustrate a basic test deployment.
The following tutorial is intended to provide a basic walkthrough for manually setting up Watchtower to perform container updates on a Docker host that has enabled access to the Docker daemon using TLS.
This largely follow’s Docker’s guide for setting up TLS on the Docker host.
Tutorial Overview
In order for Watchtower to connect via TLS to a Docker daemon, the Docker daemon must be setup to accept remote connections using TLS.
Setting up TLS for Docker involves several key steps:
- Certificate Generation:
- Create a Certificate Authority (CA), server certificate, and client certificates using OpenSSL or similar tools.
- Daemon Configuration:
Start the Docker daemon with TLS options:
- `--tlsverify`: Enable TLS verification
- `--tlscacert`: Path to CA certificate
- `--tlscert`: Path to server certificate
- `--tlskey`: Path to server private key
- Client Setup:
- Prepare client certificates (`cert.pem` and `key.pem`) for authentication.
- Environment Variables:
- Set `DOCKER_HOST` to the secure endpoint (e.g., `tcp://host:2376`)
- Set `DOCKER_CERT_PATH` to the directory containing client certificates
- Set `DOCKER_TLS_VERIFY=1` to enable verification
For detailed instructions, refer to the Docker documentation on protecting the Docker daemon socket.
Certificate Generation
Generate self-signed certificates for testing (replace with proper certificates for production):
Create a CA Key and Certificate
-
Generate a 4096-bit RSA private key for the Certificate Authority and save it to
ca-key.pem:openssl genrsa -aes256 -out ca-key.pem 4096
-
Create a self-signed X.509 certificate for the Certificate Authority (valid for 365 days) using the private key and save it to
ca.pem:openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=US/ST=State/L=City/O=Org/CN=ca"
Create a Server Key and Certificate
-
Generate a 4096-bit RSA private key for the server certificate and save it to
server-key.pem:openssl genrsa -out server-key.pem 4096 -
Generate a certificate signing request (CSR) for the server with common name “localhost” and save it to
server.csr:openssl req -subj "/CN=$HOST" -sha256 -new -key server-key.pem -out server.csr -
Sign the server CSR with the CA certificate, creating a server certificate valid for 365 days with the specified extensions, and save it to
server-cert.pem:echo "subjectAltName = DNS:$HOST,IP:10.10.10.20,IP:127.0.0.1" > extfile-server.cnf
IP:10.10.10.20 is an example IP. Replace with your host’s actual IP.$HOST typically resolves to the hostname. Change this as necessary (i.e. localhost for testing)```bash
echo "extendedKeyUsage = serverAuth" >> extfile-server.cnf
```
```bash
openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile-server.cnf
```
Create a Client Key and Certificate
-
Generate a 4096-bit RSA private key for the client certificate and save it to
key.pem:openssl genrsa -out key.pem 4096 -
Generate a certificate signing request (CSR) for the client with common name “client” and save it to
client.csr:openssl req -subj '/CN=client' -new -key key.pem -out client.csr -
Sign the client CSR with the CA certificate, creating a client certificate valid for 365 days with the client authentication extension, and save it to
cert.pem:echo "extendedKeyUsage = clientAuth" > extfile-client.cnf
```bash
openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -extfile extfile-client.cnf
```
Key and Certificate Management
-
Remove both the certificate signing requests (
client.csrandserver.csr) and extensions config files (extfile-server.cnfandextfile-client.cnf) after generating the server (server-cert.pem) and clientcert.pemcertificates:rm -v client.csr server.csr extfile-server.cnf extfile-client.cnf -
Update the file permissions of the
ca-key.pem,server-key.pem, andkey.pemsecret keys:sudo chmod -v 0400 ca-key.pem server-key.pem key.pem -
Remove
ca.pem,server-cert.pem, andcert.pemfile write access:sudo chmod -v 0444 ca.pem server-cert.pem cert.pem -
Create the
/etc/docker/certsdirectory if it doesn’t exist:sudo mkdir -p /etc/docker/certs -
Copy the server-specific files:
sudo cp ca.pem ca-key.pem server-cert.pem server-key.pem /etc/docker/certs/ -
(Optional): Copy the client-specific files:
sudo cp cert.pem key.pem /etc/docker/certs/ -
Ensure root ownership:
sudo chown root:root /etc/docker/certs/* -
Set directory-level access:
sudo chmod 0755 /etc/docker/certs/ -
Verify the files:
ls -la /etc/docker/certs/
-r-------- for keys and -r--r--r-- for certs.-
(Optional) Remove the original files (in the original directory, not
/etc/docker/certs):rm -v {ca,ca-key,server-cert,server-key,cert,key}.pem
- The server files (
ca.pem,server-key.pem,server-cert.pem) stay on the daemon host. - The client’s files (
ca.pem,key.pem, andcert.pem) can be moved the client’s Docker directory (e.g.~/.docker/).
Setup the Docker Daemon with TLS
-
Edit or create
/etc/docker/daemon.jsonto enable TLS and TCP listening:{ "tls": true, "tlscacert": "/etc/docker/certs/ca.pem", "tlscert": "/etc/docker/certs/server-cert.pem", "tlskey": "/etc/docker/certs/server-key.pem", "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"] }
"tls": trueenables TLS verification."hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"]adds TCP listening on port 2376 and retains the Unix socket for local access.- Alternatively, use dockerd flags in
/etc/systemd/system/docker.service.d/override.conffor overrides without editingdaemon.json.
-
Restart the daemon:
sudo systemctl restart docker -
Verify the daemon is listening:
sudo netstat -tlnp | grep 2376
Watchtower Configuration
services:
watchtower:
image: nickfedor/watchtower
environment:
- DOCKER_HOST=tcp://remote-host:2376
- DOCKER_CERT_PATH=/etc/ssl/docker
- DOCKER_TLS_VERIFY=1
volumes:
- /path/to/certs:/etc/ssl/docker:ro
restart: unless-stoppedremote-host is used, but can be replaced with localhost for local testing.Troubleshooting
Insecure Scheme with TLS Verification
When TLS verification is enabled and the Docker host URL uses http://, Watchtower logs the following warning:
Possible Solutions:
- If using a secure connection, then use
https://. - If using
http://, then disable TLS verification.
Local Socket with TLS Verification
When TLS verification is enabled and the Docker host URL is not configured or uses unix://, Watchtower logs the following warning:
Possible Solutions:
- If the Docker host URL is not configured, then the default
unix:///var/run/docker.sockis used. - If using a local socket (i.e.
unix://), then disable TLS verification.
Missing TLS Certificates
In order for Watchtower’s Docker client to connect to the Docker daemon via TLS, the certificates must be provided to the Watchtower container.
If the certificates are available via the host filesystem of the Watchtower container’s Docker host, then this can be accomplished using bind mounts. Other options, such as building a custom Watchtower image with the certificates, are outside the scope of this documentation.
Refer to the documentation for the TLS Certificate Path and the examples.
Other Common Mistakes
- Using
tcp://without--tlsverify: This disables TLS, potentially allowing insecure connections. - Mismatched certificate paths: Ensure
DOCKER_CERT_PATHpoints to the correct directory containingcert.pemandkey.pem. - Expired or invalid certificates: Check certificate validity and SAN fields matching the host.
- Firewall blocking TLS port: Ensure port 2376 is open for remote connections.
Certificate Management
This documentation is not intended to be a guide on TLS/mTLS certificate deployment or management. Manual certificate management might be acceptable for smaller deployments; however, there are solutions for automating certificate management. If you are exposing your Docker daemon to external network connections, then both proper TLS setup and management is a highly recommended.
Here are just a few available solutions:
- Step-CA: Smallstep’s private, self-hostable certificate authority. [Tutorial]
- Vault: HashiCorp’s secret management tool with PKI secrets engine for certificate generation
- CFSSL: Cloudflare’s PKI toolkit for certificate management
Docker Socket Proxies
Docker socket proxies provide a security layer between applications and the Docker daemon by filtering API calls and preventing unrestricted access to the Docker socket.
While this documentation focuses on TLS-based connections, socket proxies represent another approach for securing Docker daemon access in environments where full socket exposure is undesirable.
The following projects are examples of Docker socket proxies: