Docker Registry with AWS & Let's Encrypt

Nov 14 2016

With pretty much all new projects moving to Docker and Docker managers like Kubernetes, the Docker Registry becomes the first piece of infrastructure that a team needs to setup. Sure there are public and private registries available but most project would want to maintain their own setup.

So in this post we will setup a secure private Docker registry with the images backed up on AWS S3 and the free HTTPS certificates provided by Let's Encrypt. With this configuration we can setup our own secure, private registry at a minimal cost.

In essence running a Docker registry is simply a matter of running a Docker image with the right configuration parameters. Unfortunately, finding the right combination of parameters is difficult. The Registry image is here, an exhaustive list of all possible parameters is available here - https://docs.docker.com/registry/configuration/ and the AWS S3 specific options are here - https://docs.docker.com/registry/storage-drivers/s3/. As you can imagine this can all get messy quickly so I hope this post will help you to setup your registry in around 15 minutes. Let's get going.

Docker Registry

The first step would be to setup a new AWS EC2 instance, the type may depend on the amount of traffic you may have to handle but usually t2.small or any medium instance should suffice. I chose AWS Linux as my OS but Ubuntu or RedHat would also be fine. Just make sure you can run a modern version of Docker on the instance and can access S3 by applying the right AWS Role while creating the instance.

With the instance running, next install docker and nginx. Then we need to create a password file, all this could look like -

$ sudo yum update
$ sudo yum install docker
$ sudo service docker start
$ sudo docker ps
$ sudo yum install nginx
$ sudo mkdir -p /etc/nginx/htpass
$ sudo vim /etc/nginx/htpass/pass

In the pass file above we need to enter a BCrypted user+password pair. We can use a command line tool or use an online service like this one here. We just need to add our user entries in the pass file. An entry may look like (for docker/docker@123)-

docker:$2y$11$MOhVHbeiLRsBR0fk4f4khOL16I5.isDlL3z81gPb.lVnfdw86iVq6

Next, make sure you own a domain and point your domain to the new EC2 instance via your name servers. Also, make sure the DNS name propagation is working. Next up we need to setup Let's Encrypt and get those valuable yet free HTTPS certificates. I recommend the certbot option. This could look like -

$ wget https://dl.eff.org/certbot-auto
$ chmod a+x certbot-auto
$ sudo ./certbot-auto --debug
$ sudo ./certbot-auto certonly --standalone -d registry.foobar.co.com

As you can guess registry.foobar.co.com is my (cheap) domain name so please change the command according to your actual domain name. Certbot will ask a few questions, verify your domain ownership and finally issue you the certificates, please make sure you note the stored certificates' locations.

Well, we are almost there, all we need to do is run the Docker Registry image with the right parameters. So let's create a config.yml file instead of passing 20 parameters as command line arguments to Docker.

log:
  level: info
  formatter: text
  fields:
    service: registry
    environment: production
storage:
  s3:
    accesskey: CHANGEME
    secretkey: CHANGEME
    region: eu-central-1
    bucket: foobar-registry #CHANGEME
    chunksize: 5242880
    v4auth: true
    encrypt: false
  delete:
    enabled: false
  redirect:
    disable: false
  cache:
    blobdescriptor: inmemory
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false
auth:
  htpasswd:
    realm: basic-realm
    path: /etc/docker/htpass
http:
  addr: 0.0.0.0:5000
  secret: foobarbaz #CHANGEME
  relativeurls: false
  tls:
    certificate: /etc/docker/certs/live/registry.foobar.co.com/cert.pem #CHANGEME
    key: /etc/docker/certs/live/registry.foobar.co.com/privkey.pem #CHANGEME

Make sure you have the right AWS access and secret keys, AWS S3 settings and the right path to the certificates. We will mount the certificates from the host as volumes. With the configuration file safely stored, let's run the Container -

$ sudo docker run -tid -p 5000:5000 --restart=always --name registry
\ -v /home/ec2-user/config.yml:/etc/docker/registry/config.yml
\ -v /etc/nginx/htpass/pass:/etc/docker/htpass
\ -v /etc/letsencrypt:/etc/docker/certs registry:2.5.1

With all settings pointing to the right things we should have our Docker Registry container running. Now let's put a nginx proxy in front of it so that we can have additional security (if we need it) and we don't have to use the 5000 port for connections.

The nginx setup is pretty standard, sample nginx configuration file below -

# nginx.conf
user nginx;
worker_processes 4;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.fedora.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    client_max_body_size 0;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    include /etc/nginx/conf.d/*.conf;

    index   index.html index.htm;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  localhost;
        root         /usr/share/nginx/html;
        return 301   https://$host$request_uri;
    }

    server {
        listen       443 ssl;
        listen       [::]:443 ssl;
        server_name  localhost;

        ssl_certificate "/etc/letsencrypt/live/registry.foobar.co.com/cert.pem";
        ssl_certificate_key "/etc/letsencrypt/live/registry.foobar.co.com/privkey.pem";

        chunked_transfer_encoding on;
        add_header Docker-Distribution-API-Version registry/2.0 always;

        # It is *strongly* recommended to generate unique DH parameters
        # Generate them with: openssl dhparam -out /etc/pki/nginx/dhparams.pem 2048
        ssl_dhparam "/etc/pki/nginx/dhparams.pem";
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  180m;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP;
        ssl_prefer_server_ciphers on;

        # Load configuration files for the default server block.
        location / {
            proxy_pass                          https://127.0.0.1:5000/;
            proxy_set_header Host               $host;
            proxy_set_header X-Real-IP          $remote_addr;
            proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto  $scheme;
            proxy_read_timeout                  900;
        }
    }
}

The only main parameter that we need to be careful about is client_max_body_size since the uploaded image sizes can be large (zero is for no check which I use). With nginx running our registry is live!! Test it on your local machine by running -

$ docker login https://registry.foobar.co.com

If you are lucky the login for one of the users we created in htpass above may work. On Ubuntu/Linux workstations though you may be out of luck. At this point you will realize that Let's Encrypt is not a valid Certificate Authority in your system and you may want to jump out of the window.

But before you take any drastic actions, go to this page and download active root and intermediate certificates in text format and save them as .crt files in /usr/share/ca-certificates/extra/. Next run -

$ sudo dpkg-reconfigure ca-certificates
$ sudo update-ca-certificates

Make sure the newly downloaded certificates are enabled on your local machine via the commands above and then restart your Docker daemon locally. Logging into the registry should work after that. You can also test the setup by -

$ curl https://registry.foobar.co.com/v2/ -v

Finally, once login works you can try pushing something to the registry -

$ docker build -t registry.foobar.co.com/hello-knex-starter:0.0.5 .
...
$ docker push registry.foobar.co.com/hello-knex-starter:0.0.6
The push refers to a repository [registry.foobar.co.com/hello-knex-starter]
5a06c9d32de8: Pushed
4d6c7bc8fd45: Pushed
b69abf0c970f: Pushed
720e0e8cdd25: Pushed
b8c5627db3a0: Pushed
0766c47c917b: Pushed
396d0b2e3854: Pushed
0eb9317dc0bb: Pushed
4bb0fbbdd887: Pushed
3c726cce2df9: Pushed
011b303988d2: Pushed
0.0.6: digest: sha256:019413d29dc9bc5cecbe224674c5925540a47a82e43038e34e4b62d8041ab103 size: 2623

You can also login into S3 and see the data there.

That's it! Hope this saves you 1-2 days, you can also split the setup on 2-3 machines and run a load balancer for more resilience. Finally make sure you refresh the Let's Encrypt certificates every 90 days since right now the certificates are only valid for that period (you can also automate it via cron).