Wagtail with Gunicorn, Nginx and SSL on Docker and optional PostgreSQL

Sun 22 Mar 2020

/Web Development

This is a tutorial on how to setup you Wagtail CMS using Gunicorn as the Http server, Nginx as a reverse proxy and SSL to improve our security. We will serve the static content through Nginx and we will deploy everything using Docker! Finally you are going to find tips on how to use PostgreSQL as your database. This is a similar process as to how erev0s.com has been setup with some small differences.

Disclaimer: It is difficult to depict every possible scenario in a tutorial and there are different ways to approach things. Please feel free to send me a message if you think that something else can be added as well.

Prerequisites:

An already made Wagtail Site
Installed Docker and Docker-Compose
Proper domain name to be used for the SSL

We are going to start by presenting the final structure of the project along with the contents of the files. This way if you already know what you are looking for it would be easier to find it. After that we are going to analyze line by line the code. You will see how to deploy the code using SQLite initially and at the end of the document you will find how to integrate PostgreSQL with it.

├── docker-compose.yml
├── nginx
│   ├── Dockerfile
│   ├── nginx.conf
│   └── nginx.confssl
└── wagtail_site
    └── Dockerfile
    └── other_app_files/directories

Lets analyze first the docker-compose.yml file. We expect it to include our wagtail site, nginx and the certbot to serve through SSL.

version: '3.7'

services:
  wagtail_site:
    container_name: wagtail_site
    build: ./wagtail_site
    command: gunicorn wagtail_site.wsgi:application --bind 0.0.0.0:8000
    restart: unless-stopped
    expose:
      - 8000
    environment:
      - DEBUG=0
      - SECRET_KEY=asuperS3cretK$yshouldgohere
      - DJANGO_SETTINGS_MODULE=wagtail_site.settings.production
    volumes:
      - ./data/static_volume:/code/static
    networks:
      - wagtailnet

  nginx:
    container_name: Nginx
    build: ./nginx
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - "/path/to/static/data/static_volume:/home/wagtail_site/static"
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    networks:
      - wagtailnet
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"

  certbot:
    container_name: CertBot
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    networks:
      - wagtailnet
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"


networks:
  wagtailnet:

It is time to see in more detail the three services present in the docker-compose file. We start from our wagtail site.

The build: ./wagtail_site directive simply states that the Dockerfile to build this service can be found in the directory wagtail_site.
The command specifies Gunicorn to be our HTTP server which as we will see later on in the Dockerfile it will have already been installed in the Docker machine.
The expose directive is used instead of ports, as we want the port to be accessible only internally.
The environment variables declared are so we can control some fine points of our site through the docker-compose file. These fine points include the DEBUG state, the secret key and the settings file to be used.
The volume creates a persistent storage of the static files of our application. We know that the static files of our wagtail site will be located at /code/static/ as we specify this in the Dockerfile. We do this as it allows us to edit any static file we might want to change from the host at a later stage after we have deployed the site. It has though one drawback, you will need to manually copy your static content to the host location specified (copy the static content into /data/static_volume)
Finally we use a network called wagtailnet in case we need to provide access to any of the services through a different docker-compose file (we will see more about it later on)

Next step is to investigate the contents of the Dockerfile for the wagtail_app.

# Use an official Python runtime as a parent image
FROM python:3.7
LABEL maintainer="[email protected]"

# Set environment varibles
ENV PYTHONUNBUFFERED 1

COPY ./requirements.txt /code/requirements.txt
RUN pip install --upgrade pip
# Install any needed packages specified in requirements.txt
RUN pip install -r /code/requirements.txt
RUN pip install gunicorn

# Copy the current directory contents into the container at /code/
COPY . /code/
# Set the working directory to /code/
WORKDIR /code/

RUN python manage.py migrate

RUN useradd wagtail
RUN chown -R wagtail /code
USER wagtail

Some things to note in the Dockerfile are: location. This includes the static content of your site which if you have specified the your

We copy the requirements.txt and we install our project`s requirements using pip.
We copy the rest of our wagtail site code into the /code/ location. This inlcludes the static content of your site which if you have specified the your STATIC_URL to be /static/ means that the static content would be located at /code/static/.
We specify also a user with username wagtail to be running our application for security reasons.

NGINX & SSL

Moving on to the Nginx service in the docker-compose file two things stand out:

The volume which contains the static content of the site and we will instruct nginx on its configuration to serve it later on. Please note that the path/to/static/ is actually the path where the docker-compose file itself is located!
The command which instructs nginx to restart every six hours. This is required to catch up any changes made to the ssl certificate as we will see further down.

The Dockerfile from which nginx service will be build is the following:

FROM nginx:1.17.3

RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d

It is pretty self explanatory, we use the image of nginx 1.17.3 with the configuration specified in the nginx.conf file located in the same directory as the Dockerfile itself.

Before going over the nginx.conf file itself I would like to also get started with the certbot service we see in the docker-compose file as Nginx and SSL are intertwined in this setup. We are using the certbot from letsencrypt to issue the certificates for SSL, which is a pretty convenient service. A thing to note for the certbot service is that we have an entrypoint line which is responsible for renewing our certificate! If you are a bit confused about docker command, entrypoint and run you could take a look at this nice article.

Also do keep in mind that the nginx service along with the certbot service do have common volumes thus they are linked together. This is essential so nginx can load all required files to serve using SSL.

Now we reach a point where we need to deploy our site but we do not have the SSL certificate yet, thus we cannot use a proper nginx.conf file which would be configured for SSL. There are two options here, first option is to create dummy certificate and deploy nginx with a proper nginx.conf while on the second one we deploy nginx without SSL support we acquire the certificates and then we redeploy nginx with the proper nginx.conf to support SSL. For the first option you can check out this article which described the whole things pretty well. I am choosing the second option as I believe it is more clear on what you are doing.

The nginx.conf we are going to use initially is the following.

upstream wagtail_site {
    server web:8000;
}

server {

    listen 80;
    server_name wagtaiSite.com www.wagtaiSite.com;
    charset utf-8;

    location / {
        proxy_pass http://wagtail_site;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

    location /static/ {
        alias /home/wagtail_site/static/;
    }


      location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

}

This will deploy our wagtail site along with the cerbot location /.well-known/acme-challenge/ as we see in the nginx.conf. Now it is time to create a small script to help us with getting the certificate. Create a file named init.sh on the same directory as your docker-compose.yml file. Add the following code to your init.sh file and chmod +x init.sh so you can execute it.

#!/bin/bash

if ! [ -x "$(command -v docker-compose)" ]; then
  echo 'Error: docker-compose is not installed.' >&2
  exit 1
fi

domains=(wagtaiSite.com www.wagtaiSite.com)
rsa_key_size=4096
data_path="./data/certbot"
email="[email protected]" # Adding a valid address is strongly recommended
staging=1 # Set to 1 if you're testing your setup to avoid hitting request limits

if [ -d "$data_path" ]; then
  read -p "Existing data found for $domains. Continue and replace existing certificate? (y/N) " decision
  if [ "$decision" != "Y" ] && [ "$decision" != "y" ]; then
    exit
  fi
fi

if [ ! -e "$data_path/conf/options-ssl-nginx.conf" ] || [ ! -e "$data_path/conf/ssl-dhparams.pem" ]; then
  echo "### Downloading recommended TLS parameters ..."
  mkdir -p "$data_path/conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
  echo
fi

echo "### Starting the docker-compose file"
docker-compose up -d

echo "### Requesting Let's Encrypt certificate for $domains ..."
#Join $domains to -d args
domain_args=""
for domain in "${domains[@]}"; do
  domain_args="$domain_args -d $domain"
done

# Select appropriate email arg
case "$email" in
  "") email_arg="--register-unsafely-without-email" ;;
  *) email_arg="--email $email" ;;
esac

# Enable staging mode if needed
if [ $staging != "0" ]; then staging_arg="--staging"; fi

echo "Requesting the certificate"
docker-compose run --rm --entrypoint "\
  certbot certonly --webroot -w /var/www/certbot \
    $staging_arg \
    $email_arg \
    $domain_args \
    --rsa-key-size $rsa_key_size \
    --agree-tos \
    --force-renewal" certbot
echo


echo "Shutting down the docker-compose"
docker-compose down

You can experiment with staging=1 so you avoid hitting the request limits. Now run the init.sh and if everything worked fine you will have a file named wagtaiSite.com.conf created under the directory data/certbot/conf/renewal/. This means that you now have your certificates issued and you can move on to alter you nginx.conf file to the following one

gzip on;
gzip_vary on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable "MSIE [1-6]\.";

upstream wagtail_site {
    server web:8000;
}


server {
    server_name www.wagtaiSite.com;
    return 301 $scheme://wagtaiSite.com$request_uri;
}

#Block for the https://www redirect to https://
#It is much much faster than using ifs
server {

    proxy_connect_timeout       600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    send_timeout                600;


    listen 443 ssl;
    server_name www.wagtaiSite.com;
    charset utf-8;


   ssl_certificate /etc/letsencrypt/live/wagtaiSite.com/fullchain.pem;
   ssl_certificate_key /etc/letsencrypt/live/wagtaiSite.com/privkey.pem;
   include /etc/letsencrypt/options-ssl-nginx.conf;
   ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

   return 301 https://wagtaiSite.com$request_uri;
}


server {

    proxy_connect_timeout       600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    send_timeout                600;


    listen 443 ssl;
    server_name wagtaiSite.com;
    charset utf-8;


   ssl_certificate /etc/letsencrypt/live/wagtaiSite.com/fullchain.pem;
   ssl_certificate_key /etc/letsencrypt/live/wagtaiSite.com/privkey.pem;
   include /etc/letsencrypt/options-ssl-nginx.conf;
   ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;


    location / {
        proxy_pass http://wagtai_site;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

    location /static/ {
        alias /home/wagtai_site/static/;
    }

}


server {

    listen 80;
    server_name wagtaiSite.com www.wagtaiSite.com;
    charset utf-8;

    location / {
	return 301 https://$host$request_uri;
   }


      location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

}

The main things to note here are:

We redirect traffic from www.wagtaiSite.com to wagtaiSite.com.
We use the SSL certificated from the shared volume the nginx and certbot service have.
We serve the static content as before.
We redirect non https traffic to https.
and we still serve on port 80 the acme-challenge for future renewals.

Last thing to do, it to run docker-compose up --build -d. This will rebuild the nginx service and therefore getting the new nginx.conf we specified.

Now you should have the wagtail site being deployed along with nginx and SSL and you should have access to it through your domain.

PostgreSQL with Wagtail

Ok now, what about if you want to use PostgreSQL for your database. The things we have to do are:

Add a new service to the docker-compose.yml file
Update the Wagtail settings
Install Psycopg2

Lets start by adding postgres as a new service to our docker-compose file and adjust the environmental variables of the wagtail_site service. The result should be similar to the following

version: '3.7'

services:
  wagtail_site:
    container_name: wagtail_site
    build: ./wagtail_site
    command: gunicorn wagtail_site.wsgi:application --bind 0.0.0.0:8000
    restart: unless-stopped
    expose:
      - 8000
   depends_on:
      - db
    environment:
      - DEBUG=0
      - SECRET_KEY=asuperS3cretK$yshouldgohere
      - DJANGO_SETTINGS_MODULE=wagtail_site.settings.production
      - SQL_ENGINE=django.db.backends.postgresql
      - SQL_DATABASE=hello_wagtail
      - SQL_USER=hello_wagtail
      - SQL_PASSWORD=hello_wagtail
      - SQL_HOST=db
      - SQL_PORT=5432
    volumes:
      - ./data/static_volume:/code/static
    networks:
      - wagtailnet

  db:
    container_name: wagtail_site_psql
    image: postgres:latest
    volumes:
      - ./data/postgresql:/var/lib/postgresql/data/
    environment:
      - POSTGRES_USER=hello_wagtail
      - POSTGRES_PASSWORD=hello_wagtail
      - POSTGRES_DB=hello_wagtail
    networks:
      - wagtailnet

  nginx:
    container_name: Nginx
    build: ./nginx
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - "/path/to/static/data/static_volume:/home/wagtail_site/static"
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    networks:
      - wagtailnet
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"

  certbot:
    container_name: CertBot
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    networks:
      - wagtailnet
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"


networks:
  wagtailnet:

A copy of the database is stored in /data/postgresql, so the data are persisted at all times in the host machine.

Next step to add to the wagtail site Dockerfile to install Psycopg2. We do this by adding a line RUN pip install psycopg2.

Finally we edit the settings of the wagtail site to specify the use of PostgreSQL as the database. We do this by changing the DATABASES to:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': os.environ.get('SQL_DATABASE', 'backupdb'),
        'USER': os.environ.get('SQL_USER', 'backupuser'),
        'PASSWORD': os.environ.get('SQL_PASSWORD', 'password'),
        'HOST': os.environ.get('SQL_HOST', 'localhost'),
        'PORT': os.environ.get('SQL_PORT', '5432'),
    }
}

The environmental variables are there to make it easy for us to adjust the credentials from the docker-compose file.

The rest of the process is the same as using the SQLite database described earlier.

Conclusion

In this tutorial we saw how to deploy your Wagtail site with Nginx and SSL using Docker and optionally to use PostgreSQL as well. A very similar process is how erev0s.com is deployed and this was the spark to create this article. There are many small details that you could differentiate depending on what you want to achieve but it is not possible to cover everything here.

If you have some question/problem you might want to share with me, please feel free to contact me.