How to use docker to deploy Django technology stack project

With the popularity and maturity of Docker, it has gradually become the first choice for deploying projects. Today I will share with you how to use docker to deploy Django technology stack projects.

The Django technology stack we are talking about here is: python3.6, Django2.2, redis, mysql, celery, gunicorn and nginx. In actual production projects, these components are distributed on different machines in the cluster. For example, Nginx, redis, and Mysql may be managed by separate teams or departments. The deployment architecture and container orchestration involved are more complex and will not be explored in depth in this article. This article mainly introduces how to use docker-compose to orchestrate these components. This method is suitable for the deployment of test environment or your personal sideproject.

This article assumes that you already know some basic knowledge of docker and docker-compose. If you don’t, you can read the following materials:

• Docker Knowledge
• Docker official documentation
• Docker Compose documentation

Let's talk about how to deploy it.

Project Organizational Structure

First, let's take a look at our project organizational structure, which is as follows:

├── LICENSE
├── README.md
├── compose
│ ├── celery
│ │ ├── Dockerfile
│ │ ├── celery-beat.sh
│ │ └── celery.sh
│ ├── mysql
│ │ └── my.cnf
│ ├── nginx
│ │ └── nginx.conf
│ └── web
│ ├── Dockerfile
│ ├── entrypoint.sh
│ ├── gunicorn.conf
│ └── gunicorn.sh
├── docker-compose.yml
├── docker_django_demo
│ ├── __init__.py
│ ├── celery.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
├── env.tpl
├── manage.py
├── requirements.txt

In addition to the Django project files, the main additions are the compose configuration file directory and the docker-compose.yml configuration file.

• The compose directory mainly stores the dockerfile files and startup scripts of each component.
• docker-compose.yml is the orchestration configuration file of docker-compose.

Write Dockerfile and start initialization script

In docker-compose, there are two ways to start the container. One is to directly use a public image to start the container, and the other is to use a Dockerfile written by ourselves. Because we need to install additional toolkits and initialize related configurations, we use a custom Dockerfile for the web and celery components.

compose/web/Dockerfile for the web container:

FROM python:3.6
ENV PYTHONUNBUFFERED 1

RUN mkdir /code
WORKDIR /code

COPY ./requirements.txt /code/
RUN pip install --no-cache-dir -r requirements.txt \
  && rm -rf requirements.txt

COPY ./code/
COPY ./compose/web/*.sh /code/
RUN sed -i 's/\r//' gunicorn.sh \
  && chmod +x gunicorn.sh \
  && sed -i 's/\r//' entrypoint.sh \
  && chmod +x entrypoint.sh

ENTRYPOINT ["/bin/bash", "entrypoint.sh"]

Other files of the web container:

• compose/web/entrypoint.sh is the startup script of the web container, which executes some initialization or detection logic.
• compose/web/gunicorn.conf Gunicorn configuration file.
• compose/web/gunicorn.sh is the startup script of gunicorn.

Dockerfile for celery:

FROM python:3.6
ENV PYTHONUNBUFFERED 1

RUN mkdir /code
WORKDIR /code

COPY ./requirements.txt /code/
COPY ./compose/celery/*.sh /code/
RUN pip install --no-cache-dir -r requirements.txt \
  && rm -rf requirements.txt && sh init_env.sh

COPY ./code/
COPY ./compose/celery/*.sh /code/
RUN sed -i 's/\r//' celery.sh \
  && chmod +x celery.sh \
  && sed -i 's/\r//' celery-beat.sh \
  && chmod +x celery-beat.sh

Other files of celery:

• compose/celery/celery.sh Celery startup script.
• compose/celery/celery-beat.sh The startup script of celery-beat.

Write the Compose startup configuration file

The docker-compose configuration is as follows:

version: '2'

services:
 redis:
  image: redis
  ports:
   - "6379:6379"

 db:
  restart: always
  image:mysql:5.7.19
  # command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
  volumes:
   - ./compose/mysql/:/etc/mysql/conf.d
   - ./db:/var/lib/mysql
  # for test
  ports:
   - "127.0.0.1:3307:3306"
   # (HOST:CONTAINER)
  env_file:
   - .env

 web:
  # restart: always
  build:
   context: .
   dockerfile: ./compose/web/Dockerfile
  command: sh gunicorn.sh # ["/bin/bash", "gunicorn.sh"]
  ports:
   - "8080:8002"
   # (HOST:CONTAINER)
  volumes:
   - ./logs:/var/logs/
   - ./collect_static:/code/collect_static
   - ./static:/code/static
   - ./templates:/code/templates
   - ./uploads:/code/uploads
  env_file: .env
  depends_on:
   - redis
   -db

 nginx:
  restart: always
  image: nginx:1.13.0
  volumes:
   - ./compose/nginx:/etc/nginx/conf.d/
   - ./staticfiles:/code/staticfiles
   - ./logs:/var/log/nginx
  ports:
   - "80:80"
   # (HOST:CONTAINER)
  depends_on:
   - web

 celery:
  build:
   context: .
   dockerfile: ./compose/celery/Dockerfile
  command: sh celery.sh
  volumes:
   - ./logs:/var/logs/
   - ./uploads:/code/uploads
  depends_on:
   - redis
   -db
  env_file: .env

 celery-beat:
  build:
   context: .
   dockerfile: ./compose/celery/Dockerfile
  command: sh celery-beat.sh
  volumes:
   - ./logs:/var/logs/
  depends_on:
   - redis
   -db
  env_file: .env

Celery worker and beat Here we use the same image Dockerfile. According to the principle of one image and one process, we start two containers to run the worker and beat processes respectively.

Compile test

After writing the configuration file, compile the image and test it:

docker-compose build 
docker-compose up # Run in the foreground docker-compose up -d # Run in the background after everything is correct

docker-compose ps You can see the started container:

$ docker-compose ps  
       Name Command State Ports     
--------------------------------------------------------------------------------------------------
docker djangodemo_celery-beat_1 sh celery-beat.sh Up               
docker djangodemo_celery_1 sh celery.sh Up               
dockerdjangodemo_db_1 docker-entrypoint.sh mysqld Up 127.0.0.1:3307->3306/tcp
dockerdjangodemo_nginx_1 nginx -g daemon off; Up 0.0.0.0:80->80/tcp   
dockerdjangodemo_redis_1 docker-entrypoint.sh redis ... Up 0.0.0.0:6379->6379/tcp 
dockerdjangodemo_web_1 /bin/bash entrypoint.sh sh ... Up 0.0.0.0:8080->8002/tcp 

The mapping port can be adjusted according to your actual situation.

question

Here are a few issues that need attention during the construction process.

mysql encoding problem

The default encoding of the MySQL image provided by Docker is latin1, which will display garbled characters when saving Chinese. The official website provides a method to modify the encoding method. You can specify the encoding format after starting the script. The document can be found here. For MySQL container version 5.7.19, you can add the parameters --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci directly after command in docker-compose.yml. This method only modifies the encoding on the server side. You can directly use the configuration file to override and specify all encoding formats.

The configuration is as follows:

[mysqld]
default-storage-engine=INNODB
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
init-connect='SET NAMES utf8mb4'
init_connect = 'SET collation_connection = utf8mb4_general_ci'
skip-character-set-client-handshake # Skip the client encoding configuration, and the client directly uses the server encoding configuration bind-address = 0.0.0.0

Note: The configuration file method of MySQL 5.7.19 succeeds, while 5.7.4 and 5.7.17 fail. You can use this as a reference.

Web waits for MySQL to start before continuing

The mysql container cannot accept database links before it is started. When the web is initialized, if the database has not been started yet, the web container will fail to start and exit directly. We can add a detection script when the web container is started, and then continue after the database is connected.

The script is as follows:

#!/usr/bin/env bash
set -o errexit
set -o pipefail

echo $MYSQL_PASSWORD
echo $MYSQL_DATABASE
echo $MYSQL_HOST
echo $MYSQL_USER
echo $MYSQL_PORT

function mysql_ready(){
python << END
import sys
import pymysql
try:
  conn = pymysql.connect(host="db", port=3306, user="root", passwd="$MYSQL_ROOT_PASSWORD", db='$MYSQL_DATABASE', charset='utf8')
except pymysql.err.OperationalError:
  sys.exit(-1)
sys.exit(0)
END
}

until mysql_ready; do
 >&2 echo "MySQL is unavailable - sleeping"
 sleep 1
done

>&2 echo "MySQL is up - continuing..."

Summarize

At this point, the deployment of the Django technology stack service using Docker is complete. For the complete project code, you can refer to docker-django-demo.

As mentioned at the beginning of the article, this deployment method is not suitable for online production services of large websites. It has many problems such as high coupling and poor maintenance. However, deploying your own sideproject or test environment is still very good when hardware resources are limited. In addition to reducing the trouble of environment deployment and setup, it is also very convenient to migrate.

The demo project also has some examples of how to use docker in a development environment, but I personally think that docker is more suitable for deployment, and it is not as flexible and convenient as building it directly in a development environment. Welcome everyone to leave a message and discuss the experience of using Docker in development and deployment.

refer to

cookiecutter-django

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.