How to implement nginx smooth restart

1. Background

During the server development process, it is inevitable to restart the service to load new code or configuration. If the service can be guaranteed to be uninterrupted during the server restart, the impact of the restart on the business can be reduced to zero. I recently researched nginx smooth restart and found it very interesting. I recorded it for interested students to read.

2. Restart process

• Restarting means the replacement of the old with the new. In the process of handing over tasks, the old and new servers will inevitably coexist. Therefore, the restart process is roughly as follows:
  • Start a new server
  • The old and new servers coexist, and both handle requests and provide services together.
  • The old server exits gracefully after processing all requests
• Here, the main problem is how to ensure that the old and new servers can coexist. If the server ports before and after the restart are the same, how to ensure that both can listen to the same port.

3. nginx implementation

In order to verify the smooth restart of nginx, the author first tried to start a new server instance again when nginx was started. The result is as shown in the figure:

Obviously, reopening the server instance does not work because the old and new servers use the same port 80. If the socket reuseport option is not enabled to reuse ports, the bind system call will fail. By default, nginx will retry bind 5 times and exit directly after failure. Nginx needs to listen to the IPV4 address 0.0.0.0 and the IPV6 address [::], so 10 emerg logs are printed in the figure.

Next, we will try the smooth restart command, which consists of two commands:

kill -USR2 `cat /var/run/nginx.pid`
kill -QUIT `cat /var/run/nginx.pid.oldbin`

The first command sends the USR2 signal to the old master process. The pid of the process is stored in the /var/run/nginx.pid file, where the nginx.pid file path is configured by nginx.conf.

The second command sends the signal QUIT to the old master process. The pid of the process is stored in the /var/run/nginx.pid.oldbin file, and then the old master process exits.

So the question is, why does the pid of the old master process exist in two pid files? In fact, after sending the USR2 signal to the old master process, the old master process renamed the pid, and the original nginx.pid file was renamed to nginx.pid.oldbin. In this way, the new master can use the file name nginx.pid.

Execute the first command first, the result is as shown:

Yes, the old and new master and worker processes coexist. Let's run the second command. The result is as shown below:

As you can see, the old master process 8527 and its worker processes all exited, leaving only the new master process 12740.

I can't help but wonder why manually starting a new instance doesn't work, but restarting with a signal can. First look at the nginx log file:

In addition to the previous error log, there is one more notice, which means that sockets are inherited and the fd values ​​are 6 and 7. Following the log, I looked through the nginx source code and located the nginx.c/ngx_exec_new_binary function.

ngx_pid_t
ngx_exec_new_binary(ngx_cycle_t *cycle, char *const *argv)
{
  ...
  ctx.path = argv[0];
  ctx.name = "new binary process";
  ctx.argv = argv;
  n = 2;
  env = ngx_set_environment(cycle, &n);
...
  var = ngx_alloc(sizeof(NGINX_VAR)
          + cycle->listening.nelts * (NGX_INT32_LEN + 1) + 2,
          cycle->log);
...
  p = ngx_cpymem(var, NGINX_VAR "=", sizeof(NGINX_VAR));
  ls = cycle->listening.elts;
  for (i = 0; i < cycle->listening.nelts; i++) {
    p = ngx_sprintf(p, "%ud;", ls[i].fd);
  }
  *p = '\0';
  env[n++] = var;
...
  env[n] = NULL;
...
  ctx.envp = (char *const *) env;
  ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);
  if (ngx_rename_file(ccf->pid.data, ccf->oldpid.data) == NGX_FILE_ERROR) {
    ...
    return NGX_INVALID_PID;
  }
  pid = ngx_execute(cycle, &ctx);
  if (pid == NGX_INVALID_PID) {
    if (ngx_rename_file(ccf->oldpid.data, ccf->pid.data)
      == NGX_FILE_ERROR)
    {
      ...
    }
  }
...
  return pid;
}

The function flow is

1. Copy all fds monitored by the old master process to the env environment variable NGINX_VAR of the new master process.
2. rename rename pid file
3. The ngx_execute function forks a child process and execve executes the command line to start a new server.
4. In the server startup process, the environment variable NGINX_VAR is parsed. The specific code of ngx_connection.c/ngx_add_inherited_sockets is:

static ngx_int_t
ngx_add_inherited_sockets(ngx_cycle_t *cycle)
{
...
  inherited = (u_char *) getenv(NGINX_VAR);
  if (inherited == NULL) {
    return NGX_OK;
  }
  if (ngx_array_init(&cycle->listening, cycle->pool, 10,
            sizeof(ngx_listening_t))
    != NGX_OK)
  {
    return NGX_ERROR;
  }
  for (p = inherited, v = p; *p; p++) {
    if (*p == ':' || *p == ';') {
      s = ngx_atoi(v, p - v);
      ...
      v = p + 1;
      ls = ngx_array_push(&cycle->listening);
      if (ls == NULL) {
        return NGX_ERROR;
      }
      ngx_memzero(ls, sizeof(ngx_listening_t));
      ls->fd = (ngx_socket_t) s;
    }
  }
  ...
  ngx_inherited = 1;
  return ngx_set_inherited_sockets(cycle);
}

The function flow is:

Parse the value of the environment variable NGINX_VAR and get the fd to store in the array

The socket corresponding to fd is set to ngx_inherited to save the information of these sockets.

In other words, the new server does not rebind the listen port at all. These fd states and values ​​are brought over when the new master process forks. The new master process can listen to and process the inherited file descriptors. The key point here is that the listen socket file descriptor is passed through ENV.

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.