Detailed explanation of the reasons and solutions for Docker failing to start normally

Detailed explanation of the reasons and solutions for Docker failing to start normally

1. Abnormal performance of Docker startup:

1. The status is repeatedly restaring, check it with the command

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21c09be88c11 docker.xxxx.cn:5000/xxx-tes/xxx_tes:1.0.6 "/usr/local/tomcat..." 9 days ago Restarting (1) Less than a second ago xxx10

2. There are obvious problems with Docker logs:

$docker logs [container name/container ID]

2. Possible reasons for Docker startup abnormality:

2.1. Insufficient memory

Docker requires at least 2G of memory to start. First, execute the free -mh command to check whether the remaining memory is sufficient.

View memory directly

$free -mh
      total used free shared buff/cache available
Mem: 15G 14G 627M 195M 636M 726M
Swap: 0B 0B 0B

Analyze logs

Sometimes the memory is overloaded for a moment, causing some processes to be killed. It seems that the memory is sufficient, but in fact Docker will restart repeatedly. Further analysis is needed through the Docker log and system log information:

Analyzing Docker logs

Check the docker log to see the memory overflow information. You need to read it carefully to find the information. It is not at the bottom

$docker logs [container name/container ID]|less 
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2756), pid=1, tid=140325689620224
#
# JRE version: (7.0_79-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: //core or core.1
#

Analyze system logs

Checking the system log, I found a large number of records of processes being killed due to memory overflow.

$grep -i 'Out of Memory' /var/log/messages
Apr 7 10:04:02 centos106 kernel: Out of memory: Kill process 1192 (java) score 54 or sacrifice child
Apr 7 10:08:00 centos106 kernel: Out of memory: Kill process 2301 (java) score 54 or sacrifice child
Apr 7 10:09:59 centos106 kernel: Out of memory: Kill process 28145 (java) score 52 or sacrifice child
Apr 7 10:20:40 centos106 kernel: Out of memory: Kill process 2976 (java) score 54 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3577 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3631 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3634 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3640 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3654 (java) score 47 or sacrifice child
Apr 7 10:27:27 centos106 kernel: Out of memory: Kill process 6998 (java) score 51 or sacrifice child
Apr 7 10:27:28 centos106 kernel: Out of memory: Kill process 7027 (java) score 52 or sacrifice child
Apr 7 10:28:10 centos106 kernel: Out of memory: Kill process 7571 (java) score 42 or sacrifice child
Apr 7 10:28:10 centos106 kernel: Out of memory: Kill process 7586 (java) score 42 or sacrifice child

2.2. Port Conflict

The Docker listening port has been occupied by other processes. This problem is prone to occur in newly deployed services or when deploying new background services on the original machine. Therefore, before deployment, you should execute a command to check whether the port is occupied. If it is found to be occupied after going online, you should change it to an available port and restart it.

Check command: $netstat -nltp|grep [planned port number]

3. Countermeasures

3.1. Countermeasures for insufficient memory:

Countermeasure 1:
3.1.1 After running for too long, the saltstack minion may occupy a large amount of memory and need to be restarted. The reboot command may not work sometimes. Mainly check the running status. If it fails to stop, restart it.

Countermeasure 2:
3.2.2 If the ELK log collection program or other Java processes occupy too much space, use the top and ps commands to check, carefully determine the role of the process, and stop the relevant process without affecting the business;

Strategy 3:
Release the occupied memory (buff/cache):
$sync #Write memory data to disk
$echo 3 > /proc/sys/vm/drop_caches #Release the occupied memory

Countermeasure 4:
Sometimes it is not that the buff/cache is too high that causes insufficient memory, but that the memory is indeed consumed by many necessary processes. In this case, it is necessary to consider and solve the problem from the perspective of machine resource allocation and usage.

3.2 Countermeasures for port conflicts

Countermeasure 1:
Generally, this problem is prone to occur in newly deployed services, or when deploying new background services on the original machine. Therefore, before deployment, you should execute a command to check whether the port is occupied. If it is found to be occupied after going online, you should change it to an available port and restart it.
Check command: $netstat -nltp|grep [planned port number]

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:
  • CentOS7 installs docker to solve the problem of not being able to start
  • Detailed explanation of how Docker starts a CentOS image
  • Solution and implementation steps for CentOS7 docker service failure to start
  • How to start multiple services simultaneously in Docker
  • How to enter the startup container in Docker
  • The difference between two ways to create a docker image and start the container (summary)
  • Solution to MySQL initialization and startup failure in Docker
  • Example of how to automatically start an application service in a Docker container
  • How to implement Docker container self-start
  • How to create, start, and stop a Docker container

<<:  Use of Vue filters and custom instructions

>>:  MySQL 8.0.11 MSI version installation and configuration graphic tutorial

Recommend

How to manage users and groups when running Docker

Docker is a management tool that uses processes a...

Implementation of formatting partitions and mounting in Centos7

Linux often encounters situations such as adding ...

Simple method to install mysql under linux

When searching online for methods to install MySQ...

Detailed explanation of BOM and DOM in JavaScript

Table of contents BOM (Browser Object Model) 1. W...

Implementation of react automatic construction routing

Table of contents sequence 1. Centralized routing...

MySQL replication table details and example code

MySQL replication table detailed explanation If w...

A brief discussion on the comparison of varchar, char and text in postgresql database

As shown below: name describe character varying(n...

CSS Problems with Using Position:fixed and Margin-top Together on Same-Level Elements

Problem Description I want to use CSS to achieve ...

VMware, nmap, burpsuite installation tutorial

Table of contents VMware BurpSuite 1. Virtual mac...

Mysql specifies the date range extraction method

In the process of database operation, it is inevi...

Implementation of Mysql User Rights Management

1. Introduction to MySQL permissions There are 4 ...

Vue.js performance optimization N tips (worth collecting)

Table of contents Functionalcomponents Childcompo...

Causes and solutions for MySQL too many connections error

Table of contents Brief summary At noon today, th...

Detailed explanation of two table copy statements: SELECT INTO and INSERT INTO SELECT (Differences between SQL database and Oracle database)

1. INSERT INTO SELECT statement The statement for...

Forever+nginx deployment method example of Node site

I recently bought the cheapest Tencent cloud serv...