Solution to the problem that Docker cannot stop or delete container services

Preface

Today, a developer gave me feedback that a container service cannot be stopped, rm (docker rm -f), or killed. In other words, the container service cannot be terminated.

Procedure

(1) The docker directory cannot be deleted by executing the delete command:

# ll /var/lib/docker/containers | grep caf8ef20f3c1
# cd /var/lib/docker/containers 
# rm -rf caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8

At this time we will receive an error like this:

rm: cannot remove "/var/lib/docker/containers/caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8/secrets": the device or resource is busy

Unable to delete "/var/lib/docker/containers/caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8/shm": Device or resource busy

(2) From the above error, we can see that the "secrets" and "shm" shared mounts cannot be deleted. First find the mount location, then cancel the mount, and then delete it:

# cat /proc/mounts |grep "docker" |grep "caf8ef20f3c1"

(3) Unmount:

# umount /data/sys/var/docker/containers/caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8/secrets
# umount /data/sys/var/docker/containers/caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8/shm

(4) Check again:

# cat /proc/mounts |grep "docker" |grep "caf8ef20f3c1" // no longer exists

(5) Now delete the docker directory:

# cd /var/lib/docker/containers 
# rm -rf caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8

(6) Delete container service

Now we use rm or kill to remove the container service:

# docker rm -f caf8ef20f3c1c

or

# docker kill --signal=SIGINT caf8ef20f3c1

If the above command hangs after running, please restart the Docker service:

# systemctl restart docker

After completing the above steps, the problem you encountered can basically be solved~

Supplement: Docker container cannot be stopped or killed

Problem Process

A MySQL container in a certain environment cannot be stopped, killed, or rm

sudo docker ps | grep mysql to view the container

7844250860f8 mysql:5.7.22 "/.r/r docker-entr..." 41 minutes ago Up 8 minutes r-dlrel-mysql-1-66df8f33

After using docker stop / docker kill / docker rm -f and other commands, the container will automatically restart immediately

Check the container immediately. The running time is: Up Less than a second, indicating that the container started immediately.

7844250860f8 mysql:5.7.22 "/.r/r docker-entr..." 42 minutes ago Up Less than a second r-dlrel-mysql-1-66df8f33

Kill the physical process corresponding to the container and restart it automatically

How to get the physical process: 1. The State.Pid field in docker inspect is the physical process ID; 2. ps command

Check the container restart policy. The policy is no, which means it will not restart automatically.

If you need to update the restart policy of a running container, you can use this command: docker update –restart=no my-container

"RestartPolicy": {
  "Name": "no",
  "MaximumRetryCount": 0
},

The magical way programmers solve problems

Have you ever encountered this scenario:

I was puzzled by a problem, but as soon as I walked in front of my colleague and explained the problem clearly, I suddenly understood it.

The problem was obviously very simple, but there was a problem when the program was running. Then I asked a colleague to help check the basic configuration, and I suddenly realized the answer.

This time I belong to the first type. As soon as I finished asking the question, I immediately remembered: Damn, it was the container orchestration tool Rancher that was doing the scheduling, and the container would automatically restart after it crashed.

I logged into Rancher and saw that it was indeed the case, a "wrong" problem. Although this is not a problem this time, Docker does have the problem of not being able to stop, and there is a lot of information about it.

The above is my personal experience. I hope it can give you a reference. I also hope that you will support 123WORDPRESS.COM. If there are any mistakes or incomplete considerations, please feel free to correct me.