Solution to 700% CPU usage of Linux process that cannot be killed

1. Problem Discovery

[root@zwlbs3 ~]# top

i. I found that the CPU usage of a process was 700%, and COMMAND was composed of some random strings. I was done! The first thing I thought of was to kill it.

[root@zwlbs3 ~]# kill -9 "PID"

ii. However, it was found that the process started again after being quiet for a while after being killed.

Note: The old image is reused, and the PID and COMMAND have changed.

2. View detailed information of the process

[root@zwlbs3 ~]# cd /proc/748/
[root@zwlbs3 748]# ls -ial

# "748" is the PID of the process. You can check it according to your PID.

As shown in the figure:

It is found that the process is in the /dev/shm directory. What is /dev/shm?

Let's take a paragraph from the Internet and explain /dev/shm

1) First of all, we can see that /dev/shm is a device file. We can think of /dev/shm as the entrance to the system memory. We can think of it as a physical storage device, a tmp filesystem. You can use this device to read and write files to the memory to speed up certain high I/O operations, such as frequent open, write, and read of a large file.

2) It is said that Oracle uses /dev/shm (shitou has never used Oracle), and the mount command can be used to list the current /dev/shm mounted file systems.

3) Since it is a memory-based file system, the files under /dev/shm will no longer exist after the system is restarted. The default Linux (CentOS) /dev/shm partition size is 50% of the system's physical memory, although using /dev/shm will be much more efficient for file operations. However, currently, few software distributions use it (except Oracle mentioned above). You can use ls /dev/shm to check whether there is a file under it. If not, it means that the current system is not using the device.

Check if there are any related files in the /dev/shm directory

[root@zwlbs3 ~]# ls -a /dev/shm/
. ..

# There is no related file, which is strange.

• There is no crontab related scheduled tasks.
• The which command also did not find any related files.
• Checking the system log is also normal, which is very strange.
• Almost no files related to this process were found.

3. Solution

i. View the thread occupancy analysis within a process

[root@zwlbs3 ~]# top -H -p "PID" 

ii. There are so many related processes, kill them all

iii. Check again after a few minutes and find that the system load has returned to normal

I thought it was solved, but when I checked a few hours later, it appeared again. Damn it.

Since it is not convenient to restart the server in the production environment, I had no choice but to try the restart method.

4. Restart Dafa

One hour after restarting the server, I checked again and it had returned to normal. Restarting still works well.

What does this malicious program do? Why only consume CPU resources? Since no relevant file information was found, the reason is not clear at the moment.

If anyone knows please tell me, thank you very much!

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. Thank you for your support of 123WORDPRESS.COM.