A solution to the abnormal exit of Tomcat caused by semaphore

I'm playing with big data recently. A friend came to me and said that his online tomcat would inexplicably exit. He was very distressed and asked me for help. Every time he found himself logged out, he would log in through Tencent Cloud's WEB console and start Tomcat.

In the spirit of helping others (shao kao hao chi), I connected and started analyzing. The first thing to do is to check the tomcat log to see if there is any relevant information recorded and how it exited.

From the log, it can be seen that Tomcat received the exit request and closed the container as required. So can we assume that someone executed shutdown.sh? No. The shutdown log after executing the shutdown script is as follows.

The related tomcat source code screenshots are as follows. There are line numbers on the left side of the screenshot.

When tomcat starts, set await to wait for the shutdown instruction to enter. org\apache\catalina\startup\Bootstrap.java

The definition of catalinaDaemon is as follows.

org\apache\catalina\startup\Catalina.java

When instantiating, the instance of the interface Server will be pointed to StandardServer. The classpath is as follows.

org\apache\catalina\Server.java

org\apache\catalina\core\StandardServer.java

The source code of the output related logs in StandServer is as follows:

The configuration file read is org\apache\catalina\core\LocalStrings.properties

When Tomcat receives a proper shutdown instruction, this log is output, indicating that an instruction was received to shut down the container.

The proper instructions to close the container, the relevant code is as follows.

So, the current evidence shows that this Tomcat is not shut down through the SHUTDOWN message. Moreover, the following figure also shows that it is not so easy to issue the SHUTDOWN command successfully.

So the most likely way now is to operate through the KILL command. Executing a bash script requires logging into the machine, so how about looking for the login records at that time from wtmp and utmp?

Below are the results of IPIP.

In other words, when tomcat exited abnormally on the morning of the 23rd, an address from Tencent Cloud's BGP data center also coincidentally disconnected the session. My friend's machine is placed in Tencent Cloud. A bit strange, isn't it?

I continued to investigate, tracing back the tomcat logs for several days in a row, comparing the utmp, wtmp results, and then comparing the IPIP results, and the results were all the same. The session from Tencent Cloud BGP data center is disconnected, and Tomcat exits at the same time. Accurate to seconds. This happens many times over many consecutive days, indicating that there is a causal relationship between the tomcat exit and the WEB session exit.

After asking, my friend confirmed that he was accustomed to logging into the server using the WEB console. After starting Tomcat, he put it aside and began to debug the interface. So what could possibly cause such a causal relationship to occur? This brings us to the long-standing inter-process communication mechanism of the Linux system - semaphore.

For specific information about semaphores, please consult relevant materials to learn more. For this problem, it can be simply understood as a mechanism for inter-process communication.

Process A needs process B to do something, but the memory areas between processes are in a sense invisible to each other. At this time, it needs to be completed through semaphore. Process A can send a semaphore to process B according to the pre-defined semaphore specification. When process B receives it, it determines the processing logic based on the specific value of the semaphore. The specific semaphore list can be queried through the following command on the command line. The commands are all letters, without the number 1.

The most common one is 9, SIGKILL. When a process receives this semaphore, it will be KILLed. This semaphore is handled by the operating system and cannot be handled by applications. In Windows systems before Vista, there was a way to penetrate the kernel. At this time, you can intercept messages like WM_CLOSE to prevent a program from being closed. In the era of win7 and win10, such techniques can no longer be used.

In addition, the familiar CTRL + C operation sends SIGINT. In some scenarios, we need to notify the program to exit gracefully. In this case, we can issue SIGQUIT, which is kill -3.

So what signal will be issued when the WEB console session is disconnected? Let’s try it and find out. Although Java cannot operate the underlying system, the sun.misc package has some surprises. The code is as follows.

As shown in the figure, this code will output the thread name, semaphore name, and translate it into a specific number when receiving a semaphore. Then, in the main function, I "registered" four semaphores: HUP, INT, ABRT, and TERM. I registered four types because I wasn't sure what would come out, so I just registered all the possible ones.

Compile and package. There is one issue that needs attention here. Since semaphores belong to the underlying mechanism of the operating system, the semaphores supported by each different operating system are different. The JVM provides support through private static native int findSignal(String paramString). The native method involves specific VM implementation, so I won’t post the code. But it is easy to think that Windows and Linux are of course different. So here we need to write on the upper computer and debug on the lower computer. Don't panic if an error message appears while running under Windows.

The rest is easy. Upload the program to the server, log in to the server through the WEB console, redirect the execution results to a text file, and then wait for the console to time out. The results are as follows.

There is no need to analyze the results. The WEB console will issue a SIGHUP when exiting, which is equivalent to kill -1. What will tomcat do when it receives SIGHUP? You can try to kill -1 pid and then look at the log to understand.

The solution is actually simple. SIGHUP means HANG UP. You can use nohup xx.sh & to completely block SIGHUP and SIGINT. In addition, after testing, it was found that the semaphore can also be triggered by clicking the × on the right side of the SecureCRT tab page, but directly logging out or clicking the red cross on the menu will not trigger it.

Interested friends can expand this program and test it out. Maybe your abnormal exit problem can be solved.

The solution to this problem is inseparable from Liu Jiangang from Shanghai ZTO, and special thanks to him.

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. Thank you for your support of 123WORDPRESS.COM.