Detailed explanation of the solution to the problem of nohup log output being too large under Linux

Recently, I ran a spark streaming program in a hadoop test cluster, and then used nohup ./execute.sh & to run the program in the background. After just a few days, the log file was over G. If there is a problem and I want to view the log file, it is obviously a troublesome thing to open the file, so I tried to reduce the file size by:

1. Explanation of nohup command:

a. Syntax: nohup [command] [args] [&]

b. Description: The nohup command runs the command specified by the Command parameter and any related Arg parameters, ignoring all hangup signals. Use the nohup command to run the program in the background after logging out. To run the nohup command in the background, add & (the symbol for "and") to the end of the command. If you do not specify redirection, the log is output to the nohup.out file in the current directory by default.

Generally submit like: nohup ./execute.sh & so that the log or output is in the current running directory.nohup.out

Redirection: nohup ./execute.sh > /home/xxx/log.log 2>&1 & : This way the log will be redirected to the specified directory

2. Split nohup.out and prevent it from growing indefinitely

The general submission command I use here is: nohup ./execute.sh &, so there will be a nohup.out file in the current directory. At this time, you can find a way to regularly split nohup.out into multiple small files, but at the same time make sure that nohup.out does not grow indefinitely (generally, the program cannot be interrupted):

a. Every day (set the time as needed), split the log of the previous day regularly (for example, if it is about 1g per day, you can split it into about 100m each time),

b. After splitting, the nohup.out file will be saved to ensure that the new output log will continue to be output to nohup.out

The above in the shell

current_date=`date -d "-1 day" "+%Y%m%d"`

split -b 65535000 -d -a 4 nohup.out ./log/log_${current_date}_ The split command is used here to split the nouhup file according to the specified size (65535000b is about 60 MB, the size can be customized), and divided into the specified format (-d -a 4 with a 4-digit suffix starting from 0000, for details, you can Baidu split command usage), the final output format is log_20160610_0001

cat /dev/null > nohup.out (This command will instantly clear the nohup.out file, and will continue to write to the file later), directing the log to /dev/null

The same can be done using redirected output, except that the redirected file name is used instead.

Define these commands in a shell file and run them regularly every day. This way, the daily logs will be divided into several parts, which is convenient for troubleshooting and if the log backlog is too large. You can delete historical logs regularly and keep only the last few days.

The overall code is as follows:

this_path=$(cd `dirname $0`;pwd)
 
cd $this_path
echo $this_path
current_date=`date -d "-1 day" "+%Y%m%d"`
echo $current_date
split -b 65535000 -d -a 4 /home/.../nohup.out /home/.../log/log_${current_date}_
 
cat /dev/null > nohup.out

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.