Realization of real-time file synchronization between Linux servers

Usage scenarios

For existing servers A and B, if the content in the specified directory of server A (for example, /home/paul/rsync/ ) changes (additions, deletions, modifications, and attribute changes), these changes will be synchronized to the target directory of server B (for example, /home/paul/rsync/ ) in real time.

Data mirroring backup tool Rsync

Rsync is an extremely fast and flexible file copying tool. It supports file copying between the local machine and a remote server. Rsync uses a delta-transfer algorithm, which only needs to transfer the difference between the source and target files, greatly reducing the consumption of network bandwidth and the time spent on copying. Rsync is mostly used for data backup and mirroring.

Rsync uses a fast check algorithm to determine whether a file needs to be synchronized by comparing changes in file size or last modification time.

There are two ways to connect to a remote host using Rsync: using ssh or using rsync daemon. Rsync is used here to implement remote file backup.

Installation and operation of Rsync

Install Rsync

Execute in the terminals of Server A and Server B respectively:

sudo yum install rsync

After the installation is complete, you will find that the rsync configuration file is located in etc/rsyncd.conf. This file is required when daemon is used for synchronization and is not introduced here.

Configure password-free login between servers A and B

Server A executes:

ssh-keygen
ssh-copy-id IP address of server B

Create source and target directories

In Server A:

mkdir /home/paul/rsync

In Server B:

mkdir /home/paul/rsync

Create a test file on Server A

echo "Hello from Server A" >> /home/paul/rsync/demo.txt

Execute file transfer command

On Server A, run:

# (1)
rsync -avPz --progress /home/paul/rsync 192.168.100.130:/home.paul/rsync
# (2)
rsync -avPz --delete --progress /home/paul/rsync 192.168.100.130:/home.paul/rsync

You will find that demo.txt also appears in the /home/paul/rsync directory on server B.

Command analysis:

(1) Copy the files in the /home/paul/rsync directory on server A to /home.paul/rsync on server B (192.168.100.130).

(2) Compare the files on the target side and the files on the source side. If the file on the target side does not exist on the source side, delete the file on the target side.

Problems with Rsync

Rsync is only a file copy tool, it cannot monitor the addition, deletion, and modification operations of source files. After making changes on the source side, you need to execute the rsync command to synchronize the changes to the target side.

Rsync needs to scan the entire directory before each synchronization. If there are many files in the source directory, scanning may take a long time.
In order to meet the requirements of real-time monitoring, we need to introduce another tool: inotify.

File system event monitoring tool inotify

inotify-tools provides a simple interface to inotify. It is a library written in C language and also includes command line tools.

For a detailed introduction to inotify-tools, please click: https://github.com/rvoicilas/inotify-tools/wiki

Installation of inotify-tools

For centos7 system, execute in sequence:

yum install -y epel-release
yum --enablerepo=epel install inotify-tools

Use the inotifywait command to monitor events

The monitoring script is as follows (inotifywait-rsync.sh):

inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f' -e modify,delete,create,attrib /home/paul/rsync/ | while read file
do
rsync -avPz --progress /home/paul/rsync/ 192.168.100.130:/home/paul/rsync/
rsync -avPz --delete /home/paul/rsync/ 192.168.100.130:/home/paul/rsync/
echo "${file} was synchronized"
done

Parameter parsing

• -m keeps monitoring. If this parameter is not set, inotifywait will exit after monitoring one event.
• -r monitor directories recursively.
• -q Quiet mode, print less output.
• --timefmt specifies the output format of time.
• --format Specifies the format of event output.
• -e Set the event type to monitor. Here we monitor additions, deletions, modifications and metadata changes.

For each triggered listening time, inotifywait executes the code between do and done. Here, we call the rsync command mentioned earlier to synchronize files.

Add the monitoring script to crontab

crontab -e
* * * * * sh /home/paul/inotifywait-rsync.sh

References
https://rsync.samba.org
https://github.com/rvoicilas/inotify-tools/wiki

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.