How to quickly copy large files under Linux

Copy data

When copying data remotely, we usually use the rsync command, but if a large number of small files are copied, the rsync transmission speed will be slow. This problem can be solved by using tar pv lz4 to package and compress the transmission. Using this method is equivalent to using scp and rsync to transfer large files.

In actual tests, 1200G is transferred using rsync. The size of a single file is tens of KB~2GB. With a gigabit network card, 6 rsyncs need to be run simultaneously to fully utilize the bandwidth. The speed of each rsync is about 20MB, and the speed fluctuates greatly. About 4.5GB can be copied per minute.

However, using tar pv lz4, you only need to run one, and the speed fluctuation is small. About 6.8GB can be copied per minute.

Rsync usage examples

rsync installation: yum install -y rsync

# Push [root@vm5 ~]# rsync -auvzP -e "ssh -p22" mssh.tar.gz [email protected]:/data/
sending incremental file list
mssh.tar.gz
     1,977 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
sent 2,069 bytes received 35 bytes 4,208.00 bytes/sec
total size is 1,977 speedup is 0.94

# Pull [root@vm5 ~]# rm -f mssh.tar.gz
[root@vm5 ~]# rsync -auvzP -e "ssh -p22" [email protected]:/data/mssh.tar.gz .
receiving incremental file list
mssh.tar.gz
     1,977 100% 1.89MB/s 0:00:00 (xfr#1, to-chk=0/1)
sent 43 bytes received 2,069 bytes 4,224.00 bytes/sec
total size is 1,977 speedup is 0.94

Explanation of parameter auvzP: Parameter a is archive transfer, which retains file attributes; u is update transfer, and if the source file modification time is newer, it will be transferred. v means display detailed process, z means compressed transmission, and P means breakpoint transmission.

Note: When rsync transfers folders, if folder/ has /, the files in the directory will be transferred; if it does not have /, the folder will be transferred as well.

Use compressed transmission

Install pv and lz4 tools

Note: This needs to be installed on both ends of the server.

pv is not in the yum source, you can find it on the pv official website

# Go to the pv official website, get an rpm package link, and install it directly with the rpm command [root@vm5 ~]# rpm -ivh http://www.ivarch.com/programs/rpms/pv-1.6.6-1.x86_64.rpm
Get http://www.ivarch.com/programs/rpms/pv-1.6.6-1.x86_64.rpm
WARNING: /var/tmp/rpm-tmp.mFbA6u: Header V3 DSA/SHA1 Signature, Key ID 3fc56f51: NOKEY
Preparing... ################################# [100%]
Upgrading/installing...
  1:pv-1.6.6-1 ################################### [100%]
  
# lz4 can be installed directly with yum [root@vm5 ~]# yum install -y lz4

use

[root@vm5 ~]# time tar -c go |pv |lz4 -B4 |ssh -p22 -c aes128-ctr 192.168.176.11 "lz4 -d |tar -xC /data/"
using blocks of size 64 KB
18.1MiB 0:00:00 [49.5MiB/s] [ <=> ]
real 0m0.376s
user 0m0.080s
sys 0m0.108s
# Comparison with rsync
[root@vm5 ~]# time rsync -auvzP -e "ssh -p22" go 192.168.176.11:/data/
......
sent 11,741,677 bytes received 10,451 bytes 7,834,752.00 bytes/sec
total size is 18,502,481 speedup is 1.57
real 0m1.130s
user 0m0.797s
sys 0m0.160s
[root@vm5 ~]#

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.