24
[root@centos /]# rsync -av --exclude thumbs /storage root@xx.27.1.xx:/storage
root@xx.27.1.xx's password: 
building file list ...

I've been sitting for about an hour... it's 135GB of images and folders

/storage is a mounted ext3 scsi drive.

Is it normal for rsync to be sitting this long calculating the files/directories?

Tombart
  • 2,193
Andrew Fashion
  • 1,665
  • How many files? With very large numbers of files it's gonna take a while. – Helvick Dec 02 '10 at 19:24
  • probably thousands... images directory of user uploads... – Andrew Fashion Dec 02 '10 at 19:37
  • I have seen this portion of rsync take well over an hour when there's thousands and thousands and thousands of files. If you run 'top' are you seeing I/O wait? –  Dec 02 '10 at 19:52
  • Just because I am curious, can you run df -i and tell us what you have for IUsed on /storage? It will give us a rough idea of how many files+folders you have. – Zoredache Dec 02 '10 at 21:06
  • 1383641, is that the amount of files? – Andrew Fashion Dec 02 '10 at 21:24
  • It is the number of inodes. There is one inode for every filesystem object like files and directories. So no, it isn't the number of files, but it does tell us that you have a pretty large set of files. – Zoredache Dec 03 '10 at 19:24

5 Answers5

28

rsync 2.x does build a full file list up front.

Add the -P option and you'll see a progress indicator.

If you think it really is hanging, in another terminal find the pid of rsync and then

  1. Run strace -p PID and see what it's doing. (Hit ^C to stop.)

  2. Run ps -o wchan PID to see where it is in the kernel.

poolie
  • 1,175
  • 1
  • 9
  • 17
  • 1
    Wow strace is cool! You'll probably find that something circular is going on: Like a link that refers to one of its ancestor directories. In my case I was using a virtual filesystem that had no limit on its depth, i.e. it had "infinitely" many subdirectories tagfs/books/+/books/+/books/+/ ... – Zaz May 17 '15 at 16:28
  • @Zaz I also have a huge amount of subdirectories, in depth. But this is in my system design. Is there a way to solve this? – user1641443 Sep 14 '16 at 11:56
  • @user1641443: I'm sorry, that's beyond me. – Zaz Sep 16 '16 at 07:30
  • @user1641443: 1: Try rsync>3.0 as Martin says. 2: Run multiple rsync processes each on a on smaller sub-tree. 3: Open your own separate question. – poolie Sep 17 '16 at 15:25
16

You should upgrade to rsync 3.0.x where you will get the benefit of incremental file lists, explained here. I am rsyncing millions of images (~200GB total) and saw an enormous speedup when going from rsync 2.x to 3.x.

Even so, it will probably take a long time to go through all that data. In my case it still takes over an hour between two pretty powerful DL380 G5 servers.

Martin
  • 716
  • 1
    Also note that certain flags will force the old behavior; for me, it was --delay-updates. – Xiong Chiamiov Jun 06 '16 at 22:02
  • 1
    @XiongChiamiov The only options I'm using are "-a" and "--progress" and version 3.1.x is still building an incremental file list. Why? – Michael Apr 23 '17 at 22:35
3

You noted earlier that du was hanging as well. Something is up with this filesystem, and rsync is hitting whatever du was.

One quick thing you can check is for kernel error messages indicating a disk problem. Type dmesg and see what's at the bottom.

Community
  • 1
mattdm
  • 6,670
  • 1
  • 27
  • 49
1

Based on this and your other question of du /storage hanging makes me think that neither du or rsync is the problem but rather there are some issues with /storage.

Mark Wagner
  • 18,129
1

The hang after the message "building file list..." can be caused by MTU mismatch, e.g. if you've set up the network interfaces with MTU 9000 but not the switch in between, small packets (like connecting to the rsync server) will pass through but not bigger ones (like sending the file list).

The Mutator
  • 21
  • after much searching I came across this and it prompted me to go and check - was running rsync over a vpn with standard 1450 mtu but I am not in control of the equipment in between the sites - changed the vpn mtu down to 1430 and viola the rsync works 100% (and suddenly other network strangeness disappeared too!) thanks! – l0ft13 Nov 10 '18 at 10:34