3

I have just setup a Debian server for backup of Windows PCs. I am currently testing various methods including: rsync (using Deltacopy on Windows), SMB via Samba and scp. Currently rsync (without SSH and with no compression) and scp take about 16 minutes over gigabit LAN for a 9 GBs file transfer. However SMB only takes a couple of minutes. Why is rsync in particular so slow? I feel I might be misunderstanding where the real benefits of rsync come from, that being from transferring only the changed bits. However I still feel it doesn't explain the initial slow transfer speed and I must be doing something wrong.

I am transferring to a Samba share on my Linux box in all instances.

Cristian Ciupitu
  • 5,583
Ben
  • 127
  • 1
  • 6
  • Duplicati is another nice backup tool. – Cristian Ciupitu Jan 21 '14 at 00:30
  • It certainly looks nice and I will definitely consider it but it appears to lack true folder synchronization, rather it just pushes an archived backup file across the network a la Cobian Backup. – Ben Jan 21 '14 at 00:41
  • 1
    It seems that the question is answered here:

    http://superuser.com/questions/153176/how-to-rsync-a-large-file-with-as-little-cpu-and-bandwidth-expense-as-possible

     – mestia Jan 21 '14 at 10:08

1 Answers1

1

the problem is that rsync uses the modification time and size of a file as a quick-reference to judge whether or not a file has changed. There is a more detailed and precise way to check whether files have changed and rsync supports that aswell, however if the modification time and size of a file defer in the first place, these "more detailed" checks are skipped and then rsync assumes the file has changed, hence tries to copy the whole file over.

Specifically when in a heterogeneous environment (Windows+Linux, ...) and when not working on local filesystems (i.e. when using SMB mounts/shares or other protocols), there is a slight possiblity, that the modification times are not passed through correctly between the rsync source and it's destination.

It may very well be that the modification times are rounded up or otherwise altered by OS and/or network protocol used and hence appear be different (even if just slightly, even if the local filesystem has the "correct" mtime).

Please check whether this may or may not be the case & test it via command line (if possible) and utilities like "stat" (Linux) or if absolutely necessary perl/python mini-programs, which output the exact modification time as propagated via the OS and local filesystem properties (ultimately bypassing any network protocols like Samba).

Example output for the "stat" utility under unix:

$ stat somefile.txt 
  File: `somefile.txt'
  Size: 1014        Blocks: 8          IO Block: 4096   regular file
Device: 805h/2053d  Inode: 1448800     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-07-21 17:20:33.548997182 +0530
Modify: 2011-08-16 23:27:19.648480473 +0530
Change: 2011-08-16 23:27:19.648480473 +0530

It can't be that your rsync commands consistently take "16 minutes" or so. That doesn't make any sense. Usually, rsyncs are getting quicker the less the differences are before & after each run. Only scp will take a constant time every time you run it, depending on the copy volume. So that fact, plus the fact that you already ruled out compression & encryption (ssh) can be the performance bottle-neck really make me assume there may be something going on with the modification time comparisons.

I specifically had this case myself with Samba mounts from my Synology NAS. However, whether that is also the case with you or not, I hope you find the real cause of your problem soon.

Have fun.

Paxsali
  • 303