9

When running rsync on a directory with lots of files and directories at multiple levels, can we estimate the amount of work or time to finish?

  • the progress option only shows the progress of transferring a single file, not the progress of transferring all files and directories under the source directory.

  • One way I guess is to look at the what directories it has transferred so far, and compare that to the source.

    It will help a lot if I know the order of files and directories in which rsync transfers them.

    I guess that it may be related to that rsync runs multiple threads and what each thread does?

    I am not sure what order it chooses, and my previous guess of bread-first order seems not correct (so I strike it through).

Community
  • 1
Tim
  • 101,790

5 Answers5

9
rsync -arv /source desc --info=progress2

gives estimated time until rsync finishes job

2,863,290,944 64%  5.71MB/s  0:04:12

0:04:12 is remaining time

Tested on CentOS 7/8

overflowed
  • 191
4

You can make rsync print one line per file using -i and then use pv -l to report progress based on line count (in effect file count).

You will need pv (pipe viewer): http://www.ivarch.com/programs/pv.shtml

rsync -ai sourcedir/ targetdir/ | pv -l -s filecount > logfile

Use the following command to get file count:

find sourcedir | wc -l

Note: this command will show progress information based on number of files copied. This works best if there are many smallish files. If there are only a few files which are huge then you will not have much fun.

To see progress when you are updating (or comparing) an existing copy:

(more information: Compare directories but not content of files)

rsync -aii --delete sourcedir/ targetdir/ | pv -l -s filecount > logfile

The second -i makes rsync print one line per file even if they are equal.

Add -n to compare (not actually copy or delete anything).

Leave out --delete as needed.

This command will print to screen in real time the files that differ:

rsync -aii --delete sourcedir/ targetdir/ | pv -l -s filecount | 
    tee logfile | grep -v "^\."

The commands above work best when there are many smallish files. Here some workarounds if you have few huge files

Rsync has a built in progress report. See the rsync man page for -P or --progress or --info=progress2. I have not tested those much. Also those options will not work well with pv. Or at least I have not found out how.

Here is another crude workaround to see progress based on size:

  • note the free space of target partition before copying using df -h.
  • note the size of the source dir using du -sh.
  • use watch df -h on target and watch the size grow.

Obviosly this only works when copying and not when updating or comparing.

Lesmana
  • 27,439
1

So I know this is old, but I use a way that (to me) is much better than monitoring file count. Below is a script that monitors based on file size.

On the destination server, you run it like:

# rsync-monitor.sh <folder> <expected final size of folder> <seconds between measuring start/stop size>
rsync-monitor /folder 516835 300

That would monitor /folder expecting it to reach 516835MB, and every 300 seconds it would print out something like:

[Sun 21 Mar 2021 03:53:25 PM UTC] Starting              | Sleeping 300s...
[Sun 21 Mar 2021 03:58:25 PM UTC] 11900MB of 516835MB   | ~13MB/s       | 10h 15m 0s remaining  | Sleeping 300s...
[Sun 21 Mar 2021 04:03:27 PM UTC] 16497MB of 516835MB   | ~15MB/s       | 9h 0m 0s remaining    | Sleeping 300s...
[Sun 21 Mar 2021 04:08:29 PM UTC] 20974MB of 516835MB   | ~14MB/s       | 9h 10m 0s remaining   | Sleeping 300s...
[Sun 21 Mar 2021 04:13:31 PM UTC] 25600MB of 516835MB   | ~15MB/s       | 8h 50m 0s remaining   | Sleeping 300s...

Here's the script:

#!/usr/bin/env bash

The folder


FOLDER=$1


The expected size of folder


MEGS=$2


How many seconds to refresh


SLEEP=$3


SECONDSLEFT=1
START=$(du -sm $FOLDER |awk '{print $1}')
echo -e "[$(date)] Starting\t\t| Sleeping ${SLEEP}s..."
while (( $SECONDSLEFT > 0 )); do
        sleep $SLEEP
        END=$(du -sm $FOLDER |awk '{print $1}')
        DOWNLOADED=$(( $END-$START ))
        MEGSPERSECOND=$(( $DOWNLOADED/$SLEEP ))
        SECONDSLEFT=$(( ($MEGS-$END)/$DOWNLOADED$SLEEP ))
        HOURS=$(( $SECONDSLEFT/60/60 ))
        MINUTES=$(( ($SECONDSLEFT-$HOURS6060)/60 ))
        SECONDS=$(( $SECONDSLEFT-$HOURS6060-$MINUTES60 ))
        echo -e "[$(date)] ${END}MB of ${MEGS}MB\t| ~${MEGSPERSECOND}MB/s\t| ${HOURS}h ${MINUTES}m ${SECONDS}s remaining\t| Sleeping ${SLEEP}s..."
        START=$END
done
echo -e "[$(date)] Finished"
Fmstrat
  • 348
1

Use the flags --info=progress2 --no-inc-recursive to get mostly-accurate global progress stats. This needs rsync 3.1.0 at least. Incremental recursion was introduced in rsync 3.0.0, it avoids building a full list of files on the sender side, but it needs to be disabled so that estimates can take into account the whole set of files earlier. progress2 means more detailed progress, specifically this is an estimate of the overall progress of the transfer.

Tobu
  • 6,593
  • It takes quite a while for this command to calculate all the items it has to transfer. I find that it will constantly update this number. So much so that I find the ETA to be useless on transfers over a TB. I have a transfer going currently that says it's 90% complete, but it's only transferred about half the files. I use it like rsync -hasu --info=progress2 – t950 Feb 22 '23 at 23:23
0

You can use $ time rsync * /temp When you want to know estimate time.You should check total size of files before you did rsync. Fast or slow of rsync depended on your network when you did rsync thru network. You can use $ ls -ltr to check which directories have been backup it.

supriady
  • 167