9

I need to copy one very large file (3TB) on the same machine from one external drive to another. This might take (because of low bandwidth) many days.

So I want to be prepared when I have to interrupt the copying and resume it after, say, a restart. From what I've read I can use

rsync --append

for this (with rsync version>3). Two questions about the --append flag here:

  1. Do I use rsync --append for all invocations? (For the first invocation when no interrupted copy on the destination drive yet exists and for the subsequent invocations when there is an interrupted copy at the destination.)

  2. Does rsync --append resume for the subsequent invocations the copying process without reading all the already copied data? (In other words: Does rsync mimic a dd-style seek-and-read operation?)

Kusalananda
  • 333,661
halloleo
  • 567

2 Answers2

9

Do I use rsync --append for all invocations?

Yes, you would use it each time (the first time there is nothing to append, so it's a no-op; the second and subsequent times it's actioned). But do not use --append at all unless you can guarantee that the source is unchanged from the previous run (if any), because it turns off the checking of what has previously been copied.

Does rsync --append resume for the subsequent invocations… without reading all the already copied data?

Yes, but without rsync --partial would probably have first deleted the target file.

The correct invocation would be something like this:

rsync -a -vi --append --inplace --partial --progress /path/to/source/  /path/to/target

You could remove --progress if you didn't want to see a progress indicator, and -vi if you are less bothered about a more informational result (you'll still get told if it succeeds or fails). You may see -P used in other situations: this is the same as --partial --progress and can be used for that here too.

  • --append to continue after a restart without checking previously transferred data
  • --partial to keep partially transferred files
  • --inplace to force the update to be in-place

If you are in any doubt at all that the source might have changed since the first attempt at rsync, use the (much) slower --append-verify instead of --append. Or better still, remove the --append flag entirely and let rsync delete the target and start copying it again.

Chris Davies
  • 116,213
  • 16
  • 160
  • 287
  • "because it turns off the checking of what has previously been copied" -- Rather than turning off, isn't it because it checks by way of filesize? I would have expected that changes to either destination or source wouldn't cause corruption with --append if you use it with -c/--checksum. EDIT: Indeed it turns it off instead of checksumming to the minimum between the 2 sizes. That's a shame. – JoL Sep 07 '22 at 22:20
  • Apparently for --append to behave like --append-verify when used with -c. I hadn't noticed there was that separate option. EDIT: Yes. – JoL Sep 07 '22 at 22:27
  • Cool answer! Clears up all my questions in great detail. Thanks. – halloleo Sep 08 '22 at 00:22
0

For this you'll want to use the --partial option. From the manpage:

--partial                keep partially transferred files
meijin3
  • 139
  • 1
    also --inplace option might be needed.

    --inplace This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is com‐ plete, rsync instead writes the updated data directly to the destination file.

     – toppk Sep 07 '22 at 14:20
  • 1
    Tahnks for chiming in! - So I would need --partial --inplace for both, first and subsequent invocations? – halloleo Sep 07 '22 at 14:35
  • I'd also care about the advice given 10 years ago. Which version of rsync in these times ? They possibly meant what more recent rsync would name append-verify. Anyway, I always use append-verify – MC68020 Sep 07 '22 at 14:39