rsync --delete with superset destination folder

Question

I have an rsync process which is syncing content from a source repository (which is version controlled) into a shared NFS mount.

The scenario (as awful as it is) is that the destination folder contains more content than the source folder because other content is synced to the destination folder from different sources. So for instance, folder structures may look like this:

source

a/a1.txt
a/a2.txt
b/b1.txt

destination

a/a1.txt
a/a2.txt
a/a3.txt
b/b1.txt
c/c1.txt

(in this example, a/a3.txt and c/c1.txt are synced to the destination from elsewhere. In practice this involves multiple other sources and the content/processes for these can’t be influenced.)

Now say that source folder deletes the a/a2.txt file. Using the existing setup, this file would not be deleted on the destination; but using --delete would result in other files being deleted, and it is a requirement to not do this.

How could --delete be used on this rsync but meet the requirement? Because the source directory is version controlled, it is simple enough to get a before-and-after of this directory, so a differential backup could be calculated using the original source directory as a reference, but is this the best way?

Answer 1

You cannot use rsync --delete like this. It's stateless and keeps no record of which files have been deleted between runs. The --delete flag simply instructs rsync to delete every file on the destination that does not exist on the source.

In order to implement this constrained deletion I think you need to maintain your own state. Neither rsync nor unison can do this for you.

The following is not a full error-safe solution; it's a starting point. (However, it does handle files with strange names - including those containing an embedded newline.)

Assume two directories src and dst. (For the purposes of the example it doesn't really matter whether dst is local or remote.)

# Find the current list of files (do this just once, to prep the cache)
( cd src && find . -type f -print0 ) | LC_ALL=C sort -z > .state.src

Each time we perform a backup, run the following code

# Run the rsync to transfer files. "dst/" could be local
rsync -av src/ remote:dst/

# Determine the set of files to delete in "dst/"
( cd src && find . -type f -print0 ) | LC_ALL=C sort -z | tee .state.src.new |
    LC_ALL=C comm -z - -13 .state.src |
    ssh remote 'while IFS= read -d "" -r f; do rm -f "dst/$f"; done'

# That seemed to work, so update the state cache
[[ 0 -eq $? ]] && mv -f .state.src.new .state.src

---

If your version of comm (like mine) is older than GNU coreutils 8.25 and does not have the -z flag, you can use this alternative workaround:

# Find the current list of files (do this just once, to prep the cache)
( cd src && find . -type f -print0 ) | tr '\0\n' '\n\0' | LC_ALL=C sort > .state.src

Each time we perform a backup, run the following code

# Run the rsync to transfer files. "dst/" could be local
rsync -av src/ remote:dst/

# Determine the set of files to delete in "dst/"
( cd src && find . -type f  -print0 ) | tr '\0\n' '\n\0' | LC_ALL=C sort | tee .state.src.new |
    LC_ALL=C comm -13 - .state.src |
    tr '\0\n' '\n\0' |
    ssh remote 'while IFS= read -d "" -r f; do rm -f "dst/$f"; done'

# That seemed to work, so update the state cache
[[ 0 -eq $? ]] && mv -f .state.src.new .state.src