Split and merge - with DVDs!

Question

Say I have a large file called foo.tar.xz. I split the file into parts of just under 4.7GB each, using split -b 4689887232 foo.tar.xz foo.tar.xz., which gives me files named foo.tar.xz.aa, foo.tar.xz.ab, etc. Then I write each file to a different DVD and send it to Alice using a rather large homing pigeon.

Now Alice could insert each DVD, copy each file to her PC, and then use cat and xz to get the contents of foo: cat foo.tar.xz.* | tar xfJ -

Now Assume that Alice has just enough space on her PC to store the extracted contents of foo. Is there some flavour of cat that will read these files directly from DVDs and pause the stream to allow her to insert the next DVD? Something like pausecat or volumecat?

Is re-creating the tarball an option? I.e. on your end, do you have the original files, or enough room to extract the tarball? — Stephen Kitt, May 10 '18 at 09:49
I was going to suggest using a multi-volume archive, but that doesn’t support seamless compression, so I imagine it wouldn’t be appropriate in this case. — Stephen Kitt, May 10 '18 at 14:48
It may be for some people finding this question. Feel free to write an answer and give some command line examples. — rudolfbyker, May 12 '18 at 16:26

score 3 · Answer 1 · answered May 09 '18 at 23:21

I don't know of such a cat flavor, but here's a solution which almost works:

In one terminal run mkfifo myfifo; tail -c +1 -f myfifo | tar xfJ -
Insert first DVD into DVD-ROM drive (ex. dev/sr0).
In a second terminal run dd if=/dev/sr0 of=myfifo
When dd completes, remove the DVD.
If you have another DVD, insert into the DVD-ROM drive and go to step 3, else go to step 6.
In the first terminal, press CTRL-C to kill tail and tar.

How it works

The trick is to use tail to continuously read from a named pipe. The output of tail is then piped to the command you want to run. Initially the pipe is unopened so nothing happens. But when you begin pumping data into the pipe with dd, tail picks it up and forwards it to your command.

The magic happens when the flow of data to the pipe stops: tail keeps its standard out file descriptor open, which causes your command to pause. Meanwhile, tail simply waits for more input.

The problem

The reason I said it's almost working is because there seems to be a buffering issue which causes tail not to write the last bit of data that's fed into it. My hope is that someone can provide the insight to address this.

score 3 · Answer 2 · answered May 10 '18 at 09:20

You can write your own catpause script:

#!/bin/bash
while read -p 'ready ? ' >&2
do if [ y = "$REPLY" ]
   then cat /dev/dvd 
   fi
done |
tar xJf -

It prompts you on stderr to reply, and if it is y you copy the dvd into a pipe read by the tar command. When done enter control-D for end-of-file.

Split and merge - with DVDs!

2 Answers2

How it works

The problem