Piping with Process substitution and joining output again

Question

I'm trying to make use of a powerful remote server in terms of video encoding.

I have a local DVD drive for ripping DVD to memory, finally residing in an mbuffer.

From there, I would like to split streams, redirecting the raw video stream over the network to the remote agent for encoding and piping the resulting stream back, whilst, at the same time, converting the audio stream locally to another format stream. Then, finally, I would like to join both resulting streams to a new file.

Part of this can be solved by teeing the mbuffer contents and redirecting appropriately:

(reading commands) | mbuffer -p 1 -m 5G | tee <(ffmpeg -i - (splitting video stream here) -f avi | ssh 1.2.3.4 'ffmpeg -i - (doing some encoding here) -f <format> - ') | <( ffmpeg -i (processing audio adequately) )

but this leaves me with two pipes redirected without logical separation. In clear: how do I join both streams again (I need to receive distinct input streams for another command: ffmpeg -i <s -tream1> -i <stream2> (doing final conversion) ? Any chance to do that?

Answer 1

I don't fully understand the command but your description looks like it may be a job for named pipes. To make this concept clear my example uses four of them. With proper substitutions you can reduce this number to two, I think, maybe even to one; but for now let's keep it simple.

mkfifo pre-audio-pipe pre-video-pipe audio-pipe video-pipe # creating pipes
(reading commands) | mbuffer -p 1 -m 5G | tee pre-audio-pipe > pre-video-pipe # splitting

This process will fill whatever buffers are created for the two named pipes, then it will wait for this data to be read elsewhere.

The "elsewhere" is in another console:

<pre-audio-pipe (isolate audio) | (process audio) > audio-pipe

and in yet another console:

<pre-video-pipe (isolate video) | (process video) > video-pipe

Again these two commands will wait until we read some data from the pipes. In the final console:

ffmpeg -i video-pipe -i audio-pipe (doing final conversion)

You may experience a lockdown in case the final command wants to read one stream ahead of the other. I don't know how likely this is. Additional buffers may be useful to avoid this. My first try would be to remove the mbuffer (before tee) and insert two independent buffers between respective (isolate) and (process).

After it's all done:

rm pre-audio-pipe pre-video-pipe audio-pipe video-pipe # cleaning

---

Edit

From the OP's comment:

do you see any chance to implement a solution without using separate named pipes?

I've been thinking about coprocesses (coproc builtin) but I don't know them much. There is this comprehensive answer about them. Search for the phrase "why they are not so popular". From therein:

The only benefit of using coproc is that you don't have to clean up of those named pipes after use.

I totally agree. Look at the example there – it's basically your case with data stream forked three-way instead of two-way. The example uses shells other than bash but from my experience it would be similarly awful in bash.

Ideally, there would be a one-line-command working with unnamed pipes only, since the job should be started with an "economic effort" from the command prompt.

Seriously? With all those (doing some encoding here) expanded? In my opinion, no matter whether you use named or unnamed pipes, an "economic effort" here will be to write a script, even if it's a one-time job. Comparing long one-liner and equivalent well written script, I find the latter easier to debug.

But since you asked for one-liner, you'll get it, still with named pipes though. My idea for maintaining named pipes is to create a temporary directory for them. The general concept:

my_temp=`mktemp -d` ; pre_audio_pipe="${my_temp}/pre-audio-pipe" ; pre_video_pipe="${my_temp}/pre-video-pipe" ; audio_pipe="${my_temp}/audio-pipe" ; video_pipe="${my_temp}/video-pipe" ; mkfifo "$pre_audio_pipe" "$pre_video_pipe" "$audio_pipe" "$video_pipe" ; (reading commands) | tee "$pre_audio_pipe" > "$pre_video_pipe" & <"$pre_audio_pipe" (isolate audio) | mbuffer -p 1 -m 1G | (process audio) > "$audio_pipe" & <"$pre_video_pipe" (isolate video) | mbuffer -p 1 -m 4G | (process video) > "$video_pipe" & ffmpeg -i "$video_pipe" -i "$audio_pipe" (doing final conversion) ; rm -rf "$my_temp"

According to this answer you probably can fit it into one command line, even after you dig into the command and expand all those (do something) placeholders.

OK, the one-liner form was to show you how inconvenient it could be. The same concept as a script:

#!/bin/bash

my_temp=`mktemp -d`
pre_audio_pipe="${my_temp}/pre-audio-pipe"
pre_video_pipe="${my_temp}/pre-video-pipe"
audio_pipe="${my_temp}/audio-pipe"
video_pipe="${my_temp}/video-pipe"

mkfifo "$pre_audio_pipe" "$pre_video_pipe" "$audio_pipe" "$video_pipe" #creating actual pipes

# Main code here.
# Notice we put few commands into the background.
# In this example there are two separate mbuffers.
(reading commands) | tee "$pre_audio_pipe" > "$pre_video_pipe" & # splitting
<"$pre_audio_pipe" (isolate audio) | mbuffer -p 1 -m 1G | (process audio) > "$audio_pipe" &
<"$pre_video_pipe" (isolate video) | mbuffer -p 1 -m 4G | (process video) > "$video_pipe" &
ffmpeg -i "$video_pipe" -i "$audio_pipe" (doing final conversion)

# Then cleaning:
rm -rf "$my_temp"