0

The question here is the bold sentence below. Suppose I'm trying to do this:

diff -u <(some command) {mystery-syntax}

I am diffing the output of some command and some data that comes from standard input—for instance, a copy-and-paste in the terminal. Like this:

$ diff -u <(echo foo) {mystery-syntax}
bar
[Ctrl-D]
--- /dev/fd/63  2021-11-04 11:28:19.360366909 -0700
+++ /dev/fd/62  2021-11-04 11:28:19.360366909 -0700
@@ -1 +1 @@
-foo
+bar

What actual syntax is the {mystery-syntax} meta-token covering?

The diff utility doesn't use the - convention for specifying standard input, so we cannot use that.

One answer is: /dev/fd/0.

$ diff -u <(echo foo) /dev/fd/0
bar
[Ctrl-D]
--- /dev/fd/63  2021-11-04 11:31:02.837040697 -0700
+++ /dev/fd/0   2021-11-04 11:30:56.807964644 -0700
@@ -1 +1 @@
-foo
+bar

OK, so we have it working in one way, on systems that have Linux-style /dev/fd. But is there a way without referencing the /dev filesystem, which is a system-specific implementation detail? I believe Bash process substitution is supported on systems which have mechanisms other than /dev/fd. The manual states that:

Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files.

For instance, the obvious:

diff <(echo foo) <(cat)

doesn't work; for whatever reason, even though the <(cat) process substitution needs only the output side of cat, cat is nevertheless being invoked in such a way that it has no standard input:

$ diff -u <(echo foo) <(cat)
cat: -: Input/output error
--- /dev/fd/63  2021-11-04 11:33:59.303347814 -0700
+++ /dev/fd/62  2021-11-04 11:33:59.303347814 -0700  
@@ -1 +0,0 @@
-foo

There are some "clever" behaviors here, too. If we explicitly redirect /dev/fd/0 into cat's input, that is still borked somehow:

$ diff -u <(echo foo) <(cat < /dev/fd/0)
cat: -: Input/output error

But! If we give cat it a here-document as its standard input, there is no interference:

$ diff -u <(echo foo) <(cat <<<"bar")
--- /dev/fd/63    2021-11-04 11:40:03.612616179 -0700
+++ /dev/fd/62    2021-11-04 11:40:03.612616179 -0700
@@ -1 +1 @@
-foo
+bar
Kaz
  • 8,273

2 Answers2

1

If one of the filenames given to diff on the command line is - (a dash), the utility will read from standard input in its place.

$ echo hello | diff -u <( echo ok ) -
--- /dev/fd/12  Fri Nov 12 21:25:18 2021
+++ -   Fri Nov 12 21:25:18 2021
@@ -1 +1 @@
-ok
+hello

From the GNU diff manual:

If a FILE is -, read standard input.

From the OpenBSD manual:

If either file1 or file2 is -, the standard input is used in its place.

From the POSIX standard:

file1, file2
A pathname of a file to be compared. If either the file1 or file2 operand is -, the standard input shall be used in its place.

Kusalananda
  • 333,661
  • Thanks for the quick response! As you can see, there is a mistaken claim in my question; somehow I came to the conclusion that "the diff utility doesn't use the - convention", but, prompted by your answer, I immediately see that it's actually working fine, in either argument position, opposite to a process-substituted operand. (There is a bug that diff - - assumes that the two arguments are the same object and doesn't read from standard input twice; but that is neither here nor there with regard to this Q&A.) – Kaz Nov 12 '21 at 20:27
  • Still, I'd like to know the answer for situations when the - convention really isn't supported. – Kaz Nov 12 '21 at 20:27
  • @Kaz Let us know what system you are on where using diff with - does not work. It's a convention required by the POSIX standard. – Kusalananda Nov 12 '21 at 20:28
  • I'm not sure what I tried; I might have just tried the diff - - case and saw that it returns without reading anything. (Why I don't remember is that I wrote this question maybe two weeks ago, and just noticed today that I hadn't finished submitting it, due to getting a warning while closing numerous browser tabs.) So I thought, oh, it's just treating - as a file name, and optimizing the case of a file being diffed with itself. – Kaz Nov 12 '21 at 20:29
  • Ah, OK, I remember now. I was actually trying to diff two units of data, both coming from TTY input. – Kaz Nov 12 '21 at 20:34
  • @Kaz If you're on Linux, try diff -s - - and you will see what happens (in short, you are comparing the same data against itself, so there should not be any output). – Kusalananda Nov 12 '21 at 20:34
  • @Kaz Consider updating your question if you need to clarify it. – Kusalananda Nov 12 '21 at 20:35
  • No, it's pretty clear: how do we feed standard input as a command line argument that a process can open: non-portably in the sense of relying on Bash, but otherwise portably in the sense of simply relying on some /dev/fd kernel-specific thing. – Kaz Nov 12 '21 at 21:39
1

diff -u <(echo foo) <(cat) doesn't work in an interactive bash because commands in process substitution are not put in the foreground so can't read from the terminal device. Like in cat & where cat is also run in background (not in the foreground process group of the terminal device), the process running cat is suspended with a SIGTTIN signal as soon as it tries to read from its controlling terminal.

echo hello | diff -u <(echo ok) <(cat) works because the whole pipeline made of two subshells is in foreground, and anyway cat is reading from the pipe, not the terminal. The process substitutions end up being in foreground because they were started by the subshell (the second component of that pipeline) that is in foreground.

bash-5.0$ diff -u <(echo ok) <(ps -o pid,ppid,pgid,args)
--- /dev/fd/63  2021-11-12 22:39:41.985207894 +0000
+++ /dev/fd/62  2021-11-12 22:39:41.985207894 +0000
@@ -1 +1,6 @@
-ok
+    PID    PPID    PGID COMMAND
+   9779    9772    9779 /bin/zsh
+1957388    9779 1957388 bash --norc
+1958861 1957388 1957388 bash --norc
+1958862 1957388 1958862 diff -u /dev/fd/63 /dev/fd/62
+1958863 1958861 1957388 ps -o pid,ppid,pgid,args

--- /dev/fd/63  2021-11-12 22:39:01.017237420 +0000
+++ /dev/fd/62  2021-11-12 22:39:01.017237420 +0000
@@ -1 +1,7 @@
-ok
+    PID    PPID    PGID COMMAND
+   9779    9772    9779 /bin/zsh
+1957388    9779 1957388 bash --norc
+1958325 1957388 1958324 diff -u /dev/fd/63 /dev/fd/62
+1958326 1958325 1958324 [bash] <defunct>
+1958327 1958325 1958324 bash --norc
+1958328 1958327 1958324 ps -o pid,ppid,pgid,args

See how ps is running in a different process group in the first case above, and in the same process group as the rest in the second case.

If you run:

bash-5.0$ (diff -u <(echo ok) <(ps -o pid,ppid,pgid,args))
--- /dev/fd/63  2021-11-12 22:42:04.761112701 +0000
+++ /dev/fd/62  2021-11-12 22:42:04.761112701 +0000
@@ -1 +1,6 @@
-ok
+    PID    PPID    PGID COMMAND
+1960605    9772 1960605 /bin/zsh
+1960612 1960605 1960612 bash --norc
+1960757 1960612 1960757 diff -u /dev/fd/63 /dev/fd/62
+1960759 1960757 1960757 bash --norc
+1960760 1960759 1960757 ps -o pid,ppid,pgid,args

It's back to being OK as then we start a whole subshell in foreground.

So the easy work around here is to run the pipeline in a subshell.

In any case, again, that only applies to interactive shells (where terminal job control is being performed) and when stdin is the controlling terminal of the session.

In scripts (and you generally only care about portability in scripts), there's no problem with diff -u <(echo ok) <(cat) even when stdin is a terminal as the shell is non-interactive and doesn't do job control.

That cat however does nothing useful, it's the same UUOC as in:

cat | diff -u <(echo ok) -

Or

cat | diff -u <(echo ok) /dev/stdin

On systems that don't support /dev/fd/x / /dev/stdin¹, for commands which unlike diff, don't support - to specify stdin, <(cat) in bash or zsh (but not in ksh versions prior to ksh93u where that feature comes from) uses a temporary named pipe instead, so diff -u <(echo ok) <(cat) will still work on those systems.

Since we mention portability though, note that in zsh, you need:

echo hello | { diff <(echo ok) <(cat); }

For the stdin of cat to also be the pipe from echo.

Also note that:

echo hello | cmd /dev/stdin # or /dev/fd/0, /proc/self/fd/0²

Doesn't work with ksh93 on Linux, because there, ksh93's | uses socketpairs instead of pipes³ and on Linux (and Cygwin), contrary to other systems, opening /dev/stdin is not like dup(0), it opens the same file as open on stdin, and you can't open() a socket.

More reading in connected Q&As here:

¹ very few these days and there are probably far more systems without bash than without /dev/fd/x

² except for those commands such as gawk that treat or can treat /dev/stdin as meaning stdin by themselves without actually opening the file in the same was as for -.

³ as Linux' pipe implementation doesn't let you peek at the contents which ksh93 needs to perform some (dubious) optimisations.

Stéphane Chazelas
  • 544,893

  • are not put in the foreground so can't read from the terminal. Ah, is that what is is? That makes sense because if two or more processes were reading from the TTY, there is no order to it. Simply put scheduler doesn't know that one of the processes is a left argument in a command line and so its TTY read calls should all wake up first.

     – Kaz Nov 13 '21 at 01:01
  • The above observation indicates that the problem is not solvable with processes; the only way Bash could provide platform-independent access to stdin as a file would be to provide some piece of syntax that acts as a macro which expands to /dev/fd/0 or whatever else. But "whatever else" could not be something that connects to a process, unless it is a special process dedicated to providing the feature: e.g. some "octopus" with multiple pipes which itself reads stdin sequentially and serves it to the pipes in order. Whenever it gets EOF, it closes the current pipe and goes to the next. – Kaz Nov 13 '21 at 01:09
  • So that is to say, given multiple instances of this magic syntax on a platform without /dev/fd, Bash could create a named FIFO for each instance of the syntax, and then have a process which coordinates between standard input and each one of those FIFO's in turn. That process is inserted into the foreground process group so it can have TTY access. – Kaz Nov 13 '21 at 01:14
  • @Kaz, see edit for clarifications. – Stéphane Chazelas Nov 13 '21 at 08:00
  • "there are probably far more systems without bash than without /dev/fd/x" well FreeBSD (in the default setup) has /dev/fd/x only for x = 0..2 which is not enough to implement process substitution. –  Nov 13 '21 at 08:12
  • @UncleBilly true. But it's enough here for a portable bash cmd <(foo) /dev/stdin that avoids the UUOC. – Stéphane Chazelas Nov 13 '21 at 08:17