Using process substitution to trick programs expecting files, with specific extensions as argument?

Question

Here is my use case: the command line utility melt can accept a file name, with the extension .melt on the command line, and open it; as an example, this is a proper test_p.melt file:

colour:blue
out=24

colour:red
out=48

... which opens and plays with melt test_p.melt.

Now, the thing is that .melt files do not support comments, which I wish they did (you'll get error messages for any line that contains an unparsable argument, including those with, say, a #). So here is a commented test_c.melt file:

# master comment here

colour:blue  # this is blue!
out=24

colour:red
out=48

Opening this in melt directly gives:

$ melt test_c.melt
Failed to load "# master comment here"
...

... and there is no blue screen shown.

So I thought - well, I can put in comments anyway, and then use Bash process substitution to filter the file with sed, and simply provide that to the melt application. First, tried a test with cat, which is successful:

$ cat <(sed 's/#.*$//' test_c.melt)



colour:blue  
out=24

colour:red
out=48

... looks good; but, if I try that with melt, it sees through my trickery:

$ melt <(sed 's/#.*$//' test_c.melt)
Failed to load "/dev/fd/62"
Failed to load "/dev/fd/62"

Basically, melt got the filename of the pipe Bash provided for the process substitution - but unfortunately, what melt does is that it processes argv[i] directly; and in case of a file, it needs to see a .melt extension in the filename; if it doesn't - the process fails.

So my question is: how could I use process substitution - so the filename of the pipe has a specific extension, in this case .melt? Basically, as a result of the substitution, I'd want a pipe filename of /dev/fd/62.melt, which I think will pass.

NB: of course, I can always do:

sed 's/#.*$//' test_c.melt > test_c_temp.melt
melt test_c_temp.melt

... but first, there are two commands here - and I'd want a one-liner pipeline; and for another, it opens up another problem of me thinking about removing temporary files afterwards, which I don't like.

Is this possible with Bash process substitution - or somehow with standard Linux tools?

Answer 1

A possibility would be to point melt to a filesystem that shows modified copies of files. FUSE is a generic way to build filesystem driver implemented by an ordinary program and requiring no privileges. There are many FUSE filesystems around, and there's a good chance that one of them can help you. The idea is to provide a mount point where reading a .melt file reads the “real” file but with comments filtered out.

ScriptFS looks promising (but I've never used it). Something like this should work:

mkdir ~/uncommented-melt
scriptfs -p "$HOME/bin/uncomment-melt;&*.melt" ~/work ~/uncommented-melt

where ~/work is the root of the tree that contains your .melt files and ~/bin/uncomment-melt is

#!/bin/sh
sed 's/#.*$//' "$1"

Then if you have a file ~/work/test_c.melt with comments, you can run melt ~/uncommented-melt/test_c.melt.

Other potential helpful FUSE filesystems:

• Execfuse — lets you build a simple FUSE driver with shell scripts
• AVFS or other FUSE filesystems that transparently uncompress files: define the stripping of comments as an uncompression rule.

Answer 2

With the zsh shell, you can use the =(...) form of process substitution (which uses temporary files instead of /dev/fd/x files and pipes) for which you can specify a suffix:

(TMPSUFFIX=.melt; melt =(sed 's/#.*$//' test_c.melt))

See info zsh TMPSUFFIX (assuming the info pages are installed, you may need to install a zsh-doc package) for details.

Answer 3

Ok, it turns out that in my specific case, these kind of melt scripts are to be interpreted strictly as commands line arguments; so just the one sed in the OP doesn't quite cut it (plus, there are other things like profiles that can be set). So here is what I ended up doing - can probably serve as inspiration in other cases which the title of the OP would cover.

I eventually settled on using this: make a test.shmelt file, which is actually a bash script that contains my commented melt script code; make this file executable chmod +x test.shmelt; then after editing the script, run it as ./test.shmelt.

Upon running, it will create a "cleaned up" test.melt file in /tmp, and call melt on this file instead. Since melt usually keeps running in terminal after the end of its programme, with a trap on SIGINT this temporary file can be cleaned up when Ctrl-C is pressed (but doesn't have to).

In that way, I still have comments; can edit quickly in the source file and run melt and see results; and have a file "cleaned up" of comments too, that I can use later.

Here is the code of test.shmelt:

#!/bin/bash

# call with: ./test.shmelt


#TMLTFILE="test.melt" # for final export
TMLTFILE="${0%%.shmelt}.melt" # for final export

function finished() { echo ; } ; # use this when testing or montaging to keep exported (tmp) .melt file; uncomment the below to remove the tmp file
#~ function finished() { rm /tmp/"$TMLTFILE"; echo "fin1 $0" ; } ; trap finished SIGINT ;

echo 'Remember `pulseaudio --start` to hear audio with `melt`!'
pulseaudio --check
if [ $? -ne 0 ]; then
    pulseaudio --start
fi

DIRNAME=$(readlink -f $(dirname $0))
PROFILE="square_ntsc"


echo "
# the profile doesn't work from here;
# still has to be specified on command line
# but including it here so the path is saved for testing
#~ -profile
#~ ${PROFILE}

-video-track

  # can avoid pixmap: here, but that s the producer;
  # qimage: also works
  # NB it is NOT '-out 1645'; but 'out=1645'!!
  /media/myimg/%05d.bmp
  in=0
  out=1645
  #~ length=1645
  #~ loop=0
  #~ eof=stop

-audio-track

  /media/mysnd/snd.wav
  in=0
  out=1645
  #~ loop=0
  #~ eof=stop

#-consumer xml # doesn't work here

" | sed -e 's/#.*$//' -e 's/^[[:space:]]*//' -e '/^[[:space:]]*$/d' -e 's/[[:blank:]]*$//' > ${TMLTFILE}

# the sed: remove comments; remove indents; remove empty lines; remove spaces at end of line


# eof: one of: stop, loop, continue or pause (eof=stop)
# to loop, use `melt eof=loop ...` (but make sure first,
#  that the edit as a whole stops - use xml to check!)
# due to caching issues, preview pieces (up to ~300 frames) like this:
melt eof=loop -profile ${PROFILE} ${TMLTFILE} in=200 out=400

# must have profile here, for checking w/ -consumer xml (else segfault)
# this command may add additional producers to the xml!:
## melt -profile ${PROFILE} ${TMLTFILE} -consumer xml

# use this to generate xml if needed:
#melt -profile ${PROFILE} $(cat ${TMLTFILE} | tr '\n' ' ') -consumer xml

# if exited normally, cleanup:
finished

Answer 4

It's been a while, but I'll try to answer the title question in a way that would've been useful to me.

This is related: Why does BASH process substitution not work with some commands?

So the specific problem I was trying to solve was this:

1. I have an image converter;
2. it converts from [insert random format] to simple formats like BMP/PNM/TGA;
3. I want to convert to JPEG, so need to chain it with ImageMagick convert;
4. I don't want to use temporary files.

So, unfortunately, the tool doesn't like to output to stdout/fds/pipes because it derives the output format from the output filename's extension. So how to trick it?

Create a symbolic link with the appropriate extension to /dev/fd/N and use that as the "temporary file".

Example:

ln -s /dev/fd/4 temp.pnm
[TOOL] -i [INPUTFILE] -o temp.pnm 4>&1 | convert - [OUTPUT.JPG]