How to pipe a man page (without pagination) while intentionally preserving formatting?

Question

Minimal example:

man git | cat

Real example:

man git | grep --color=always -C 3 "pathspec"

FWIW, I've tried --pager="cat" using various pagers, as well as piping to various pagers. I've also tried using vimcat with the same, but unfortunately it freezes. I've even tried using unbuffered. All to no avail.

Perhaps it's possible using -T or {g,n,t}roff, but I'm not sure how? Is there a way I can pipe to less/vim/etc and have it pass through to stdout with formatting (colors/etc) and without paginating?

---

---

---

Edit: Thanks to Stéphane's help below, I now have a beautiful way of easily searching through (and highlighting!) all matched man pages. This is a game changer for me. Thank you, Stéphane!

For those who might be interested, here is the key part of the script (no doubt it can be improved; feedback is always welcome):

Edit: I've now incorporated Stéphane's additional feedback. Huge improvement!

Edit: I've further improved this solution to work with both Ubuntu 20.04.4 LTS and macOS 12.6, by instead piping to ul.

#!/bin/bash
arg_search="$1"
typeset -A seen=()
while IFS= read -ru3 filename; do
    if (( ! seen[$filename]++ )); then
        # 'man -awK' sometimes returns bogus results due to searching through
        # everything, including comments in the man source file, so filter
        # these out. Note that this creates a secondary problem: legitimate
        # results may get filtered out when the search term is two or more
        # words spanning across lines. This can be solved with a multiline
        # regex using pcre2grep -M 'search\s+term, but this is perhaps
        # growing beyond the scope of things here.
        { man --no-hyphenation --no-justification --regex -- "$filename" | 

            grep -iqE -- "$arg_search"; } || continue
    # Print filename. Fill terminal width with highlighted background color.
    filename_bar="$(printf '%s\t' "$filename" | expand -t $(tput cols))"
    printf '\e[1;103;30m%s\e[0m\n' "$filename_bar"
    # Per Stéphane's feedback, here's a cleaner version of the same for zsh:
    # psvar=${(mr[COLUMNS])filename} print -P '%K{yellow}%F{black}%1v%k%f'

    MAN_KEEP_FORMATTING=1 man \
        --no-hyphenation --no-justification --regex -- "$filename" | \
        ul | \
        grep -iE --color=always -- "$arg_search"
fi
# 'unbuffer' seems to be required for macOS in order to unbuffer the output,
# as 'stdbuf -oL -eL' doesn't seem to work. Install on macOS using
# 'brew install expect' [sic], which contains unbuffer. This otherwise
# doesn't hurt anything on Ubuntu 20.04.4 LTS. Unbuffering the output of
# 'man -awK' is important because it can take a long time to run.

done 3< <(unbuffer man -awK --regex -- "${arg_search}")

Answer 1

Try:

GROFF_SGR=1 man -P 'grep --color=always -C 3 pathspec'  git

If piping to grep, you'll likely want GROFF_SGR=1 (needed on Debian-based systems at least) for bold/underline to be implemented with ANSI SGR escape sequences (like \e[1mBOLD\e[m) instead of the traditional B\bBO\bOL\bLD\bD which would make it difficult to grep. See Grep: unexpected results when searching for words in heading from man page for details.

Note that the pager (here grep) is only run if man's stdout goes to a terminal. Piping to a command such as less would disable both that grep and the formatting.

Setting MAN_KEEP_FORMATTING used to disable SGR in my tests when I answered Grep: unexpected results when searching for words in heading from man page back then, but it seems to no longer be the case (at least not on Ubuntu 20.04).

So, to use a pager, you could either do

GROFF_SGR=1 MAN_KEEP_FORMATTING=1 man git |
  grep --color=always -C 3 pathspec | less -R

Though as it's not started by man, less is not configured to display a relevant footer line, or you could pass an inline shell script to -P that calls grep and the pager.

For instance, with zsh:

mangrep()
  GROFF_SGR=1 CODE="
     grep --color=always -C3 ${(j[ ])${(qq)@[2,-1]}} | pager
  " man -P 'sh -c "eval \"$CODE\""' $1

And then for instance:

mangrep git -i pathspecs

If your pager is less, as it's started by man, it will inherit the LESS* environment variables that man sets which look like:

LESS=-ix8RmPm Manual page git(1) ?ltline %lt?L/%L.:byte %bB?s/%s..?e (END):?pB %pB\%.. (press h for help or q to quit)$PM Manual page git(1) ?ltline %lt?L/%L.:byte %bB?s/%s..?e (END):?pB %pB\%.. (press h for help or q to quit)$
LESSCHARSET=utf-8

A bash (4.4+) equivalent could look like:

mangrep() {
  local page=$1 IFS=' '
  shift
  GROFF_SGR=1 CODE="
    grep --color=always -C3 ${*@Q} | pager
  " man -P 'bash -c "eval \"$CODE\""' ${page:+"$page"}
}

(using "$*" and IFS=' ' instead of j[ ] to join with space, @Q (which requires bash for decoding) instead of (qq) (which does sh-compatible quoting), using shift and saving the first argument in $page as @Q can't be combined with array ranges).

To allow options for man (in zsh):

mangrep() {
  local page=$@[(i)[^-]*]
  GROFF_SGR=1 CODE="
     grep --color=always -C3 ${(j[ ])${(qq)@[page+1,-1]}} | pager
  " man -P 'sh -c "eval \"$CODE\""' "$@[1,page]"
}

And then for instance:

mangrep -a printf -i precision

To look for precision case insensitively (-i passed to grep) in all printf man pages (-a passed to man).

Make it a script instead of a function:

#! /bin/zsh -
page=$@[(i)[^-]*]
GROFF_SGR=1 CODE="
  grep --color=always -C3 ${(j[ ])${(qq)@[page+1,-1]}} | pager
  " exec man -P 'sh -c "eval \"$CODE\""' "$@[1,page]"

And then you'll be able to use from any shell (or non-shell), not just zsh.

---

Some comments on the code you posted in your edit:

[...]

man -w -a -K "$arg_search" | \

Should be man -w -a -K -- "$arg_search" or you won't be able to search for things that start with - (other than using things like arg_search='[-]-foo' as a work around).

    stdbuf -oL -eL uniq | \

uniq only removes duplicate if they are consecutive

    xargs -d $'\n' -n 1 -I {} env bash -c "

-I and -n 1 are redundant. Also, a bash inline script can take more than one argument, no need to call bash for each.

        $(
        cat <<EOF

As EOF is not quoted, expansions are being performed in the here doc and that's passed as the shell code argument. That is bad practice, that's a typical code injection vulnerability. The external data should be passed either as arguments or env variables instead.

            filename='{}'

Same thing, the expansion (by xargs) of {} ends up in the shell code.

            filename_length="\${#filename}"
            terminal_width="$(tput cols)"
            delta_length="\$(("\$terminal_width" - "\$filename_length"))"
        # Print filename, and fill width with highlighted background color.
        padding="\$(seq -s' ' "\$(("\$delta_length" + 1))" | \
            tr -d '[:digit:]')"
        printf '\e[1;103;30m%s\e[0m' "\$filename\$padding"

Quite convoluted. I using zsh, that'd just be:

psvar=${(mr[COLUMNS])filename} print -P '%K{yellow}%F{black}%1v%k%f'

With other shells, you can also do right padding with of a string with:

printf '%s\t' "$string" | expand -t "$(tput cols)"

Which at least on BSDs and like zsh's right padding expansion flag when used with the m flag takes into account the display width of each character (escape sequences not supported though, so the colouring ones must not be included in the text sent to expand).

            GROFF_SGR=1 man --no-hyphenation --no-justification \
                "\$filename" | grep --color=always "$arg_search"

[...]

I suppose you have MAN_KEEP_FORMATTING defined? Without it, the formatting should be lost when piping to grep.

Also note that man -K does a fixed-string search and an extended regexp search with --regex and in either case is case insensitive, so you'd need the -i option and either add --regex to man and -E to grep or add -F to grep for the search logic to match between the two.

So, here, I'd rather do:

#! /bin/zsh -
ere=${1-GROFF_NO_SGR}
typeset -A seen=()
while IFS= read -ru3 filename; do
  if (( ! seen[\$filename]++ )); then
    psvar=${(mr[COLUMNS])filename} print -P '%K{yellow}%F{black}%1v%k%f'
    GROFF_SGR=1 MAN_KEEP_FORMATTING=1 man --no-hyphenation \
      --no-justification -l "$filename" |
      grep -iF --color=always -C4 -- "$ere"
  fi
done 3< <(man -awK -- "$ere")

Answer 2

The only method I've discovered that works with both macOS 12.6 and Ubuntu 20.04.4 LTS is quite simply the following:

MAN_KEEP_FORMATTING=1 man git | ul

(I'm leaving Stéphane Chazelas' answer as the accepted answer, since it is far more detailed, and as a thanks for lending his substantial time and expertise.)