If I compile the simpler
$\left(a\mathrel{}\middle|\mathrel{}b\right)\showlists$\showlists
I get the following in the log file
\mathinner
.\leftheight=0.00052 depth=0.00256 class=172 "28300
.\mathord
..\fam1 a
.\mathrel
..{}
.\middleheight=0.00682 depth=0.00638 "26A30C
.\mathrel
..{}
.\mathord
..\fam1 b
.\right"29301
which seems OK, but is translated into boxes and glue as
\mathon
\hbox(7.5+2.5)x28.46626, direction TLT
.\hbox(7.5+2.5)x3.8889, direction TLT
..\OT1/cmr/m/n/10 (
.\OML/cmm/m/it/10 a
.\glue(\thickmuskip) 2.77771 plus 2.77771
.\hbox(0.0+0.0)x0.0, direction TLT
.\hbox(7.5+2.5)x2.77779, direction TLT
..\OMS/cmsy/m/n/10 j
.\glue(\thickmuskip) 2.77771 plus 2.77771
.\hbox(0.0+0.0)x0.0, direction TLT
.\glue(\thickmuskip) 2.77771 plus 2.77771
.\OML/cmm/m/it/10 b
.\hbox(7.5+2.5)x3.8889, direction TLT
..\OT1/cmr/m/n/10 )
\mathoff
There is a mysterious box which causes two \thickmuskip glues to be inserted.
Solution:
$\left(a\mathrel{}\middle|\mathopen{}\mathrel{}b\right)$
which works also with pdflatex.

Addendum
As explained by Hans on the thread starting at http://tug.org/pipermail/luatex/2017-November/006642.html at the luatex mailing list, there is a simpler workaround:
\documentclass{article}
\newcommand{\relmiddle}{\Umiddle class 5 }
\begin{document}
$\left(a \relmiddle| b\right)$
$(a\mid b)$
\end{document}
The class numbers to be used are different from the standard math classes in legacy TeX (section 7.1.3.3.1 in the LuaTeX manual):
0 ord
1 op normal
2 op limits
3 op no limits
4 bin
5 rel
6 open
7 close
8 punct
9 inner
10 under
11 over
12 vcenter

We can make it work with the other engines by doing
\documentclass{article}
\ifdefined\Umiddle
\newcommand{\relmiddle}{\Umiddle class 5 }
\else
\newcommand{\relmiddle}[1]{\mathrel{}\middle#1\mathrel{}}
\fi
\begin{document}
$\left(a \relmiddle| b\right)$
$(a\mid b)$
\end{document}
\middle|and\mathrel{}. – egreg Nov 14 '17 at 10:55