Let's play with boxes.
At some point, TikZ introduces the keys behind path and in front of path (to switch back) which basically specifies one or another path box the nodes are placed on. When the path is finally drawn (inside the ; at the end of paths) these boxes are “used”:
- the background one (
\tikz@figbox@bg)
- the actual path gets drawn
- the foreground one (
\tikz@figbox, which is the default one for nodes)
When TikZ finds a node on the path it doesn't actually just draws it, it draws the node on one of these two boxes.
These boxes get reset at the start of the path.
These boxes are the reason that nodes are placed on top of their paths (while they are their own paths to begin with).
With the PGF layers, something very similar happens but instead of just a node everything between \begin{pgfonlayer} and \end{pgfonlayer} will be placed inside that box.
At the end of the picture these boxes get “used”. (There is some additional house-keeping which I believe to be for nesting pictures.)
Now, instead of placing the node on the box that gets used by TikZ in the ; we can try to place it directly on one of the layers.
Since this happens somwhere deep in a few groups, this needs to be global (and indeed \pgfonlayer does it global as well since that can be used inside a scope somewhere – and indeed it will when you use on background layer).
To implement this, I swap out one \setbox (luckily the first one) for a custom \tikz@setbox@which – which is let to \setbox so that it works as it usually would – inside the that macro that places the node on that box for the path.
The node on layer key just makes it so that \tikz@setbox@which does a global \setbox and that the box is one of the layers and not one of the path boxes.
Now, you can say
\begin{scope}[
X/.append style={anchor=east},
LC'/.style={yshift=-2mm}, % no transform canvas
on background layer]
\draw[LC=teal] ([LC'] b1) % ........ ←
to node[X, node on layer=foreground] {ACK(AckNum$=$1000)} ([LC'] a2);
\end{scope}
And this already shows what does not work with this:
The main layer can't be chosen (it's a special one that can't be chosen with \pgfonlayer either).
transform canvas won't work (but it also doesn't with behind path). There may be ways around it but the manual recommends:
In short, you should not use canvas transformations unless you really know what you are doing.
and I'll add … unless you really need it.
I've replaced your addition to LC with a normal transformation inside the LC' key which is added manually to every coordinate. (I do wish there would be some way to “unfix” coordinates/nodes so that they act as normal coordinates and the transformations would apply to them, too, but even the manual only offers an @-ridden workaround.)
Additionally don't use it with something else than nodes. The “decider” macro \tikz@whichbox is not only used for nodes but also for edges, plot marks, matrices, child nodes (not the same as plain nodes) and pics. Using \node on layer there will not work, at best your edges, plot marks, matrix, children and pics simply won't show up. (Because the addition to the box is forgotton after the current path anyway.) More patching has to be done for that.
That said, you could also just place an empty node/coordinate along the path (to save the position and the rotation) and then reference that in placing the actual node at the end of all the line drawing. Though, that will need some nice house-keeping to make it easy to use.
And then there's the whole Deferred Node Positioning where you first “do” a node but place it later (used by forest and the graph drawing library). That isn't very straight forward either, though. (More boxes!)
Code
\documentclass[12pt, tikz, border=3mm]{standalone}
\usetikzlibrary{arrows, arrows.meta, backgrounds, positioning}
\pgfdeclarelayer{foreground}\pgfdeclarelayer{background}
\pgfsetlayers{background,main,foreground}
\makeatletter\ExplSyntaxOn % replace only first one
\tl_replace_once:Nnn \tikz@fig@continue { \setbox } { \tikz@setbox@which }
\ExplSyntaxOff
\let\tikz@setbox@which\setbox
\tikzset{node on layer/.code={%
\expandafter\def\expandafter\tikz@whichbox\expandafter
{\csname pgf@layerbox@#1\endcsname}%
\def\tikz@setbox@which{\global\setbox}}}
\makeatother
\usepackage{amsmath}
\begin{document}
\begin{tikzpicture}[
node distance = 0mm,
LC/.style = {draw=#1, line width=1mm,
arrows={-Stealth[fill=#1,inset=0pt,length=0pt 1.6,angle'=90]}},
X/.style = {draw, very thin, fill=white, fill opacity=0.75,
font=\scriptsize, text=black, text opacity=1, align=left,
inner sep=2pt, sloped, anchor=west,pos=0.07}]
\sffamily\linespread{0.8}
\coordinate (a0) at (0,0);
\coordinate[right=77mm of a0] (b0);
\foreach \i [count=\xi from 0] in {1,2,...,4}
\coordinate[below=7mm of a\xi] (a\i)
coordinate[below=7mm of b\xi] (b\i);
\draw[|->] (a0) -- (a3) node[above left] {$t$};
\draw[|->] (b0) -- (b3) node[above right] {$t$};
\draw[LC=gray] (a1) to node[X] {data\$(\text{SeqNum}=0,\ell=1000)$} (b2);
\begin{scope}[
X/.append style={anchor=east},
LC'/.style={yshift=-2mm}, % no transform canvas
on background layer]
\draw[LC=teal] ([LC'] b1)
to node[X, node on layer=foreground] {ACK(AckNum$=$1000)} ([LC'] a2);
\end{scope}
\end{tikzpicture}
\end{document}
Output

\draw[LC=teal] (b1) to node[yshift=-2mm, node on layer=foreground,X] {ACK(AckNum$=$1000)} (a2);. – Kpym Jun 29 '15 at 16:55