The source of the problem seems to be PGF's deferred node positioning (PGF manual 102.2.3) using \pgfpositionnodelater and \pgfpositionnodenow, as shown by the example below. (Forest uses deferred node positioning for placing all nodes!)
Run the example using pdflatex and xelatex. In the example, two nodes (A and B) are created in a deferred fashion within the same picture. pgfpositionnodenow positions them at different points, however. The example is set up so that it is clear that PdfLaTeX yields the correct result (0,0) for node A in the lower left corner of tha page. XeLaTeX incorrectly yields the same result for both nodes (I guess because \pgfpositionnodenow is used at the same current point, but I can't figure out the meaning of the resulting dimensions).
I hope someone can shed some light on how to resolve this problem. I'm afraid I'm no good at such low level stuff.
\documentclass{article}
\usepackage[margin=0pt]{geometry}
\parindent 0pt
\usepackage{tikz}
\usepackage{etoolbox}
\newbox\alaterbox
\newbox\blaterbox
\def\asavelaternode{%
\global\setbox\alaterbox=\box\pgfpositionnodelaterbox
\global\let\alatername\pgfpositionnodelatername
\global\let\alaterminx\pgfpositionnodelaterminx
\global\let\alaterminy\pgfpositionnodelaterminy
\global\let\alatermaxx\pgfpositionnodelatermaxx
\global\let\alatermaxy\pgfpositionnodelatermaxy
}
\def\bsavelaternode{%
\global\setbox\blaterbox=\box\pgfpositionnodelaterbox
\global\let\blatername\pgfpositionnodelatername
\global\let\blaterminx\pgfpositionnodelaterminx
\global\let\blaterminy\pgfpositionnodelaterminy
\global\let\blatermaxx\pgfpositionnodelatermaxx
\global\let\blatermaxy\pgfpositionnodelatermaxy
}
\def\arestorelaternode{%
\setbox\pgfpositionnodelaterbox=\box\alaterbox
\let\pgfpositionnodelatername\alatername
\let\pgfpositionnodelaterminx\alaterminx
\let\pgfpositionnodelaterminy\alaterminy
\let\pgfpositionnodelatermaxx\alatermaxx
\let\pgfpositionnodelatermaxy\alatermaxy
}
\def\brestorelaternode{%
\setbox\pgfpositionnodelaterbox=\box\blaterbox
\let\pgfpositionnodelatername\blatername
\let\pgfpositionnodelaterminx\blaterminx
\let\pgfpositionnodelaterminy\blaterminy
\let\pgfpositionnodelatermaxx\blatermaxx
\let\pgfpositionnodelatermaxy\blatermaxy
}
\gdef\pgfsyspdfmark#1#2#3{\csgdef{#1}{(#2 sp, #3 sp)}}
\begin{document}
\makeatletter
\tikzset{every node/.style={inner sep=0, outer sep=0, anchor=north west}}%
\tikz{%
\pgfpositionnodelater{\asavelaternode}\node(a){\pgfsys@markposition{posa}A};
\pgfpositionnodelater{\bsavelaternode}\node(b){\pgfsys@markposition{posb}B};
}%
\vfill
A: \csuse{posa}
B: \csuse{posb}
\tikz{%
\arestorelaternode\pgfpositionnodenow{\pgfqpoint{1cm}{2cm}}%
\brestorelaternode\pgfpositionnodenow{\pgfqpoint{5cm}{4cm}}%
}%
\end{document}
PdfLaTeX result:
A: (0 sp, 0 sp)
B: (7458720 sp, 3729360 sp)
XeLaTeX result:
A: (-1864679 sp, -3281531 sp)
B: (-1864679 sp, -3281531 sp)