This is not a solution, but a workaround.
It seems that what contour does is somewhat exclusive to pdf, and apparently XeTeX does not support that. e.g. this, this, and this.
Apparently the way to make contours in XeTeX is to use low-level pdf commands, like the ones showed in the answers linked above (P.S.: At the time I was writing this, remco proved otherwise. Anyway...).
I merged the solutions to a few other questions and made a macro that simulates the \contour:
\fillstroke{<fill color>}{<contour color>}{<stroke width>}{<text>}
where <fill color> and <contour color> are valid color names, <stroke width> is a number, NOT a dimension, so no units allowed, and <text> is the text (wow).
Here are the macros:
\usepackage{xcolor}
\def\rgbtoarray#1,#2,#3\null{[#1 #2 #3]}
\def\csvtoarray#1{%
\expandafter\rgbtoarray#1\null%
}
\newcommand{\extractrgb}[2]{%
\extractcolorspecs{#1}{\model}{\mycolor}%
\convertcolorspec{\model}{\mycolor}{rgb}{\printcol}%
\edef#2{\printcol}%
}
\newcommand*{\fillstroke}[4]{%
\extractrgb{#1}{\colorvector}%
\extractrgb{#2}{\strokevector}%
\special{pdf:bcolor \csvtoarray{\colorvector} \csvtoarray{\strokevector}}%
\special{pdf:literal direct #3 w 2 Tr}%
#4%
\special{pdf:ecolor}%
\special{pdf:literal direct 0 Tr}%
}
The only required package is xcolor.
And here is your MWE:
\documentclass[border=3mm]{standalone}
\usepackage{xcolor}
\pagecolor{blue!30}
\def\rgbtoarray#1,#2,#3\null{[#1 #2 #3]}
\def\csvtoarray#1{%
\expandafter\rgbtoarray#1\null%
}
\newcommand{\extractrgb}[2]{%
\extractcolorspecs{#1}{\model}{\mycolor}%
\convertcolorspec{\model}{\mycolor}{rgb}{\printcol}%
\edef#2{\printcol}%
}
\newcommand*{\fillstroke}[4]{%
\extractrgb{#1}{\colorvector}%
\extractrgb{#2}{\strokevector}%
\special{pdf:bcolor \csvtoarray{\colorvector} \csvtoarray{\strokevector}}%
\special{pdf:literal direct #3 w 2 Tr}%
#4%
\special{pdf:ecolor}%
\special{pdf:literal direct 0 Tr}%
}
\begin{document}
\fillstroke{black}{yellow}{0.5}{\Huge This is a sample text}\par
\end{document}

contourmethod overlaps two slightly differently-sized copies of the text (with\rlap) and the PDF literal method works directly on the glyphs. (I did an experiment at https://tex.stackexchange.com/questions/464402/xelatex-and-black-outline-characters-in-devanagari/516401#516401). – Cicada Nov 17 '19 at 10:03