\documentclass[12pt]{article}
\usepackage{systeme}
\usepackage{amsmath}
\begin{document}
...
\[
\systeme*{(x+y)\sqrt{xy}=504,x^2+6xy+y^2=2016}
\]
...
\end{document}
I'm getting a compiler error and the message is in French... The error seems to occur after I type "6xy" or just "xy" after the "6."
Anyone know what's wrong?

systemeonly supports linear systems. – egreg Mar 27 '16 at 00:21