As pointed out correctly by egreg, xdvipdfmx overrides any producer you try to set in the PDF metadata with its own xdvipdfmx (<version number>). However, you can include so-called XMP metadata information in addition to the normal metadata catalog in a PDF document. This offers you the possibility of setting the <pdf:Producer>, which is displayed in Adobe Reader instead of the normal /Producer field if present.
Currently, XMP data can be added to a LaTeX document using one of the packages xmpincl or hyperxmp. While the former is more flexible, it requires you to create an .xmp file yourself. Furthermore, as it only works with pdfTeX, it won't work for your configuration.
So in your case, I would recommend using the hyperxmp package: It takes care of generating the necessary stucture automatically, and it works with a number of TeX engines. As it uses the pdfauthor, pdfproducer, ... fields set with hyperref, simply loading it with \usepackage{hyperxmp} is enough to generate the XMP metadata:
\documentclass{article}
\usepackage{hyperref}
\usepackage{hyperxmp}
\hypersetup{pdfproducer={xelatex}}
\begin{document}
hello
\end{document}
Result:

If you're using XeLaTeX in combination with the memoir class, the question Is there a poisonous interaction between memoir, hyperref, and hyperxmp? might be of interest to you.
However, keep in mind that this only works for XMP-aware programs, other programs will still display xdvipdfmx (<version number>). Furthermore, xdvipdfmx compresses all PDF objects including the XMP metadata, which is fine for Adobe, but may confuse other programs (see the hyperxmp manual for more information).
xdvfile contains the requested "Producer" field. – egreg Jun 08 '11 at 14:53XeTeXproject. – 0 _ Jun 18 '14 at 02:47