After Internal error: bad native font flag I stumbled upon another weird problem with the combination XeLaTeX/fontspec/Libertine/newtxmath:
\documentclass{scrartcl}
\usepackage[no-math]{fontspec}
\usepackage{libertineotf}
\usepackage{amsmath}
\usepackage[libertine]{newtxmath}
\begin{document}
\begin{equation}
( \frac{1}{1} )
\left( \frac{1}{2} \right)
\bigl( \frac{1}{3} \bigr)
\Bigl( \frac{1}{4} \Bigr)
\biggl( \frac{1}{5} \biggr)
\Biggl( \frac{1}{6} \Biggr)
\end{equation}
\end{document}
The up-scaled result is:
http://img225.imageshack.us/img225/5576/bracketsv.png
It seems to me that the Libertine font is used for ( and and CM for ).
Compare the closing parentheses from the above image to the following where the fontspec package was not used:
http://img94.imageshack.us/img94/2015/brackets2.png
The parentheses have the correct size but again CM is used for the numbers.
pdfLaTeX and the libertine package produce the same output as XeLaTeX without fontspec.
newtxmathpackage was updated significantly just a few days ago. Are you running the latest version (0.98, I believe)? – Mico Jul 15 '12 at 18:36newtxmathbeforefontspecseems to solve the issue. You should also loadamsmathbeforenewtxmath– egreg Jul 15 '12 at 21:15newtxmathbeforefontspecresults in the same output: correct sized parentheses (but as it seems from CM) and numbers in CM. – Qrrbrbirlbel Jul 16 '12 at 00:28\left((but not with\left[, for instance). I've tried other fonts andnewtxmathis used for the left parenthesis in those cases. – egreg Jul 16 '12 at 08:02