DeclareMathOperator does not respect math font

Question

I use the Libertine font for text and math. If I declare a new operator with \DeclareMathOperator*, the font used to typeset the operator is not Libertine. For example:

\documentclass{article}
\usepackage{libertine}
\usepackage[libertine]{newtxmath}
\usepackage{amsmath}
\DeclareMathOperator*{\argmin}{arg,min}
\begin{document}
This is some text.
\begin{equation}
  x = y + \sum_{z \in \mathcal{Z}} \beta_z
\end{equation}
\begin{equation}
  x = \argmin_{y \in Y} \bigg { f(y) + \frac{1}{g(y) - h(y)} \bigg}
\end{equation}
\end{document}

Produces:

Where \argmin is not in Libertine font.

I also noticed that commenting out \usepackage{libertine} gives me \argmin in the correct font but, of course, normal text is now not in Libertine any more:

EDIT: I just noticed this is a XeTeX-specific problem. If I compile with pdflatex instead of xelatex, the problem disappears. In the project I am working on, however, I need XeTex, so just switching to pdflatex is not an option for now.

EDIT2: I tried the suggestions I received in an answer.

• Using \usepackage[type1]{libertine} and then \usepackage[libertine]{newtxmath} again gives me normal text in Computer Modern, and not in Libertine. The math operator, though, is now in Libertine.

• Using \usepackage{libertinus} before \usepackage{amsmath}, both with and without \usepackage[libertine]{newtxmath}, gives a compilation error: ! LaTeX Error: Command \dddot already defined.

• Inverting the order of package loads fixes the error. I mean that I load first amsmath and then libertinus, without loading newtxmath. In this case, I get the correct result for \argmin, but the rest of the math looks funny. See, for example, \mathcal{Z} or the curly braces in the screenshot below.

• Loading amsmath first, libertinus second, and newtxmath third, gives another compilation error: ! LaTeX Error: Command '\arrowvert' already defined..

• The last permutation, which is 1st amsmath, 2nd newtxmath, 3rd libertinus, gives one more compilation error: ! LaTeX3 Error: Control sequence \not= already defined..

EDIT3: Ok, I have a finally working version thanks to the answer below:

\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage[type1]{libertine}
\usepackage[libertine]{newtxmath}

Answer 1

Using T1-encoding with xelatex is not a good idea. See e.g. https://tex.stackexchange.com/a/470987/2388.

Changing the order works for me:

\documentclass{article}

\usepackage{amsmath}
\usepackage[libertine]{newtxmath}
\usepackage{libertine}

\DeclareMathOperator*{\argmin}{arg\,min}

\begin{document}

This is some text. arg min

\begin{equation*}
  x = y + \sum_{z \in \mathcal{Z}} \beta_z
\end{equation*}

\begin{equation*}
  x = \argmin_{y \in Y} \bigg \{ f(y) + \frac{1}{g(y) - h(y)} \bigg\}
\end{equation*}

\end{document}

If you get an error, show your log-file.

Answer 2

Edit

Ulrike Fischer posted a simpler and workaround than I came up with, but I’ll leave this up here in order to discuss some of the alternatives and problems you ran into.

The reason this happens when you compile with XeLaTeX but not PDFLaTeX is that the libertine package loads the OpenType version of the font on XeTeX. There are clashes between how the packages you load define the operator font. A mismatch between the font encoding and the font name causes XeTeX to load the default font instead. It turns out that loading the packages in the right order fixes this.

However, there are a few alternatives to consider. If you load mathastext after libertine, you get the OpenType version of Libertine rather than jury-rigged Type 1 fonts:

\usepackage{amsmath}
\DeclareMathOperator*{\argmin}{arg\,min}

\usepackage[libertine]{newtxmath}
\usepackage{libertine}
\usepackage{mathastext}

If you check the output with pdffonts, this removes half the fonts used, many of which are 8-bit, and allows you to copy and paste more of the letters.

I also suggested libertinus, which loads unicode-math. You write, “but the rest of the math looks funny. See, for example, \mathcal{Z} or the curly braces[....]”

That’s just how the Libertinus Math font looks, so it’s what you get when you do things the easy way. But, if you don’t like one of its alphabets, you can mix and match. If you want to change the \mathcal alphabet to something more like what you had before, you can do that by adding, after libertinus,

\setmathfont[range={cal,bfcal}, Scale=MatchUppercase]{Latin Modern Math}

Or you might prefer

\setmathfont[range={cal,bfcal}, Scale=MatchUppercase, StylisticSet=1]{XITS Math}

If you want to match the appearance of the original more closely, using only OpenType fonts, you want to load your alphabets individually instead of by package.

\documentclass{article}

\usepackage{amsmath}
\DeclareMathOperator*{\argmin}{arg\,min}

\usepackage{unicode-math}
\defaultfontfeatures{Scale = MatchLowercase}
\setmainfont[Ligatures={Common, Discretionary, TeX}, Scale=1.0]{Libertinus Serif}
\setsansfont{Libertinus Sans}
\setmonofont{Libertinus Mono}
\setmathfont{TeX Gyre Termes Math}
\setmathfont[range={up,it,bfup,bfit,sfup,sfit,bfsfup,bfsfit,tt}]{Libertinus Math}
\setmathfont[range={cal,bfcal}, Scale=MatchUppercase]{Latin Modern Math}

\begin{document}

This is some text.

\begin{equation*}
  x = y + \sum_{z \in \mathcal{Z}} \beta_z
\end{equation*}

\begin{equation*}
  x = \argmin_{y \in Y} \bigg \{ f(y) + \frac{1}{g(y) - h(y)} \bigg\}
\end{equation*}

\end{document}

Original answer

To load the Type 1 font instead, even on XeTeX or LuaTeX, use

\usepackage[T1]{fontenc}
\usepackage[type1]{libertine}

Since you’re using XeTeX, consider the libertinus package instead of libertine and \newtxmath. This loads unicode-math with the Libertinus Math font.