The OT1 encoding has no slot for the braces, because the normally allocated ones are for different symbols (the en-dash and the closing double quotes respectively). Instead, the braces are in a T1 encoded font.
Jump to the end for a short summary; otherwise, read on.
TeXnical details
When the encoding is OT1, \{ and \} choose a different font. Let's see; I assume that xpatch is loaded for \xshowcmd:
\show\{
> \{=macro:
->\x@protect \{\protect \{ .
\xshowcmd\{
> \{ =\long macro:
->\ifmmode \lbrace \else \textbraceleft \fi .
Since we are in text mode, \textbraceleft is expanded:
\show\textbraceleft
> \textbraceleft=macro:
->\OMS-cmd \textbraceleft \OMS\textbraceleft .
This one is harder: one has to know what to show: here it is
\expandafter\show\csname OMS\string\textbraceleft\endcsname
> \OMS\textbraceleft=\char"66.
However first of all LaTeX tries to see if \?\textbraceleft is defined, which it is:
\expandafter\show\csname ?\string\textbraceleft\endcsname
> \?\textbraceleft=\long macro:
->\UseTextSymbol {OMS}\textbraceleft .
OK. We're at it! LaTeX executes in a group
{\fontencoding{OMS}\selectfont\textbraceleft}
which will print character 0x66 from the selected font. But font substitutions are at the corner: indeed, the current family is LinuxLibertineT-TLF and there's a message
LaTeX Font Warning: Font shape `OMS/LinuxLibertineT-TLF/m/n' undefined
(Font) using `OMS/ntxsy/m/n' instead
(Font) for symbol `textbraceleft' on input line 7.
because there's no external font corresponding to what's in the first line and so LaTeX uses the default font for the OMS encoding, which happens to by ntxsy (NewTXMath, because of your packages). You don't get a warning with the default Computer Modern fonts, because LaTeX knows an OMS/cmr/m/n font, which is precisely the same as OMS/cmsy/m/n (look in the file omscmr.fd to see how this is done).
When the base encoding is T1, you get a different answer from \show\textbraceleft, namely
> \textbraceleft=macro:
->\T1-cmd \textbraceleft \T1\textbraceleft .
and now we can do
\expandafter\show\csname T1\string\textbraceleft\endcsname
to get
> \T1\textbraceleft=\char"7B.
so LaTeX prints the character in slot 0x7B in the current font.
What does the above mean?
The difference is due to the fact that the braces in NewTXMath are slightly different from those in Linux Libertine. The former are used with the OT1 encoding, the latter with T1.
OMS/ntxsy/m/n). – egreg Jan 26 '13 at 17:43fontenc. You’ll get the ⟨Th⟩ ligatures and I think the braces look right, but I’m not entirely sure. – doncherry Jan 26 '13 at 18:06