The appended plain TeX example shows three cases:
- maths alphabets
\rm,\bf,\itusing plain TeX setup - similarly maths alphabets loaded from "classical" Latin Modern fonts
- similarly maths alphabets loaded with OpenType Latin Modern fonts
For the first two cases, text is set using ligatures; e.g., {\fam\itfam fin} will produce a ligature. In the final case when an OpenType font is used, the ligature doesn't appear. Using the font in-text shows that ligatures are being selected otherwise.
So what's going on? David Carlisle spotted this happening trying to fix up the behaviour of unicode-math, and I've been trying to figure out whether this is my fault! The same happens for both XeTeX and LuaTeX+luaotfload, so if it's an oversight, they are both culpable.

%!TEX TS-program = XeTeX
\ifx\XeTeXversion\undefined
\input luaotfload.sty
\def\otf#1{file:#1.otf}
\else
\def\otf#1{[#1.otf]}
\fi
\hsize=10cm
\parindent=0pt\relax
%%%%%%%%%%%%%%%%%%%%%%%%%%
\hrule
\bigskip
{\bf Default \TeX}
Roman, italic, and bold math families:
$$ {\fam0 abcfin} \quad {\fam\itfam } \quad {\fam\bffam abcfin} $$
%%%%%%%%%%%%%%%%%%%%%%%%%%
\font\myrm = ec-lmr10 at 10pt\relax
\font\mybf = ec-lmbx10 at 10pt\relax
\font\myit = ec-lmri10 at 10pt\relax
\newfam\rmmathfam
\newfam\bfmathfam
\newfam\itmathfam
\textfont\rmmathfam\myrm
\textfont\bfmathfam\mybf
\textfont\itmathfam\myit
\hrule
\bigskip
{\bf Loading new tfm fonts}
Roman ({\myrm abcfin}), italic ({\myit abcfin}), and bold ({\mybf abcfin}) families:
$$ {\fam\rmmathfam abcfin} \quad {\fam\itmathfam abcfin} \quad {\fam\bfmathfam abcfin} $$
%%%%%%%%%%%%%%%%%%%%%%%%%%
\font\Urm = "\otf{lmroman10-regular}" at 10pt\relax
\font\Ubf = "\otf{lmroman10-bold}" at 10pt\relax
\font\Uit = "\otf{lmroman10-italic}" at 10pt\relax
\newfam\Urmmathfam
\newfam\Ubfmathfam
\newfam\Uitmathfam
\textfont\Urmmathfam\Urm
\textfont\Ubfmathfam\Ubf
\textfont\Uitmathfam\Uit
\hrule
\bigskip
{\bf Loading new OpenType fonts}
Roman ({\Urm abcfin}), italic ({\Uit abcfin}), and bold ({\Ubf abcfin}) families:
$$ {\fam\Urmmathfam abcfin} \quad {\fam\Uitmathfam abcfin} \quad {\fam\Ubfmathfam abcfin} $$
\bye

luaotfloaddoesn't do text shaping in math nodes? – michal.h21 Jun 27 '15 at 08:08unicode-mathapplies thebasemode (search forRenderer=Basicin the source if you're interested) … and I'm not sure where to go next – Will Robertson Nov 09 '15 at 22:53\mathrm{a\neq b}. – Ulrike Fischer Nov 10 '15 at 11:44\mathrmis a dangerous habit anyway: If the range is mapped to some text font they are lost. Also loading the font in mode base works, and with unicode math there is a quite natural interface. See my edit http://tex.stackexchange.com/a/277395/2388. So perhaps one only need some interface for the fontspec solution. – Ulrike Fischer Nov 11 '15 at 09:09