The Latin Modern fonts are preloaded at format creation. If you look in
/usr/local/texlive/2018/texmf-var/web2c/xetex/xelatex.log
(the location may vary depending on the flavor of TeX Live you're using) you find
(/usr/local/texlive/2018/texmf-dist/tex/latex/base/tuenc.def
File: tuenc.def 2018/08/11 v2.0j Standard LaTeX file
)
(/usr/local/texlive/2018/texmf-dist/tex/latex/base/tulmr.fd
File: tulmr.fd 2017/01/26 v1.8 Standard LaTeX font definitions for Latin Modern
)
(/usr/local/texlive/2018/texmf-dist/tex/latex/base/tulmss.fd
File: tulmss.fd 2017/01/26 v1.8 Standard LaTeX font definitions for Latin Modern
)
(/usr/local/texlive/2018/texmf-dist/tex/latex/base/tulmtt.fd
File: tulmtt.fd 2017/01/26 v1.8 Standard LaTeX font definitions for Latin Modern
)
Similarly for lualatex.log.
This is because if the engine is XeLaTeX or LuaLaTeX, the default encoding is set to TU and the default font family to lmr. The file tulmr.fd contains
\DeclareFontFamily{TU}{lmr}{}
\DeclareFontShape{TU}{lmr}{m}{n}%
{<-5.5> \UnicodeFontFile{lmroman5-regular}{\UnicodeFontTeXLigatures}
<5.5-6.5> \UnicodeFontFile{lmroman6-regular}{\UnicodeFontTeXLigatures}
<6.5-7.5> \UnicodeFontFile{lmroman7-regular}{\UnicodeFontTeXLigatures}
<7.5-8.5> \UnicodeFontFile{lmroman8-regular}{\UnicodeFontTeXLigatures}
<8.5-9.5> \UnicodeFontFile{lmroman9-regular}{\UnicodeFontTeXLigatures}
<9.5-11> \UnicodeFontFile{lmroman10-regular}{\UnicodeFontTeXLigatures}
<11-15> \UnicodeFontFile{lmroman12-regular}{\UnicodeFontTeXLigatures}
<15-> \UnicodeFontFile{lmroman17-regular}{\UnicodeFontTeXLigatures}
}{}
and other font definitions.
It's no longer a job for fontspec to define the standard fonts, as it used to be up to TeX Live 2015.
fontspec-luatex.styandfontspec-xetex.sty. – Bernard Jan 13 '19 at 23:53