Since you asked specifically about using them in math mode, one way to do that in unicode-math is:
\documentclass[varwidth, preview]{standalone}
\usepackage{amsmath}
\usepackage[math-style=ISO]{unicode-math}
\setmathfont{Latin Modern Math}
\setmathfont[range=up/{Latin,latin,Greek,greek}]{CMU Serif Upright Italic}
\begin{document}
\begin{align*}
\symup{e}^{\symup{i} x} &= \cos{x} + \symup{i} \sin{x} \\
\symup{e}^{\symup{i \pi}} + 1 &= 0
\end{align*}
\end{document}

The above is set in ISO style, with the constants e, i and π set upright, the variable x in italic, the digits 1 and 0 left unchanged (therefore still matching the default text font, so 10 looks the same as $10$) and the operator names cos and sin set in the text font. The unicode-math package is smart enough to tell the difference between upright math symbols, \symup, and snippets of text such as \operatorname, \text or \mathrm.
If you instead want to use upright italic letters by default, but still have the usual slanted italic available with \symit{x}, pass the [math-style=upright] option to \usepackage{unicode-math}. If you want to replace the italic rather than the upright math latters, change range=up to range=it.
There is no corresponding bold upright italic font, but should you wish to use one (such as for vectors), you might reload the regular-weight font with the range=bfup/{latin,Latin,greek,Greek} option from unicode-math and the FakeBold= option from fontspec.
The most famous upright italic math font Donald Knuth was involved with was AMS Euler, by Hermann Zapf. This is available in OpenType as the font Neo Euler.
To use a TrueType or OpenType upright italic font as your text font, use fontspec (see Heiko Oberdiek’s answer). You can also use options on commands such as \setmainfont to replace only the italic style of a font family. You should only use a legacy NFSS toolchain if you’re forced to, for compatibility with an old document or old software.
{math mode)in the title but all your examples are text fonts? – David Carlisle May 15 '18 at 19:05@DavidCarlisle: not all the above are defined for
– jessexknight May 16 '18 at 00:43\math**. I've updated the question - but it only makes my case stronger!