(I updated this answer significantly after learning that modifying \everydisplay can cause serious problems with various display math style environments.)
Setting the Color=... option while loading a math font appears to override anything that \textcolor is supposed to do.
Instead of setting a Color option at the \setmathfont stage, you may achieve your objective by (a) modifying the \everymath token list for inline math material and (b) providing an \apptocmd directive to set the default color for \[ ... \] material and, similarly, \AtBeginEnvironment directives for the equation and equation* environments. It turns out not be necessary to provide analogous commands for the display math environments of the amsmath package.
Observe that this approach does not work for $$ ... $$ material. This is deliberate. For an in-depth discussion of why one should not use $$ in a LaTeX document to initiate and terminate display math mode, please see the posting Why is \[ ... \] preferable to $$ ... $$?.

\documentclass{article}
\usepackage{amsmath} % provides various displaymath environments
\usepackage{xcolor} % for '\definecolor', '\color', and '\textcolor'
\definecolor{navy}{rgb}{0.0,0.0,0.5}
\usepackage{unicode-math}
\setmathfont[slash-delimiter=frac]{Cambria Math}
% Auto-color inline math material:
\everymath=\expandafter{\the\everymath\color{navy}}
% Auto-color display math material:
\usepackage{etoolbox} % <-- required for XeLaTeX; optional for LuaLaTeX
\apptocmd{[}{\color{navy}}{}{}
\AtBeginEnvironment{equation}{\color{navy}}
\AtBeginEnvironment{equation*}{\color{navy}}
\begin{document}
% First, some inline math material.
(0+\textcolor{red}{1}=1),
\begin{math}1+1=\textcolor{red}{2}\end{math},
$\textcolor{red}{2}+1=3$.
%% Second, various forms of display math material.
[
x^2-(\textcolor{red}{1})^2=1
]
\begin{equation}
x^2-(\textcolor{red}{1})^2=1
\end{equation}
\begin{gather}
x^2-y^2=1\
x^2-(\textcolor{red}{1})^2=1
\end{gather}
\begin{align}
x^2-y^2&=1\
x^2-(\textcolor{red}{1})^2&=1
\end{align}
\end{document}
fontencpackage when using XeLaTeX or LuaLaTeX. (b) Don't load bothxcolorandcolor. – Mico Jul 03 '20 at 00:48amssymbif you loadunicode-math, which will just overwrite everything in it. You should not loadfontenctogether with Unicode fonts. You would not normally need to loadamsmathwithunicode-math, but you are also loadingamsthm, so load ``amsmath**before**amsthm`. – Davislor Jul 03 '20 at 11:30unicode-mathmakes almost all legacy font packages redundant, although it’s written to be robust if you load many of them anyway. – Davislor Jul 03 '20 at 11:31