You need to declare a new caption font.
\documentclass{article}
\usepackage[lf]{ebgaramond}
\usepackage[cmintegrals,cmbraces]{newtxmath}
%\pdfmapfile{+EBGaramond-Maths.map} % not necessary on my system
\usepackage{ebgaramond-maths}
\usepackage{amsmath}
\usepackage{caption}
\newcommand{\fakebf}{\fontfamily{mdugm}\fontseries{b}\selectfont}
\DeclareTextFontCommand{\textbf}{\fakebf}
\DeclareCaptionFont{fakebf}{\fakebf}
\captionsetup{labelfont=fakebf}
\begin{document}
This text is in EB Garamond,
with \textbf{fake boldface}.
\textbf{This text is not in EB Garamond}
Some math $a+b=\textbf{T}$
\begin{figure}[htp]
\centering
A FIGURE HERE
\caption{A caption}
\end{figure}
\end{document}

But it looks like actors in Ben Hur wearing wristwatches.