When \ifnum is evaluated, \else is not removed, so \mathf sees it and it never scans further for a next _. You should have \expandafter\mathf instead, so the \else is removed up to the matching \fi before \mathf is expanded.
However, I suggest you some better code:
\documentclass{article}
\usepackage{mathspec}
\setmainfont{Hoefler Text}
\setmathfont(Digits,Latin)[Numbers=Lining]{Hoefler Text}
\setmathrm[Numbers=Lining]{Hoefler Text}
\makeatletter
\AtBeginDocument{%
\edef\SAVEfCODE{\the\Umathcodenum`f } % just for the test
\Umathcharnumdef\math@f\Umathcodenum`f
\mathcode`f=\string"8000 }
\begingroup\lccode`~=`f
\lowercase{\endgroup
\def~{%
\math@f
\ifnum\mathgroup=\m@ne
\expandafter\@firstofone
\else
\expandafter\@gobble
\fi
{\@ifnextchar_{\relax}{\,}}%
}
}
\makeatother
\begin{document}
$f(x)f_1(x)\mathbf{f}(x)\mathbf{f}_1(x)\mathbf{f_1}(x)$
%% Just to show the difference
\Umathcodenum`f=\SAVEfCODE\relax
$f(x)f_1(x)\mathbf{f}(x)\mathbf{f}_1(x)\mathbf{f_1}(x)$
\end{document}
The parts with \SAVEfcode are just for showing the difference.

Note: in \ifnum\the\mathgroup, \the is redundant, because \mathgroup is a number. I used \Umath... because its safer in the context of mathspec.
\mathfis always\elseso never _(you are missing some\expandafter` – David Carlisle Nov 02 '14 at 18:51