The reason why replacing \bfseries with \titlecap does not work is that \bfseries is a declaration that does not take an argument, whereas \titlecap is a macro requiring an argument.
However, this approach may work, of redefining \@sect to look for subsections (when #2 = 2) and, upon finding so, replace the subsection title #8 with \titlecap{#8}. I also turned off \bfseries in #6 since that seems to be what the OP wanted.
\documentclass{amsart}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{etoolbox}
\usepackage{titlecaps}
\makeatletter
\let\sv@sect@sect
\def@sect#1#2#3#4#5#6[#7]#8{%
\ifnum#2=2
\def\next{\sv@sect{#1}{#2}{#3}{#4}{#5}{\normalfont}[#7]{\titlecap{#8}}}%
\else
\def\next{\sv@sect{#1}{#2}{#3}{#4}{#5}{#6}[#7]{#8}}%
\fi
\next
}
\makeatother
\begin{document}
\section{Section}
\subsection{Subsection with lowercase letters}
\subsection{another test case}
\end{document}
