The primitive \vcenter can only be used in math mode, but as soon as TeX starts processing its contents it is in internal vertical mode. When processing is finished, TeX builds a \vbox and moves its reference point in such a way that the height is half the height of the box plus the math axis height and the depth js half the height of the box minus the math axis height.
The width of a \vbox is determined as that of the widest horizontal box it contains. This is the reason for using \hbox that doesn't initiate horizontal mode in the \vbox; with a single \hbox, the width of the two boxes will be the same.
Now the contents of \hbox is processed in restricted horizontal mode; you can do a font change command such as \tiny and then you need to start math mode if you want to use math symbols.
Another way to get the result, but with correct scaling in subscripts and superscripts can be
\documentclass{article}
\usepackage{amsmath,graphicx}
\makeatletter
\newcommand{\tinysymbol}[2][\mathord]{%
#1{\mathpalette\tiny@symbol{#2}}%
}
\newcommand{\tiny@symbol}[2]{%
\vcenter{\hbox{\scalebox{0.65}{$\m@th#1#2$}}}%
}
\makeatother
\newcommand{\tbullet}{\tinysymbol[\mathbin]{\bullet}}
\newcommand{\othertbullet}{\mathbin{\vcenter{\hbox{\tiny$\bullet$}}}}
\begin{document}
$a\othertbullet b_{a\othertbullet b}$
$a\tbullet b_{a\tbullet b}$
\end{document}
