$A_\frac{a}{bc}$ works because _ does expansion in order to find a left brace, which may also be implicit. Thus the expansion of \frac is pursued all the way, since the usual definition is
% latex.ltx, line 4260:
\def\frac#1#2{{\begingroup#1\endgroup\over#2}}
and so the { before \begingroup is scanned when the arguments have already been absorbed.
This doesn't happen with \quot, because it has an optional argument, so the expansion ends at \kernel@ifnextchar because this macro begins with \let.
Adding a pair of braces to the definition, like in
\newcommand{\quot}[3][c]{{\frac{#2}{#3#1}}}
will not solve the issue. You could do it in an indirect way:
\newcommand{\quot}{\bgroup\innerquot}
\newcommand{\innerquot}[3][c]{\frac{#2}{#3#1}\egroup}
but I recommend getting the habit of using braces around subscripts. If you try
$A_\notin$
you'll probably understand why. And no, there's no way to make \notin work in subscript without braces, that is, one needs to type $A_{\notin}$. The trick above works because \quot will make an ordinary symbol and the braces do no harm when used in text style.
\frac, with two arguments, gets away with it because it has no optional arguments, and the expansion is "doubly braced" so that acts as a single token when it's read. – barbara beeton Aug 14 '14 at 18:39\newcommand{\quot}[3][c]{{\frac{#2}{#3#1}}}? – Dox Aug 14 '14 at 18:56