You can load the font with a different scaling factor:
\documentclass{article}
\usepackage{wasysym} % just for comparison
\DeclareFontFamily{U}{wasysmall}{}
\DeclareFontShape{U}{wasysmall}{m}{n}{
<-5.5> s*[0.75] wasy5
<5.5-6.5> s*[0.75] wasy6
<6.5-7.5> s*[0.75] wasy7
<7.5-8.5> s*[0.75] wasy8
<8.5-9.5> s*[0.75] wasy9
<9.5-> s*[0.75] wasy10
}{}
\DeclareFontShape{U}{wasysmall}{b}{n}{
<-5.5> s*[0.75] wasyb5
<5.5-6.5> s*[0.75] wasyb6
<6.5-7.5> s*[0.75] wasyb7
<7.5-8.5> s*[0.75] wasyb8
<8.5-9.5> s*[0.75] wasyb9
<9.5-> s*[0.75] wasyb10
}{}
\DeclareFontShape{U}{wasysmall}{bx}{n}{ <-> sub * wasysmall/b/n}{}
\DeclareSymbolFont{wasysmall}{U}{wasysmall}{m}{n}
\SetSymbolFont{wasysmall}{bold}{U}{wasysmall}{b}{n}
\DeclareMathSymbol{\smallhexagon}{\mathord}{wasysmall}{57}
\begin{document}
{\tiny\varhexagon} $\scriptstyle\smallhexagon$ $A_{\smallhexagon}$
\end{document}
Note that loading wasysym is not required, I did just for the comparison.

You can avoid wasting a symbol font:
\documentclass{article}
\usepackage{amsmath}
\DeclareFontFamily{U}{wasysmall}{}
\DeclareFontShape{U}{wasysmall}{m}{n}{
<-5.5> s*[0.75] wasy5
<5.5-6.5> s*[0.75] wasy6
<6.5-7.5> s*[0.75] wasy7
<7.5-8.5> s*[0.75] wasy8
<8.5-9.5> s*[0.75] wasy9
<9.5-> s*[0.75] wasy10
}{}
\newcommand{\smallhexagon}{\text{\usefont{U}{wasysmall}{m}{n}\symbol{57}}}
\begin{document}
$A_{\smallhexagon}$
\end{document}
If you just need the symbol to change size in subscripts, you can do in a different way:
\documentclass{article}
\usepackage{amsmath}
\usepackage{wasysym}
\DeclareMathSymbol{\mhexagon}{\mathord}{wasy}{57}
\begin{document}
$A_{\varhexagon}$ (wrong)
$A_{\mhexagon}$
\end{document}

\tinyor to\largeis the same concept. – CarLaTeX Mar 20 '19 at 07:05