It should be \hbox{$\scriptstyle#1$}. However, there's a better way:
\documentclass{article}
\usepackage{amsmath}
\makeatletter
\newcommand{\oset}[2]{{\mathpalette\o@set{{#1}{#2}}}}
\newcommand{\o@set}[2]{\o@@set{#1}#2}
\newcommand{\o@@set}[3]{%
\vbox{\offinterlineskip
\ialign{\hfil##\hfil\cr
$\m@th\o@set@demote{#1}#2$\cr
\noalign{\vskip0.2pt}
$\m@th#1#3$\cr
}%
}%
}
\newcommand{\o@set@demote}[1]{%
\ifx#1\displaystyle\scriptstyle\else
\ifx#1\textstyle\scriptstyle\else
\scriptscriptstyle\fi\fi
}
\makeatother
\begin{document}
$\oset{\sim}{t}$ versus $\tilde{t}$
\end{document}
On the other hand, \tilde{t} seems much better.

For completeness here's how to define \uset and \ouset.
\documentclass{article}
\usepackage{amsmath}
\usepackage{accents} % for comparison
\makeatletter
\newcommand{\oset}[2]{{\mathpalette\o@set{{#1}{#2}}}}
\newcommand{\uset}[2]{{\mathpalette\u@set{{#1}{#2}}}}
\newcommand{\ouset}[3]{\oset{#1}{\uset{#2}{#3}}}
\newcommand{\o@set}[2]{\o@@set{#1}#2}
\newcommand{\o@@set}[3]{%
\vbox{\offinterlineskip
\ialign{\hfil##\hfil\cr
$\m@th\ou@set@demote{#1}#2$\cr
\noalign{\vskip0.2pt}
$\m@th#1#3$\cr
}%
}%
}
\newcommand{\u@set}[2]{\u@@set{#1}#2}
\newcommand{\u@@set}[3]{%
\vtop{\offinterlineskip
\ialign{\hfil##\hfil\cr
$\m@th#1#3$\cr
\noalign{\vskip0.6pt}
$\m@th\ou@set@demote{#1}#2$\cr
}%
}%
}
\newcommand{\ou@set@demote}[1]{%
\ifx#1\displaystyle\scriptstyle\else
\ifx#1\textstyle\scriptstyle\else
\scriptscriptstyle\fi\fi
}
\makeatother
\begin{document}
$\oset{\sim}{t}$ versus $\tilde{t}$
$\uset{x}{t}$ versus $\underaccent{x}{t}$
$\ouset{\sim}{x}{t}$ versus $\underaccent{x}{\tilde{t}}$
\end{document}

\hbox{$\scriptstyle #1$}– egreg Oct 29 '19 at 15:51