I managed to make this work using enumitem by looking at the moreenum package. Here is my solution, in case anyone is interested:
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{mathabx}
\newcommand\mathbfont{\usefont{U}{mathb}{m}{n}}
\def\mayaexpansion{%
\mayacntc=\mayacnta\mathbfont
\ifnum\mayacntc=0 0\else
\rotatebox[origin=c]{-90}{%
\loop\ifnum\mayacntc>5\advance\mayacntc by -5\repeat
\the\mayacntc\mayacntc=\mayacnta
\loop\ifnum\mayacntc>5\advance\mayacntc by -5 5\repeat}%
\fi}%
\makeatletter
\newcommand*{\mayalabel}[1]{%
\expandafter\@mayalabel\csname c@#1\endcsname}
\newcommand*{\@mayalabel}[1]{%
\protect\mayadigit{\number#1}}
\AddEnumerateCounter{\mayalabel}{\@mayalabel}{\mayadigit{5}}
\makeatother
\makeatletter
\newcommand*{\mayafull}[1]{%
\expandafter\@mayafull\csname c@#1\endcsname}
\newcommand*{\@mayafull}[1]{%
\protect\ensuremath{\maya{\number#1}}}
\AddEnumerateCounter{\mayafull}{\@mayafull}{\maya{5}}
\makeatother
Using [label=\mayalabel*] gives you \mayadigit{} while [label=\mayafull*] gives you \maya{}. None of them really work beyond twenty though. The former is wrong and the latter is bulky.
\mayadigit{\value{enumi}]? – Bernard Jul 02 '19 at 15:53\maya{...}. To remove the final dot, try usingenumitem(something like [label=\maya{enumi}]. – Bernard Jul 02 '19 at 16:02\maya{}is that it adds a box, which I don't like. The downside of\mayadigit{}is that it stops being correct at 21. On the other hand, while\maya{}is correct it becomes unwieldy due to the nature of the ordering (symbols stacked vertically). – Steve Jul 03 '19 at 08:56