In your example, you were nesting tikzpictures since you were putting a tikzpicture in the node of another tikzpicture. This is usually not a good idea. Then I replaced \tikzmark, which actually is a well-defined command in the tikzmark package, by a slightly modified version of Torbjørn T.'s \tikznode command. As for the arcs, they can be easily drawn with the arc syntax, and personally I like bent arrow heads better.
\documentclass{article}
\usepackage{amsmath}
\usepackage{tikz}
\usetikzlibrary{positioning,calc,arrows.meta,bending}
\newcommand{\tikznode}[3][]{\relax
\ifmmode%
\tikz[remember picture,baseline=(#2.base),inner sep=0pt]{\node[#1] (#2) {$#3$};}
\else
\tikz[remember picture,baseline=(#2.base),inner sep=0pt]{\node[#1] (#2) {#3};}%
\fi}
\begin{document}
\begin{equation*}
\tikznode[blue]{base}{b}^{\tikznode[red]{exponent}{a}}
\end{equation*}
\begin{tikzpicture}[overlay, remember picture,node distance =.2cm,inner sep=0pt]
\node[blue] (basedescr) [below right=of base]{base};
\draw[-{Stealth[bend]},thick] let \p1=($(basedescr.west)-(base.south)$),
\n1={ifthenelse(\x1>0,atan2(\y1,\x1),atan2(\y1,\x1)-180)},
\n2={veclen(\x1,\y1)/sqrt(2)}
in
(basedescr.west) arc(\n1-45:\n1-135:\n2) ;
\node[red] (exponentdescr) [above right=of exponent] {exponent};
\draw[-{Stealth[bend]},thick] let \p1=($(exponentdescr.west)-(exponent.north)$),
\n1={ifthenelse(\x1>0,atan2(\y1,\x1),atan2(\y1,\x1)-180)},
\n2={veclen(\x1,\y1)/sqrt(2)}
in (exponentdescr.west) arc(\n1+45:\n1+135:\n2);
\end{tikzpicture}
\end{document}

EDIT: Answers to your questions.
- You have
\tikzmark[blue]{base}{$b^{\,\,\tikzmark[red]{exponent}{a}}$}. This expands to \tikz[...]{\node[...]{...\tikz[...]{\node[...]{...};}};}, which is a TikZ picture inside a TikZ picture, aka nested tikzpicture.
- If you use the
below right key, say, you can pass to it two distances separated by and, e.g. below right=2pt and 2mm of base will move the thing only 2pt down but 2mm right. If you want it the basedescr node further up, you may either use negative dimensions, or something like right=2mm of base,yshift=-6pt.
Here is the updated code. There are good chances that you will get a much better answers using LoopSpace's great tikzmark library. which has new commands that are superior to the (nevertheless very nice and useful) \tikznode command by Torbjørn T..
\documentclass{article}
\usepackage{amsmath}
\usepackage{tikz}
\usetikzlibrary{positioning,calc,arrows.meta,bending}
\newcommand{\tikznode}[3][]{\relax
\ifmmode%
\tikz[remember picture,baseline=(#2.base),inner sep=0pt]{\node[#1] (#2) {$#3$};}
\else
\tikz[remember picture,baseline=(#2.base),inner sep=0pt]{\node[#1] (#2) {#3};}%
\fi}
\begin{document}
\begin{equation*}
\tikznode[blue]{base}{b}^{\tikznode[red]{exponent}{a}}
\end{equation*}
\begin{tikzpicture}[overlay, remember picture,inner sep=0pt]
\node[blue] (basedescr) [below right=2pt and 2mm of base]{base};
\draw[-{Stealth[bend]},thick] let \p1=($(basedescr.west)-(base.south)$),
\n1={ifthenelse(\x1>0,atan2(\y1,\x1),atan2(\y1,\x1)-180)},
\n2={veclen(\x1,\y1)/sqrt(2)}
in
(basedescr.west) arc(\n1-45:\n1-135:\n2) ;
\node[red] (exponentdescr) [above right=2pt and 2mm of exponent] {exponent};
\draw[-{Stealth[bend]},thick] let \p1=($(exponentdescr.west)-(exponent.north)$),
\n1={ifthenelse(\x1>0,atan2(\y1,\x1),atan2(\y1,\x1)-180)},
\n2={veclen(\x1,\y1)/sqrt(2)}
in (exponentdescr.west) arc(\n1+45:\n1+135:\n2);
\end{tikzpicture}
\end{document}
