Placement of limits

Question

I'm trying to define a directed supremum operator in latex but I can't come up with something that would behave well in all situations. Here's what I've tried:

\documentclass{article}
\usepackage{amsmath}

\newcommand\dirup{\!{}^{\upset}}
\newcommand\upset{\ensuremath{\mathord{\uparrow}\mkern1mu}}


\begin{document}

$\mathop{\bigsqcup\dirup}_{i\in I}$
$$\mathop{\bigsqcup\dirup}_{i\in I}$$

$\mathop{\bigsqcup\dirup_{i\in I}}$
$$\mathop{\bigsqcup\dirup_{i\in I}}$$

$\mathop{\bigsqcup_{i\in I}}\dirup$
$$\mathop{\bigsqcup_{i\in I}}\dirup$$
\end{document}

In the first case, the subscripts are moved to right (in both math modes). In the second case, inline math mode works as expected but the display mode doesn't put the subscript below the supremum sign. Lastly the third case works as expected in display math mode but the inline math is broken because \dirup is moved to the right.

Any ideas how to fix it? I'm fine with redefining \dirup.

Edit: Note that the operator should also behave well when the subscript is omitted, e.g. the following shouldn't display the set \{ a \} too close to the supremum:

$\mathop{\bigsqcup\dirup} \{ a \}$

Answer 1

Here's a complicated version, but, hey, it works!

The \negphantom macro can be found at Negative phantom inside equations

\documentclass{article}
\usepackage{amsmath,mathstyle}

\makeatletter
\newlength{\negph@wd}
\DeclareRobustCommand{\negphantom}[1]{%
  \ifmmode
    \mathpalette\negph@math{#1}%
  \else
    \negph@do{#1}%
  \fi
}
\newcommand{\negph@math}[2]{\negph@do{$\m@th#1#2$}}
\newcommand{\negph@do}[1]{%
  \settowidth{\negph@wd}{#1}%
  \hspace*{-\negph@wd}%
}

\newcommand\bigsqcupwithup{%
  \ifnum\mathstyle=\z@ % displaystyle
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi
  {%
   \mathop{}\!\negphantom{{}^{\uparrow}}
   \mathop{\hphantom{{}^\uparrow}{\bigsqcup}{}^{\uparrow}}%
  }%
  {\bigsqcup^{\scriptscriptstyle\uparrow}}%
}
\makeatother

\begin{document}

$\bigsqcupwithup_{i\in I}$
\[
A\bigsqcupwithup_{i\in I}B_i
\]
\[
A\bigsqcup_{i\in I}B_i
\]

\end{document}

---

A different version

\documentclass{article}
\usepackage{amsmath}

\makeatletter
\newlength{\negph@wd}
\DeclareRobustCommand{\negphantom}[1]{%
  \ifmmode
    \mathpalette\negph@math{#1}%
  \else
    \negph@do{#1}%
  \fi
}
\newcommand{\negph@math}[2]{\negph@do{$\m@th#1#2$}}
\newcommand{\negph@do}[1]{%
  \settowidth{\negph@wd}{#1}%
  \hspace*{-\negph@wd}%
}

\DeclareRobustCommand{\bigsqcupwithup}{%
  \@ifnextchar_{\sub@bigsqcupwithup}{\nosub@bigsqcupwithup}%
}

\newcommand{\nosub@bigsqcupwithup}{%
  \mathop{
    \mathchoice
      {\disp@bigsqcupwithup{}}
      {\nodisp@bigsqcupwithup{\hphantom{\scriptscriptstyle\uparrow}}}
      {\nodisp@bigsqcupwithup{\hphantom{\scriptscriptstyle\uparrow}}}
      {\nodisp@bigsqcupwithup{\hphantom{\scriptscriptstyle\uparrow}}}
  }
}
\def\sub@bigsqcupwithup_#1{%
  \mathop{
    \mathchoice
      {\disp@bigsqcupwithup{#1}}
      {\nodisp@bigsqcupwithup{#1}}
      {\nodisp@bigsqcupwithup{#1}}
      {\nodisp@bigsqcupwithup{#1}}
  }
}

\newcommand{\disp@bigsqcupwithup}[1]{%
  \negphantom{{}^\uparrow}%
  {\mathop{\hphantom{{}^\uparrow}{\bigsqcup}{}^\uparrow}\limits_{#1}}%
}
\newcommand{\nodisp@bigsqcupwithup}[1]{%
  \bigsqcup^{\scriptscriptstyle\uparrow}_{#1}%
}
\makeatother

\begin{document}

$\bigsqcupwithup_{i\in I}X$
$\bigsqcupwithup X$
\[
A\bigsqcupwithup_{i\in I}B_i
\qquad
A\bigsqcupwithup B_i
\]

\end{document}

Answer 2

Shown, in textstyle, displaystyle, scriptstyle, and scriptscriptstyle. EDITED to consider case of no subscripts.

\documentclass{article}
\usepackage{amsmath}
\usepackage{stackengine,scalerel}
\def\foo{\,\ThisStyle{\ensurestackMath{%
  \bigsqcup\stackengine{-0pt}{\!}{\SavedStyle\!^{\mathord{\uparrow}}}{O}{l}{F}{T}{S}}}\,}
\DeclareMathOperator*{\foobarX}{\foo}
\newcommand\foobar{\!\foobarX}
\begin{document}
\centering$a\foobar_{i\in I}a\qquad a\foobar a$
\[a\foobar_{i\in I}a\qquad a\foobar a\]
\[\scriptstyle a\foobar_{i\in I}a\qquad a\foobar a\]
\[\scriptscriptstyle a\foobar_{i\in I}a\qquad a\foobar a\]
\end{document}

Or maybe this slight variant, which was REEDITED to handle particular use cases provided by the OP. It screens for \{, [ and ( where spacing had previously been an issue.

\documentclass{article}
\usepackage{amsmath}
\usepackage{stackengine,scalerel}
\def\foo{\,\ThisStyle{\ensurestackMath{%
  \bigsqcup\stackengine{-0pt}{\,}{\SavedStyle\!^{\mathord{\uparrow}}}{O}{l}{F}{T}{S}}}\!}
\DeclareMathOperator*{\foobarX}{\foo}
\makeatletter
\newcommand\foobar{\!\foobarX\@ifnextchar\{{\,}{\@ifnextchar[{\,}{\@ifnextchar({\,}{}}}}
\makeatother
\begin{document}
\centering$x\foobar_i A_i\qquad x\foobar_{i\in I}A_i\qquad x\foobar A_i$
\[x\foobar_i A_i\qquad x\foobar_{i\in I}A_i\qquad x\foobar A_i\]
\[\scriptstyle x\foobar_i A_i\qquad x\foobar_{i\in I}A_i\qquad x\foobar A_i\]
\[\scriptscriptstyle x\foobar_i A_i\qquad x\foobar_{i\in I}A_i\qquad x\foobar A_i\]

$\quad \foobar \{ A_i : i\in I \}\quad \foobar [ A_i : i\in I ]\quad \foobar ( A_i : i\in I )$
\end{document}

Answer 3

Since you are concerned with white space, maybe \mathrlap?

\documentclass{article}
\usepackage{amsmath}
\usepackage{mathtools}

\newcommand\dirup{\!{}^{\upset}}
\newcommand\upset{\mathrlap{\mathord{\uparrow}}}


\begin{document}

$\mathop{\bigsqcup\dirup}_{i\in I}$
\[\mathop{\bigsqcup\dirup}_{i\in I}\]

$\mathop{\bigsqcup\dirup_{i\in I}}$
\[\mathop{\bigsqcup\dirup_{i\in I}}\]

$\mathop{\bigsqcup_{i\in I}}\dirup$
\[\mathop{\bigsqcup_{i\in I}}\dirup\]
\end{document}