How does a math operator specify the horizontal position of its limits?

Question

When limits are put above and below an integral (as with \int\limits_0^1), each limit is placed in the correct horizontal position. They are not directly above and below each other. The upper limit is to the right, near the top of the integral symbol, and the lower limit is to the left, near the bottom of the symbol.

How are those horizontal positions specified?

In particuar, if I define my own custom operator, how can I specify the horizontal positions where its limits should go?

As a concrete example, suppose I make an operator that looks like a giant backslash, as with \mathchardef\myop="130F. Since it looks like a backslash, I want the upper limit to be positioned further left, and the lower limit further right. I do not want the users of my operator to do the positioning manually, I want it to work automatically.

Answer 1

For big operators, TeX (ab)uses the standard metric information. Let's see \intop, which is

\mathchardef\intop="1352

So we can look at the metric information from cmex10

(CHARACTER O 122
   (CHARWD R 0.472223)
   (CHARDP R 1.111122)
   (CHARIC R 0.194446)
   (NEXTLARGER O 132)
   )
(CHARACTER O 132
   (CHARWD R 0.555557)
   (CHARDP R 2.222246)
   (CHARIC R 0.444446)
   )

because hexadecimal "52 is octal `122. As you see, the italic correction for the display style integral symbol is quite large. This is used for placing limits both in the \nolimits and the \limits case (see appendix G in the TeXbook).

For your operator, we need to somehow emulate the behavior. Just \myop\limits_{a}^{b} produces

so we need to move the limits right and left.

A proof of concept.

\documentclass{article}
\DeclareMathSymbol{\myopsymbol}{\mathop}{largesymbols}{"0F}
\NewDocumentCommand{\myop}{t\limits e{_^}}{%
  \IfBooleanTF{#1}{\myoplimits{#2}{#3}}{\myopnolimits{#2}{#3}}%
}
\makeatletter
\newcommand{\myoplimits}[2]{%
  \begingroup
  \sbox\z@{$\m@th\displaystyle\myopsymbol$}%
  \myopsymbol\limits
  \IfValueT{#1}{{,\kern0.5\wd\z@#1}}%
  \IfValueT{#2}{^{#2\kern0.5\wd\z@,}}%
  \endgroup
}
\newcommand{\myopnolimits}[2]{%
  \begingroup
  \sbox\z@{$\m@th\displaystyle\myopsymbol$}%
  \myopsymbol\nolimits
  \IfValueT{#1}{{#1}}%
  \IfValueT{#2}{^{\kern-0.5\wd\z@!#2}}%
  \endgroup
}
\makeatother
\begin{document}
[
\myop\limits_{a}^b \qquad \myop_{a}^{b}
]
\end{document}

Answer 2

There's a new package pdfmsym for this. (I learnt about it from the author's answer.)

(note that it isn't a LaTeX package so use \input instead of \usepackage.)

\documentclass{article}
\makeatletter
\input pdfmsym
\pdfmsymsetscalefactor{\f@size}  % specify font size. Alternatively \pdfmsymsetscalefactor{10}
\DeclareMathSymbol{\myopsymbol}{\mathop}{largesymbols}{"0F}
\newcommand\myopb{%
    \@skewedlim@op{\myopsymbol}{10}{-1}{10}{-12}{10}{-1}\nolimits
}

The 6 numbers control the amount of skew, refer to the documentation:

For an example: append the following

% ======== egreg's code below ========
\NewDocumentCommand{\myop}{t\limits e{_^}}{%
  \IfBooleanTF{#1}{\myoplimits{#2}{#3}}{\myopnolimits{#2}{#3}}%
}
\newcommand{\myoplimits}[2]{%
  \begingroup
  \sbox\z@{$\m@th\displaystyle\myopsymbol$}%
  \myopsymbol\limits
  \IfValueT{#1}{{,\kern0.5\wd\z@#1}}%
  \IfValueT{#2}{^{#2\kern0.5\wd\z@,}}%
  \endgroup
}
\newcommand{\myopnolimits}[2]{%
  \begingroup
  \sbox\z@{$\m@th\displaystyle\myopsymbol$}%
  \myopsymbol\nolimits
  \IfValueT{#1}{{#1}}%
  \IfValueT{#2}{^{\kern-0.5\wd\z@!#2}}%
  \endgroup
}
\makeatother
\begin{document}
egreg's version:
[
\myop\limits_{a}^b f(x) dx
\myop_{a}^{b} f(x) dx
]
pdfmsym version:
[
    \myopb\limits_a^b f(x) dx
    \myopb_a^b f(x) dx
]
\end{document}

Result:

Admittedly, the kerning of the superscript/subscript isn't quite right in \textstyle (but neither is it in egreg's answer), for which the reader can try using \mathchoice and manually tune the kerning.