Unusual symbol in equation

Question

How can I type the following in LaTeX?

Answer 1

I guess the main issues are, how to create a large open square symbol, center it on the math axis, and have it act as a math operator, i.e., place its argument below (and above, if there's an upper limit) the large open square. Sort of like \sum, right? A series of \vcenter, \hbox, and repeated \mathlarger instructions applied to the \Box macro solve these issues. (If you compare the following screenshot with the preceding one, you'll notice that the letter g to the left of the box is placed ever so slightly higher now.)

\documentclass{article}
\usepackage{amsmath,  % for "\DeclareMathOperator*" macro
            amsfonts, % for "\Box" macro
            relsize}  % for "\mathlarger" macro
\newcommand{\BigBox}{\vcenter{\hbox{$\mathlarger{\mathlarger 
     {\mathlarger{\mathlarger{\mathlarger{\Box}}}}}$}}}
\DeclareMathOperator*{\bigsquare}{\BigBox}

\begin{document}
\[
Z_{\mathcal{T}/G}\equiv \sum_{h\in G}\frac{1}{|G|} 
     \sum_{g\in G} g {\bigsquare_{\textstyle h}}
\]
\end{document}

Addendum -- Another way to create \BigBox is to load the graphicx package and to run

\newcommand{\BigBox}{\vcenter{\hbox{\scalebox{2}{$\Box$}}}}

As you've indicated in a comment that \BigBox isn't an "operator" but a partition function with boundary conditions, I think it makes sense -- from a LaTeX point of view -- to define a macro called, say \partfunc, that takes two arguments, here: g and h, and to align the first argument on the math axis.

\documentclass{article}
\usepackage{amsmath,  % for "\DeclareMathOperator*" macro
            amsfonts, % for "\Box" macro
            graphicx} % for "\scalebox" macro
\newcommand\BigBox{\vcenter{\hbox{\scalebox{2}{$\Box$}}}}
\newcommand\bigsquare{\mathop{\BigBox}\limits}
\newcommand\partfunc[2]{\vcenter{\hbox{$\textstyle #1$}}{\bigsquare_{\textstyle #2}}}

\begin{document}
\[
Z_{\mathcal{T}/G}\equiv \sum_{h\in G}\frac{1}{|G|} 
     \sum_{g\in G} \partfunc{g}{h}
\]
\end{document}

Answer 2

Geometric shape? Use picture mode!

\documentclass{article}
\usepackage{amsmath}
\usepackage{pict2e}

\makeatletter
\newcommand{\farsquare}[2]{#1\,{\mathpalette\far@square{#2}}}
\newcommand{\far@square}[2]{%
  \mathop{\vcenter{\hbox{%
    \sbox\z@{$\m@th#1\sum$}%
    \setlength{\unitlength}{0.9\dimexpr\ht\z@+\dp\z@}%
    \begin{picture}(1,1)
    \roundjoin
    \polyline(0,0)(0,1)(1,1)(1,0)(0,0)(0,0.5)
    \end{picture}%
  }}}\limits_{#1#2}%
}
\makeatother

\begin{document}

\[
Z_{\mathcal{T}/G}\equiv
\sum_{h\in G}\frac{1}{|G|} \sum_{g\in G} \farsquare{g}{h}
\]

\[
\textstyle
Z_{\mathcal{T}/G}\equiv
\sum_{h\in G}\frac{1}{|G|} \sum_{g\in G} \farsquare{g}{h}
\]

\end{document}

The second display emulates text style, just to see that the symbol scales well.