The problem is that \rotatebox typesets its argument in text mode and you would get “Missing $” errors with XeLaTeX, but nothing in LuaLaTeX (unless the current text font happens to have the character.
Anyway, using $\bigcupdot$ isn't enough.
\documentclass[twocolumn]{article}
\usepackage{amsmath}
\usepackage{unicode-math}
\usepackage{graphicx}
\setmathfont{STIXTwoMath-Regular.otf}
\makeatletter
\NewDocumentCommand{\bigcapdot}{}{%
\mathop{\mathpalette\bigcapdot@\relax}\displaylimits
}
\newcommand{\bigcapdot@}[2]{%
\vcenter{\hbox{\scalebox{1}[-1]{$\m@th#1\bigcupdot$}}}%
}
\makeatother
\begin{document}
$[\bigcupdot][\bigcapdot]$
[
[\bigcupdot][\bigcapdot]
]
\end{document}
Here twocolumn is just to make a smaller picture.

Why just adding $ around \bigcupdot isn't enough? Look at the comparison below.
\documentclass[twocolumn]{article}
\usepackage{amsmath}
\usepackage{unicode-math}
\usepackage{graphicx}
\setmathfont{STIXTwoMath-Regular.otf}
\makeatletter
\NewDocumentCommand{\bigcapdot}{}{%
\mathop{\mathpalette\bigcapdot@\relax}\displaylimits
}
\newcommand{\bigcapdot@}[2]{%
\vcenter{\hbox{\scalebox{1}[-1]{$\m@th#1\bigcupdot$}}}%
}
\makeatother
% for comparison
\newcommand{\simplebigcapdot}{\mathop{\rotatebox[origin=c]{180}{$\bigcupdot$}}}
\begin{document}
\subsubsection*{Scaling}
$[\bigcupdot][\bigcapdot]$
$\scriptstyle[\bigcupdot][\bigcapdot]$
$\scriptscriptstyle[\bigcupdot][\bigcapdot]$
[
[\bigcupdot][\bigcapdot]
]
\subsubsection*{Not scaling}
$[\bigcupdot][\simplebigcapdot]$
$\scriptstyle[\bigcupdot][\simplebigcapdot]$
$\scriptscriptstyle[\bigcupdot][\simplebigcapdot]$
[
[\bigcupdot][\simplebigcapdot]
]
\end{document}

If you need other flipped symbols:
\documentclass[twocolumn]{article}
\usepackage{amsmath}
\usepackage{unicode-math}
\usepackage{graphicx}
\setmathfont{STIXTwoMath-Regular.otf}
\makeatletter
\NewDocumentCommand{\bigcapdot}{}{%
\genericbigopflip{\bigcupdot}%
}
\NewDocumentCommand{\bigcapplus}{}{%
\genericbigopflip{\biguplus}%
}
\newcommand{\genericbigopflip}[1]{%
\mathop{\mathpalette\bigcapdot@{#1}}\displaylimits
}
\newcommand{\bigcapdot@}[2]{%
\vcenter{\hbox{\scalebox{1}[-1]{$\m@th#1#2$}}}%
}
\makeatother
\begin{document}
$[\bigcupdot][\bigcapdot][\biguplus][\bigcapplus]$
$\scriptstyle[\bigcupdot][\bigcapdot][\biguplus][\bigcapplus]$
$\scriptscriptstyle[\bigcupdot][\bigcapdot][\biguplus][\bigcapplus]$
[
[\bigcupdot][\bigcapdot][\biguplus][\bigcapplus]
]
\end{document}

\rotateboxis always text. You need\rotatebox[origin=c]{180}{$\bigcupdot$}. – campa Mar 21 '23 at 19:39\bigcupdotby\cupdotin my original code, the rotated version does typeset, without the dollar signs. – Evan Aad Mar 21 '23 at 19:44;-)– campa Mar 21 '23 at 19:46unicode-maththat\bigcupdotsimply expands to⨃which works in text given a suitable font. – David Carlisle Mar 21 '23 at 19:50\bigcapdot? – barbara beeton Mar 21 '23 at 20:09\bigcapdot(unlikeMnSymbolandfdsymbol). – Sandy G Mar 21 '23 at 20:23\bigcapplus, which would also be helpful. (\biguplusis available.) – Evan Aad Mar 21 '23 at 21:12\bigcupdotdoesn't "expand".latexdef -t lualatex -p unicode-math bigcupdot ⟹ "\bigcupdot: the character ⨃"– user202729 Mar 22 '23 at 11:48