Never use $$ in LaTeX
When you do \boxstar \boxstar \boxtimes \boxtimes you have the following sequence of atoms, due to the fact that your \boxstar is composed by an ordinary symbol (O) followed by a binary operation (B):
O (m) B (m) O (m) B (m) B (0) B
TeX will insert a medium space (m) where indicated, but zero space between the two last B's, because they are incompatible with binary operations, so they're treated as O.
How do you solve the issue? By stating how the new symbol should behave and using braces around \boxtimes so it effectively becomes an ordinary symbol:
\newcommand{\boxstar}{\mathbin{\mathrlap{\boxplus}{\boxtimes}}}
The braces are not really needed here, because a subformula consisting of OB will become OO (by the same rules mentioned above), but is conceptually better to have them.
This works because \boxplus and \boxtimes have the same width. In order to superimpose symbols with different widths, the simplest method is to exploit \ooalign, see https://tex.stackexchange.com/a/22375/4427 for a quick course on it.
Your example
\[
\boxstar \boxstar \boxtimes \boxtimes \boxtimes \boxtimes \boxtimes
\]
will now become
B B B B B B B
that's transformed into
O (m) B (m) O (m) B (m) O (m) B (m) O
because of incompatible BB sequences.
\documentclassand ending with\end{document}. – Bobyandbob Aug 26 '18 at 05:44