The question is open-ended, but I will toss some things out there. It wasn't clear whether by saying "behave like", you wanted \sq to occupy the same horizontal space as \neg. If so, one could either increase the size of \sq or add extra space around it. But, as to handling the sizing of the new symbol, the scalerel package scales one symbol the vertical footprint of another, while preserving the current math style. Thus, if I tell \sq to take up the same vertical footprint as \neg, then since \neg changes size with the math style, so will \sq.
In addition, scalerel provides (in the context of scalerel arguments) the lengths \LMex and \LMpt (LM standing for "local mathstyle"), which provide the length of 1ex and 1pt, scaled to the current math style. This allows dimensions to be scaled to lower math styles.
But, as is often the case,some dimensions are not scaled to smaller math styles, for example glyph sidebearings(*) are often not (at the discretion of the font designer). To handle this non uniformity of scale, if I am trying to mimic the spacing of a pre-existing glyph (such as \neg), I can combine both dimensions (pts and \LMpts). So, for example, when I say \kern1.23pt\kern-.9\LMpt, this is .33pt in text and display style, but more than that in the smaller math styles because, in the smaller math styles, .9\LMpt is smaller than .9pt. Such techniques can be used to correct for the glyph sidebearing issue.
Fine tuning can be achieved by tweaking the scalerel-assumed scale factors associated with \scriptstyle and \scriptscriptstyle, which by default are
\def\scriptstyleScaleFactor{0.7}
\def\scriptscriptstyleScaleFactor{0.5}
As a point of interest, I invoke \sqsymbol with an argument of 1\LMex. Try replacing that with 1ex and see the difference at the smaller math styles.
The MWE:
\documentclass{article}
\usepackage{tikz,scalerel}
\newcommand\sqsymbol[1]{%
\tikz[x=#1,y=#1] {\draw[red,line cap=round,line join=round] (0,.83) -| ++(1.2,-.53) (0,0) coordinate (a);}%
}
\newcommand\sq[0]{\ThisStyle{\kern1.23pt\kern-.9\LMpt\scalerel*{\sqsymbol{1\LMex}}{\neg}%
\kern1.23pt\kern-.9\LMpt}}
\begin{document}
\(\displaystyle A \sq B\), \(A \sq B\), \(A_{C \sq B}\), \(A_{C_{D \sq B}}\)
\(\displaystyle A \neg B\), \(A \neg B\), \(A_{C \neg B}\), \(A_{C_{D \neg B}}\)
\end{document}
One might observe that, in \displaystyle, the \sq symbol is wider than \neg, while in \scriptscriptstyle, it is narrower. The reason for this can be explained. The glyph designer of \neg chose to change the aspect ratio of \neg in the smaller math styles (the vertical stub becomes relatively shorter). In contrast, the aspect ratio of \sq , defined generically, remains constant. Since \sq is scaled to the vertical footprint of \neg, the relative widths of \neg and \sq will change as the aspect ratio of \neg changes.
An alternative would be, not to scale \sq to the vertical footprint of \neg, but to the vertical footprint of something that changes both its absolute as well as relative height at smaller math styles, for example, \rule[-.12ex]{0pt}{\dimexpr.24ex+.7\LMex}}\kern.5pt}. Doing it this way allows the sidebearings to now be more simply expressed as a fixed .5pt, using
\renewcommand\sq[0]{\ThisStyle{\kern.5pt\scalerel{\sqsymbol{1ex}}{%
\rule[-.12ex]{0pt}{\dimexpr.24ex+.7\LMex}}\kern.5pt}}
Showing the original in line 1, this revised in in line 2, \neg in line 3, and the overlay of lines 2 and 3 in line 4:
\documentclass{article}
\usepackage{tikz,scalerel,stackengine}
\newcommand\sqsymbol[1]{%
\tikz[x=#1,y=#1] {\draw[red,line cap=round,line join=round] (0,.83) -| ++(1.2,-.53) (0,0) coordinate (a);}%
}
\newcommand\sq[0]{\ThisStyle{\kern1.23pt\kern-.9\LMpt\scalerel*{\sqsymbol{1\LMex}}{\neg}%
\kern1.23pt\kern-.9\LMpt}}
\begin{document}
\(\displaystyle A \sq B\), \(A \sq B\), \(A_{C \sq B}\), \(A_{C_{D \sq B}}\)
\renewcommand\sq[0]{\ThisStyle{\kern.5pt\scalerel{\sqsymbol{1ex}}{%
\rule[-.12ex]{0pt}{\dimexpr.24ex+.7\LMex}}\kern.5pt}}
\(\displaystyle A \sq B\), \(A \sq B\), \(A_{C \sq B}\), \(A_{C_{D \sq B}}\)
\(\displaystyle A \neg B\), \(A \neg B\), \(A_{C \neg B}\), \(A_{C_{D \neg B}}\)
\stackengine{0pt%
}{\(\displaystyle A \neg B\), \(A \neg B\), \(A_{C \neg B}\), \(A_{C_{D \neg B}}\)
}{\(\displaystyle A \sq B\), \(A \sq B\), \(A_{C \sq B}\), \(A_{C_{D \sq B}}\)
}{O}{l}{F}{F}{L}
\end{document}
My answer here, How are big operators defined?, discusses the issue further, when differences between \displaystyle and \textstyle are relevant.
*Sidebearings are the amount of horizontal dead space that are intrinsically built into a glyph by the font designer. For instance, the code
\fboxsep=0pt
\fbox{$+$}
\fbox{$\neg$}
\fbox{$\scriptstyle\neg$}
\fbox{$\scriptscriptstyle\neg$}
yields
As one can see, the sidebearing space is non-trivial AND it does not necessarily scale with the smaller math styles. The OP, in creating \sqsymbol, created a symbol with no sidebearings. Thus, part of solving the spacing riddle is allotting some sidebearing and deciding whether it should scale at the smaller mathstyles or not. The choice is up to the font/symbol designer. However, it gets complicated if one wants their symbol to mimic the horizontal spacing of another pre-existing glyph. In that case, knowledge of the glyph's sidebearings at the different math sizes is essential to a satisfactory mimicry.
\sqis different than\neg. That has nothing to do with the [empty] spacing, but do you wish the non-blank portion of\sqto take the same horizontal space as\neg? – Steven B. Segletes Aug 04 '17 at 11:54\negsymbol. Hopefully that ensures that the focus is on the spacing and relative size. – Andrew Stacey Aug 04 '17 at 12:10