The four hexadecimal digits "kfab in a \mathchar specify
k is the atom type (0 = ordinary, 1 = operator, 2 = binary operation, 3 = relation, 4 = opening, 5 = closing, 6 = punctuation, 7 = variable family);
f the math group (font family) where the glyph should be taken from;
ab the slot in the font.
One can use \mathchar<15 bit number> directly or define
\mathchardef<cs>=<15 bit number>
so, for instance, after \mathchardef\with="2026 the command \with is equivalent to typing \mathchar"2026 (or \mathchar8230, for " specifies hexadecimal number).
Typing \char<8 bit number> tells TeX to use the character from the current font in the specified slot. However, when in math mode, \char"ab is equivalent to saying \mathchar"00ab, so ordinary symbol from math group 0 and the same slot. The \chardef command is the counterpart of \mathchardef and, indeed, \& is usually defined as
\chardef\&="26
Small lie: it's \chardef\&=`&, but it's not really important.
There is a big difference between $x \mathchar"2026 y$ and $x \char"26 y$, or, with the definitions above
$x \with y$
$x \& y$

In the first case the & symbol is spaced as it's good for a binary operation, in the second case no space is added, because we're specifying three ordinary symbols.
tex-corewas not appropriate. – Martin Thoma Feb 23 '15 at 15:26\withallowed only in math mode? – Sigur Feb 23 '15 at 15:35\mathchardef(or\DeclareMathSymbol) can only be used in math mode. – egreg Feb 23 '15 at 15:49