There are a couple of things wrong with your setup.
\tikzstyle are meant to be declarations which are global. This means that they are like preample statements where you initialize some styles that gets used a lot. Thus you should not put this inside the \begin{tikzpicture}. Also (as Martin correctly states), use the \tikzset or \pgfkeys commands instead, they are much more consistent).
Temporary styles can be created in the \begin{tikzpicture}[<here>] segment of the picture.
baseline is a style affecting the tikzpicture and is thus only used in the \begin{tikzpicture}[baseline].
For a reference about styles and their scoping you can see one of my previous answers: Scopes of styles
Ok, so a quick fix for your solution would be to do the following:
\begin{equation}
\mathcal{G} =
\begin{tikzpicture}[baseline,vertex/.style={anchor=base,
circle,fill=black!25,minimum size=18pt,inner sep=2pt}]
\node[vertex] (G1) at (0,0) {1};
\node[vertex] (G2) at (1,0) {2};
\node[vertex] (G3) at (1,1) {3};
\node[vertex] (G4) at (0,1) {4};
\draw (G1) -- (G2) -- (G3) -- (G4) -- (G1) -- cycle;
\end{tikzpicture}
\end{equation}
which yields:

i.e. an alignment against the circle containing 1.
However, if you wish to align against the center of the structure you should specify the baseline shift for the tikzpicture. That is baseline=<length>.
In your case the middle of the figure can be calculated using the calc library. Here you need to realize two things:
The baseline aligns the base of the line to the placement.
You would rather align against the middle of the line.
This means that you need to calculate the shift according to the height of the line (i.e. include the text height in the shift).
This can be done easily by using this style:
baseline=($(G1.base)!.5!(G4.base)$)
This will yield:

baseline=(current bounding box.center). – Tom Bombadil Oct 03 '12 at 18:55baselinemust be used on thetikzpicturenot on the nodes. Also\tikzstyleis deprecated in favor of\tikzset{vertex/.style=..}. – Martin Scharrer Oct 03 '12 at 19:04