Not sure how useful this is, but a translation of this to the language of groups via Galois correspondence would go as follows.
Let $K$ be the splitting field of the minimal polynomial $m(x)$ of $\alpha$. IOW the normal closure of $\Bbb{Q}[\alpha]$. Let $G$ be the Galois group $Gal(K/\Bbb{Q})$,
and $G_\alpha$ the stabilizer of $\alpha$ when $G$ is viewed as a group of permutations of the zeros of $m(x)$. Let $\sigma$ be the restriction of complex conjugation to $K$.
Galois correspondence then tells us that $\Bbb{Q}[\alpha]\cap\Bbb{R}$ is the fixed field of the group $H=\langle G_\alpha,\sigma\rangle$. Thus
$$
\Bbb{Q}[\alpha]\cap\Bbb{R}=\Bbb{Q}\Leftrightarrow H=G.
$$
So for example if $\alpha$ is non-real and $G_\alpha$ is a maximal subgroup of $G$, then the condition is satisfied. This is because $\sigma\notin G_\alpha$, and there are no subgroups properly between $G_\alpha$ and $G$. This happens for example when $\alpha$ is a non-real root of an irreducible quartic with Galois group $S_4$, because in that case $G_\alpha\cong S_3$ is a maximal subgroup.
If $[Q[\alpha]:\Bbb{Q}]$ is a prime, then $G_\alpha$ is automatically a maximal subgroup of $G$ also. So your observation fits into this as well.