Is there a proper subgroup closed under automorphism?

Question

While solving homework for my Modern Algebra course, I came across this problem:

If $H < G$ and $\forall \sigma \in \mathrm{Aut}(G), \; \sigma[H] = H$, then $H \lhd G$

While the problem is pretty trivial, I feel like if a subgroup is closed under automorphism, then not only $H \lhd G$ but $H = G$ will hold, since $H$ should also be closed under permutations like $\sigma = (1)(2, 3, \cdots, |G|)$. Is my thought correct?

Answer 1

The subgroups $H$ in the problem is called a characteristic subgroup, and the problem is asking whether characteristic subgroups are normal. This is true, by considering the inner automorphisms.

The question then asks: are characteristic subgroups always the whole group?

No. In particular, the trivial subgroup is always characteristic.

There are also often non-trivial examples. For example, it is a nice exercise to prove that every subgroup generated by words of any specified form will be characteristic. Examples here include words of the form $g^{-1}h^{-1}gh$ (this is the derived subgroup, and here the quotient group is abelian), and also words of the form $g^n$ for some fixed exponent $n$ (here the quotient group has exponent $n$).

Answer 2

There is a hierarchy of subsets of a group $G$:

1. Subsets. Just asking that it be contained in $G$.
2. Subgroups. We ask that they be groups with the induced operation.
3. Normal subgroups. A subgroup $H$ such that $gHg^{-1}=H$ for all $g\in G$; equivalently, a subgroup $H\leq G$ such that $\varphi(H)=H$ for all inner automorphisms of $G$.
4. Characteristic subgroups. A subgroup $H$ such that $\varphi(H)=H$ for all automorphisms of $G$ (the subgroups you are considering).
5. Fully invariant subgroups. A subgroup $H$ such that $\varphi(H)\subseteq H$ for all endomorphisms $\varphi\in\mathrm{End}(G)$.
6. Verbal subgroups. Subgroups generated by the values of a collection of words; equivalently, specify a subset $V$ of a free group $F$, and consider the subgroup $\langle f(V)\mid f\colon F\to G\text{ is a homomorphism}\rangle$. These have the further property that if we denote the subgroup of $G$ corresponding to $V$ by $V(G)$, then for any homomorphism $\varphi\colon G\to M$ we have $\varphi(V(G))\subseteq V(M)$.

If $H$ is a subset of type $k$ above, then it is also of all types $\ell\leq k$. Thus, a fully invariant subgroup is characteristic is normal is a subgroup is a subset.

None of the concepts coincide: there are subsets that are not subgroup, subgroups that are not normal, normal subgroups that are not characteristic, characteristic subgroups that are not fully invariant, and fully invariant subgroups that are not verbal. Moreover, we can always find examples where the subgroups in question are proper and nontrivial.

Examples.

1. A subset that is not a subgroup: take $G=\{1,x\}$ be cyclic of order $2$, and take $S=\{x\}$.

2. A subgroup that is not normal: Smallest example is to take $G=S_3$, and let $H=\{e,(12)\}$.

3. A normal subgroup that is not characteristic: Let $G=C_2\times C_2$ be the Klein $4$-group. Any subgroup of order $2$ is normal (because $G$ is abelian), but none of them are characteristic, since the automorphism of $G$ that sends $(x,e)$ to $(x,x)$ and $(e,x)$ to $(x,e)$ shuffles the three subgroups of order $3$.

4. A characteristic subgroup that is not fully invariant: Take $G=S_3\times C_3$; the center of this group is $\{e\}\times C_3$, and it is not hard to verify that any automorphism of a group must send the center to the center. So the subgroup $\{e\}\times C_3$ is characteristic. However, there is an endomorphism of $G$ that sends $(\tau,x^n)$ to $(\sigma^n,e)$, where $\tau$ is an arbitrary element of $S_3$, and $\sigma=(123)$. This endomorphism does not send $\{e\}\times C_3$ to itself, so this is a characteristic subgroup that is not fully invariant.

5. A fully invariant subgroup that is not verbal. (In a free group, every fully invariant subgroup is verbal, but in general groups this need not hold, as this example shows). This one is a bit tricky, but here is the smallest example: let $G$ be the semidihedral group of order $16$, $$G=\langle x,s\mid x^8=s^2=e, sx=x^3s\rangle.$$ This group contains a copy of the dihedral group of order $8$, generated by $x^2$ and $s$, since $sx^2 = x^6s=(x^2)^{-1}s$. This subgroup can be shown to be fully invariant but not verbal.