Take the convention that $D_{2n}$ is the dihedral group of order $2n$, and $\text{Dic}_{n}$ is the dicyclic group of order $4n$.
I want to show that if $p$ is a prime congruent to $3 \bmod 4$, $p>3$, then there are only two non-abelian groups of order $4p$ (which therefore must be $D_{4p}$ and $\text{Dic}_{p}$, since these are non-isomorphic and non-abelian. In fact, there will be a total of $4$ groups, since the only abelian ones can be $C_2 \times C_2 \times C_p$, $C_4 \times C_p$).
This reference confirms that for $p=7$, $p=11$, $p=19$ and $p=23$, the smallest $5$ primes with this property, that this result holds. Note it doesn't hold for all primes. For $p=5$, there are certainly more than $2$ non-abelian groups of order $20$, for example.
This page on groupprops confirms the result. May I have a hint as to how to start to prove this?