In my opinion it's hopeless to get “good looking” expressions involving \vec.
First of all, you should never use $$ and very sparingly \left and \right, which in this case produce oversized delimiters.
\documentclass{article}
\usepackage{amsmath}
\newcommand{\vcc}{\mspace{1mu}}% vector correction
\begin{document}
\begin{gather}
\left|\overrightarrow{a}-\overrightarrow{b}\right|^2=
\left|\overrightarrow{a}\right|^2+\left|\overrightarrow{b}\right|^2
-2\left|\overrightarrow{a}\right|\left|\overrightarrow{b}\right|\cos(\alpha)
\
\left|\vec{a}-\vec{b}\right|^2=
\left|\vec{a}\right|^2+\left|\vec{b}\right|^2
-2\left|\vec{a}\right|\left|\vec{b}\right|\cos(\alpha)
\
\lvert\vec{a}-\vec{b}\rvert^2=
\lvert\vec{a}\rvert^2+\lvert\vec{b}\rvert^2
-2\lvert\vec{a}\rvert,\lvert\vec{b}\rvert\cos(\alpha)
\
\lvert\vec{a}-\vec{b}\vcc\rvert^2=
\lvert\vec{a}\vcc\rvert^2+\lvert\vec{b}\vcc\rvert^2
-2\lvert\vec{a}\vcc\rvert,\lvert\vec{b}\vcc\rvert\cos(\alpha)
\end{gather}
\end{document}
(1) is your input.
(2) is the same, but with \vec instead of \overrightarrow.
(3) uses \lvert and \rvert (which are recommended over the bare |).
(4) adds a tiny space before the closing delimiter; this space is not generally needed after \vec{<letter>}, but it's better when a fence follows, to avoid clashes. See later.
In (3) and (4) I also added \, between the consecutive \rvert and \lvert, so the reader doesn't mistake the symbol for \Vert.

Compare the output of (\vec{a}+\vec{b}) and (\vec{a}+\vec{b}\vcc)

\left\rightjust use\lvert ..\rvert(also don't use$$in latex) – David Carlisle Nov 07 '22 at 22:02\lvert \vec{a} \rvert. – Sigur Nov 07 '22 at 22:17