Assume $\frac{p^2 + 1}{2}$ is not coprime with $\frac{p^{5n}(p^5 - 1)}{2}$, and that $\frac{p^2 + 1}{2} < \frac{p^{5n}(p^5 - 1)}{2}$, so then $\frac{p^2 + 1}{2} \mid \frac{p^{5n}(p^5 - 1)}{2}$.
which means that $\exists q \in \mathbb{N}$ such that:
$$\frac{p^{5n}(p^5 - 1)}{2} = \frac{p^2 + 1}{2} q$$
Then,
$$q = \frac{p^{5n}(p^5 - 1)}{p^2 + 1}$$
This means, $(p^2 + 1) \mid p^{5n}$ and/or $(p^2 + 1) \mid (p^5 - 1)$
So, $\frac{p^{5n}}{(p^2 + 1)} = p^{5n - 2} - p^{5n - 4} - p^{5n - 6}...$
If n is even, there will be a remainder value of $-\frac{1}{p^2 + 1}$, so $(p^2 + 1) \nmid p^{5n}$.
If n is odd, $(p^2 + 1) \nmid p^{5n}$.
So therefore, $(p^2 + 1) \mid (p^5 - 1)$.
However, $(p^5 - 1) = (p^2 +1)(p^3 - p) + (some\ remainder)$.
This means, $(p^2 +1) \nmid p^{5n}$ and $(p^2 + 1) \nmid (p^5 -1)$.
So, $q = \frac{p^{5n}(p^5 - 1)}{(p^2 + 1)}$ is irreducible, and so $q \notin \mathbb{N}$.
We've arrived at a contradiction, and so therefore our original assumption is false, and therefore, both $\frac{p^2 + 1}{2}$ and $\frac{p^5 -1}{2}$ are coprime.
[Addendum]
However, as pointed out by @MAS, what if they have a common factor > 1. So, again, assume both $\frac{p^2 + 1}{2}$ and $\frac{p^{5n}(p^5 - 1)}{2}$ are not coprime and suppose $\exists q,r,s \in \mathbb{N}$ such that $s$ is this common factor and $gcd(q, r) = 1$:
$$\frac{p^2 + 1}{2} = qs$$
$$\frac{p^{5n}(p^5 - 1)}{2} = rs$$
Then,
$$p = \sqrt{2qs - 1}$$
Plugging this into $\frac{p^{5n}(p^5 - 1)}{2}$, we get:
$$ \frac{\sqrt{2qs - 1}^{5n}(\sqrt{(2qs - 1)^5} - 1)}{2}$$
Then as stated, $$\frac{\sqrt{(2qs - 1)^{5n}}(\sqrt{(2qs - 1)^5} - 1)}{2} = rs$$
So,
$$\sqrt{(2qs - 1)^{5n}}(\sqrt{(2qs - 1)^5 - 1)} = 2rs$$
$$(2qs - 1)^{\frac{5n + 5}{2}} - (2qs - 1)^{\frac{5n}{2}} = 2rs$$
Since $s \mid 2rs$, then $s \mid (2qs - 1)^{\frac{5n + 5}{2}} - (2qs - 1)^{\frac{5n}{2}}$
So $s \mid (2qs - 1)^{\frac{5n}{2}}$. This means $\exists v \in \mathbb{N}$ such that
$$vs = (2qs - 1)^{\frac{5n}{2}}$$
Squaring both sides:
$$ (vs)^2 = (2qs - 1)^{5n}$$
$$ (vs)^2 = \sum_{i=0}^{5n}{5n \choose i}(2qs)^{5n-i}(-1)^i$$
$$ (vs)^2 = \sum_{i=0}^{5n - 1}{5n \choose i}(2qs)^{5n-i}(-1)^i + 1$$
While $s \mid (vs)^2$, $s \nmid \sum_{i=0}^{5n - 1}{5n \choose i}(2qs)^{5n-i}(-1)^i + (-1)^{5n}$ because while $s \mid \sum_{i=0}^{5n - 1}{5n \choose i}(2qs)^{5n-i}(-1)^i$, $s \nmid (-1)^{5n}$.
Therefore since $$s \nmid (2qs - 1)^{\frac{5n}{2}}$$
$$s \nmid (2qs - 1)^{\frac{5n + 5}{2}} - (2qs - 1)^{\frac{5n}{2}}$$
And so, since $$s \mid \frac{p^2 + 1}{2}$$, but $$s \nmid \frac{p^{5n}(p^5 - 1)}{2}$$
Therefore, $\nexists s \in \mathbb{N}$ such that $s > 1$ and
$$ s \mid \frac{p^2 + 1}{2}$$
and
$$ s \mid \frac{p^{5n}(p^5 - 1)}{2}$$
So,
$$gcd(\frac{p^2 + 1}{2}, \frac{p^{5n}(p^5 - 1)}{2}) = 1$$
Therefore, $\frac{p^2 + 1}{2}$ and $\frac{p^{5n}(p^5 -1)}{2}$ are coprime.
NB: Corrections are appreciated.
We will have $p = 1 \mod q$ and $p^2 = -1 \mod q$ which contradict.
– sku May 20 '22 at 03:46