Let $A$ and $B$ be both square matrices with the same dimension. Prove that the following is correct:
$S_p(AB)=S_p(BA)$ , where $S_p$ is the set of eigenvalues.
I am really confused by how I am supposed to find the eigenvalues of these unknown matrices. So if anyone could explain it a bit, I would really appreciate it.
It's not clear what you mean by "finding the eigenvalues of unknown matrices", but clearly you're misunderstanding how these proofs are meant to go. In any case: to show that two matrices have the same eigenvalues, the most convenient approach is often to show that they have the same characteristic polynomial.
Approach 1: Using either Sylvester's determinant identity or something equivalent, one can show that for all $t \neq 0$, we have $$ \det(tI - AB) = \det(tI - BA) $$ if the polynomials are equal for all $t \neq 0$, then they must be the same polynomial.
Approach 2: Note that $AB$ is similar to $BA$ whenever $B$ is invertible (why?). It follows that $\det(t I - BA) = \det(t I - AB)$. When $B$ is not invertible, we note that $$ \det(t I - BA) = \lim_{\epsilon \to 0} \det(t I - (B + \epsilon I)A) = \lim_{\epsilon \to 0} \det(t I - A(B + \epsilon I)) = \det(t I - AB) $$ since $B + \epsilon I$ is invertible when $|\epsilon|$ is sufficiently small and non-zero.
