Answer: For A and B commuting then we have $$\color{red}{\det \left(\begin{matrix} A& -B\\B&A \end{matrix}\right)= \bigg|\det(A+iB)\bigg|^2 }$$
Similarly like here For every $a \in \mathbb{R}$ evaluate $ \lim_{n \to \infty}\left(\begin{smallmatrix} 1&\frac{a}{n}\\\frac{-a}{n}&1\end{smallmatrix}\right)^{n}.$
by putting ourselves in the complex plan where we identify
$$ 1 \equiv \left(\begin{matrix} 1& 0\\0&1 \end{matrix}\right)~~~\text{and}~~~~ i \equiv \left(\begin{matrix} 0& -1\\1&0 \end{matrix}\right)\implies M=\left(\begin{matrix} A& -B\\B&A \end{matrix}\right)= A+iB$$
Now assume that $A$ is invertible (The case where A is not invertible will follows by density argument ): then from this
If A and B commute we obtain
$$\det \left(\begin{matrix} A& -B\\B&A \end{matrix}\right)=\det(A+BA^{-1}B)\det(A)= \det(A^2+BA^{-1}BA)=\det(A^2+B^2)$$
But we have, $$\color{red}{|A+iB|^2 =(A+iB)(A-iB)= A^2+B^2 }$$
And hence since $|z|^2=z\bar z$ we obtain , $$\det( A^2+B^2)=\det\left((A+iB)\cdot(A-iB)\right)=\det( A+iB)\cdot\det( A-iB) \\=\det( A+iB)\cdot\overline{\det( A+iB)}= \bigg|\det(A+iB)\bigg|^2$$