This is because the subscripts i and j have different depths and you are using automatic scaling.
There are two easy ways to fix this, since you are already using the mathtools and commath packages:
Solution with mathtools (preferred solution†)
Make the \norm symbols the same size by using \DeclarePairedDelimiter from mathtools to declare \norm, and give a size as an optional argument as \norm[\big]{...}. Since you are using the commath package, which itself defines \norm, you can "save" and "undefine" the old definition of \norm before (re)declaring it using mathtools. You should also place the prime outside of \mathbf{...}, otherwise the subscript is placed too far right.
\documentclass{book}
\usepackage{amsmath,amsfonts,amssymb,amsthm}
\usepackage{mathtools}
\usepackage{commath}
\usepackage[sc,osf]{mathpazo}
\let\oldnorm\norm % <-- Store original \norm as \oldnorm
\let\norm\undefined % <-- "Undefine" \norm
\DeclarePairedDelimiter\norm{\lVert}{\rVert}
\begin{document}
\begin{equation}
\mu(\mathbf{A}) = \max_{1\leqslant i,j\leqslant n, i\neq j}\dfrac{|\mathbf{A}'{:,i}\mathbf{A}{:,j}|}{\norm[\big]{\mathbf{A}{:,i}}\norm[\big]{\mathbf{A}{:,j}}}
\end{equation}
\end{document}

or without optional argument (most correct size in my opinion), i.e. with {\norm{\mathbf{A}_{:,i}}\norm{\mathbf{A}_{:,j}}:

Solution with commath
The \norm command from the commath also takes an optional argument, ranging from 0 to 4 where 0 is the smallest size and 4 the biggest. Notice how the spacing before the first \norm is wrong – you will have to manually adjust this with e.g. \; to get a pleasing result.
\documentclass{book}
\usepackage{amsmath,amsfonts,amssymb,amsthm}
\usepackage{mathtools}
\usepackage{commath}
\usepackage[sc,osf]{mathpazo}
\begin{document}
\begin{equation}
\mu(\mathbf{A}) = \max_{1\leqslant i,j\leqslant n, i\neq j}\dfrac{|\mathbf{A}'{:,i}\mathbf{A}{:,j}|}{\norm[1]{\mathbf{A}{:,i}}\norm[1]{\mathbf{A}{:,j}}}
\end{equation}
\end{document}

You should consider using \abs instead of |...| as well.
Same approach as above with mathtools:
\let\oldabs\abs % Store original \abs as \oldabs
\let\abs\undefined % "Undefine" \abs
\DeclarePairedDelimiter\abs{\lvert}{\rvert}
and then use \abs*{...} to get automatic sizing, or with optional argument to manually size \abs[\big]{...}.
For commath the approach is also the same as above: \abs[1]{...}.
†As you can see, the spacing before the first \norm in the denominator with commath (last image) is wrong, so personally I'd recommend using the mathtools approach, as that gives the correct spacing.
commath; look at http://tex.stackexchange.com/search?q=commath+user%3A4427 to see why. – egreg Mar 04 '16 at 09:08\strutor\smashor simply specifying the size explicitly with\biggand the like. No need for package bloat. – The Vee Mar 04 '16 at 13:21