Although there is not a unique solution to your question, one solution is obtained as follows.
(-1 + 3 Cos[A]^2) (-1 + 3 Cos[a]^2) + Cos[B - b] Sin[2 A] Sin[2 a] +
Cos[2 (B - b)] Sin[A]^2 Sin[a]^2
First, expand functions involving double angles.
% /. Sin[2 z_] -> 2 Sin[z] Cos[z] /. Cos[2 z_] -> 1 - 2 Cos[z]^2
(* (-1 + 3 Cos[a]^2) (-1 + 3 Cos[A]^2) + 4 Cos[a] Cos[A] Cos[b - B] Sin[a] Sin[A] +
(-1 + 2 Cos[b - B]^2) Sin[a]^2 Sin[A]^2 *)
Next designate the new variable as, for instance, x, and solve for Cos[b - B] as a function of x.
Flatten@Solve[Cos[A] Cos[a] + Cos[B - b] Sin[A] Sin[a] == x, Cos[B - b]]
(* {Cos[b - B] -> (x - Cos[a] Cos[A]) Csc[a] Csc[A]} *)
Finally, eliminate Cos[b - B] from the original expression.
Collect[%% /. %, x, FullSimplify]
(* 2 x^2 + 1/2 Cos[2 a] (1 + 3 Cos[2 A]) - Sin[A]^2 *)
as requested. Note, however, that a or A also could have been eliminated, although the resulting expression would have been much more complicated.