The equation has some notation that is difficult to find the meaning for. It is equation (3) in the paper 'Quaternion Averaging' by F. Landis Markley, et al. on page 3 under 'The Average Quaternion'. Here is the equation (forgive the poor Latex):
Using the definition of the Frobenius norm, the orthogonality of $A(\mathbf{q})$ and $A(\mathbf{q}_i)$, and some properties of the matrix trace (denoted by $\text{Tr}$) gives
$$ \begin{eqnarray} \lVert A(\mathbf{q}) - A(\mathbf{q}_i) \rVert ^2_F && = \text{Tr } \{[A(\mathbf{q}) - A(\mathbf{q}_i)]^T[A(\mathbf{q}) - A(\mathbf{q}_i)]\} \\ && = 6-2 \text{ Tr }[A(\mathbf{q})A^T(\mathbf{q}_i)] \end{eqnarray} $$
Specifically, with reference to $A(\mathbf{q})$, the brackets suggest this is a function, but $A$ would usually be the symbol for a matrix. $\mathbf{q}$ is likely a quaternion, so maybe a quaternion matrix?
Also, with reference to $A(\mathbf{q}_i)$, the $_i$ would be used in summation, but there is a lack of summation in the equation itself that would use it. Elsewhere in the paper it suggests $\mathbf{q}_i$ is a set of n attitude estimates, so a set of quaternions?
Curly braces would usually denote a set, but is it hard to tell whether ${[A(\mathbf{q}) - A(\mathbf{q}_i)]^T[A(\mathbf{q}) - A(\mathbf{q}_i)]}$ is a set.
Likewise, square brackets may denote any number of things, including equivalence class, or the floor, and so on.