Recall that a scalar isn't just any function; it needs to be a function that transforms as a scalar, i.e. one that doesn't transform at all. The general way to do this is to construct an object with all the indices contracted. We start with the general second order term
$$u_{ij} u_{k\ell}$$
and need to contract indices together. The only tensors available are the Kronecker delta $\delta^i_j$ and the volume tensor $\epsilon_{ijk}$. The volume tensor doesn't give us anything: if we just use one we get an odd number of indices, and if we use two they contract together to reduce to Kronecker deltas, i.e.
$$\epsilon_{ijk} \epsilon^{imn} = \delta_j^m \delta_k^n - \delta_j^n \delta_k^m.$$
Using only the Kronecker delta, we can contract $i$ with $j$, so $k$ must be contracted with $\ell$, giving the first term $u_{ii} u_{kk} = u_{ii}^2$. (Here I'm being sloppy with index placement because it doesn't matter.) Otherwise, $i$ can be contracted with $k$ or $\ell$, and it doesn't matter which by symmetry. If $i$ is contracted with $k$ then $j$ is contracted with $\ell$, giving $u_{ij} u_{ij} = u_{ij}^2$, the second term.
More generally, the fact that there are two scalars can be understood by representation theory. Your situation has $SO(3)$ symmetry, and a general symmetric rank two tensor is a six-dimensional representation that decomposes into a scalar (its trace) and a traceless part, which we write as
$$6 = 1 + 5.$$
The quadratic terms are formed from a tensor product of this representation with itself,
$$6 \times 6 = (1 + 5) \times (1 + 5) = 1 + 5 + 5 + 5 \times 5.$$
The first scalar is simply the trace squared, as we've seen. Now, to decompose $5 \times 5$, we use the same method you might already know from quantum mechanics. (Indeed, to translate this to spin, just subtract one from every number and divide by two.) Then
$$5 \times 5 = 1 + 3 + 5 + 7 + 9.$$
In total there are two factors of $1$ and hence two scalars.
Now we come back to the first point: why are $\delta^i_j$ and $\epsilon_{ijk}$ the only tensors available? These come from the two parts of the definition of $SO(3)$. The "orthogonal" part means that the Euclidean inner product is preserved, giving the metric $\delta^i_j$. The "special" part means that the volume is preserved, giving the volume $\epsilon_{ijk}$. Nothing else is preserved, so you can't contract with any other tensors -- those would change as well under rotation.