Naturally we would not be able to detect this effect directly by measuring the size of objects using rulers because any such ruler would be shrinking too. Also cosmological redshift can be explained by either stretching photon wavelengths or increasing energies of absorber atoms.
But I think the following experiment would be able to detect if atoms are shrinking.
Consider two charged objects with mass $m$ and charge $q$ separated by distance $d$.
Let us assume that the system is stable so that the electrostatic repulsion of the charges $q$ is balanced by the gravitational attraction of the masses $m$.
Thus we have:
$$\frac{q^2}{4\pi\epsilon_0 d^2}=\frac{Gm^2}{d^2}\tag{1}$$
Now in natural units ($\hbar=c=4\pi\epsilon_0=1$) the gravitational constant $G=1/M_{Pl}^2$ where $M_{Pl}$ is the Planck mass.
Thus equation $(1)$ becomes:
$$\frac{q^2}{d^2}=\frac{m^2/M_{Pl}^2}{d^2}\tag{2}$$
The balance of electrical and gravitational forces does not depend on the separation $d$ so that finally we have the simple equality
$$q=\frac{m}{M_{Pl}}\tag{3}$$
Now if atomic distances are shrinking then atomic rest-mass energies are increasing with the scale factor $a(t)$. But both $m$ and $M_{Pl}$ expand by the same scale factor $a(t)$ so that equation $(3)$ remains the same.
Thus the separation $d$ of the objects does not change with time.
But if our rulers are shrinking then we will measure the separation distance $d$ apparently increasing.
Does this make sense?
