I would say that your statement "most insulators and semiconductors seem to expand less than metals when exposed to heat" is simply false. Consider that rubber, plastics, and other organic insulators have huge thermal expansion coefficients compared to many metals. Ice, an insulator, has a 5x larger thermal expansion coefficient than platinum, a metal. Glass and titanium have roughly the same coefficient.
I think your observation is mostly limited to Silicon carbide, diamond, etc. where the lattice is just incredibly strong.
It is also worth mentioning that you can measure the average lattice spacing of any material at a finite temperature to many, many decimal places. All temperature does is broaden those positions, but you can still measure the center of the broadened peak to very high precision.