I think you just need to read more widely to encounter a broad range of number systems being useful in at least theoretical physics. I'll hyperlink to discussions of applications, but just mention the number systems themselves, as some have multiple applications.
There are uses for $p$-adic numbers, split-complex numbers, dual numbers, quaternions, split-quaternions, dual complex numbers, octonions, split-octonions, dual quaternions, Clifford algebras more generally, and Grassmann numbers. (For that last one, I just linked to the overall article because it frequently mentions the relevance to fermions.)
The prevalence of $\Bbb C$ can be explained, or at least half-explained, from many perspectives; Scott Aaronson has looked into why quantum mechanics runs on them in at least three long but good reads. In particular, he looks at why they $\Bbb C$ may be a sweet spot between $\Bbb R$ and $\Bbb H$.