You problem lies in "$n$".
Writing the limit theorem precisely:
For any positive integer $n$, $$\lim_{x\to a}\frac{x^{n}-a^{n}}{x-a}=na^{n-1} \tag{A}\label{A}$$ where $a$ is any non-zero real number.
Now, for proving this theorem after this step: $$\lim_{h\to 0}\frac{a^n(1+\frac{h}{a})^{n}-a^{n}}{h}$$
and after the expansion of $(1+\frac{h}{a})^n$ you are correct to say that "higher-order terms become zero anyways (after inputting h=0). So, why does $|\frac{h}{a}|$ need to be less than 1?"
Indeed $|\frac{h}{a}|<1$ is not necessary for this proof.
Also $|\frac{h}{a}|<1$ does not need to be true for us to expand $(1+\frac{h}{a})^n$ using the Binomial theorem when $n$ is positive integer
But perhaps you are trying to show that $(\ref{A})$ holds even when $n$ is a rational number and $a$ is positive.
Now, the binomial expansion, $$(1+x)^n= 1+ nx + \frac{n(n-1)}{ 2!}x^2 + \frac{n(n-1)(n-2)}{3!}x^3+\cdots \tag{B}\label{B}$$ is true for all rational number $n$ (and has infinite no. of terms when $n \in \mathbb{Q} \setminus \mathbb{W}$) only when $|x|<1$ and certainly in this case, $|\frac{h}{a}|<1$ is true, so yes $(\ref{A})$ holds even when $n$ is rational.
Note that, the binomial expansion $(\ref{B})$ is true when $n$ is a positive integer (and has exactly $n+1$ terms) regardless of $|x|$ being less than or greater than or equal to $1$.