To use universal instantiation, you substitute from the outside in. If you are looking to substitute for $y$, we must be starting with
$$(\forall y)(\forall x)[ x > 0 \to (\exists n) [ nx > y]]$$
You are asking about the restriction that prevents the term $nx$ from being substituted for $y$. The restriction on universal generalization, informally put, is that no variable in the term you are substituting can become bound when you perform the substitution. In this case both $n$ and $x$ become bound, so $nx$ is not substitutiable for $y$ here.
For a formal definition of "substitutable", see a logic text, or this question.
There is another way of formalizing universal instantiation, as in the Wikipedia article, so that there is no restriction on the term, but the substitution operation is changed so that if any variable in the new term was going to get bound, then that particular instance of the original variable is not substituted. However, for this variation to be useful, we would want to replace all instances of the original variable, which leads to the same restriction in practice.