The language of (first order) set theory consists of formulas built from
- Any number of variables $x_0,x_1,x_2,\ldots$ (finite for any single formula.)
- $\in$ for membership.
- $=$ for equality, which is constrained to be interpreted as equality in structures (this is called logic with equality. It is possible to discard equality in set theory because it's definable, but that's another subject.)
- Propositional connectives $\neg,\land,\lor,\rightarrow,\leftrightarrow$.
- Quantifiers $\forall,\exists$ over the variables.
- Parentheses for disambiguating evaluation order.
There are variations, but let's stick with this. ZFC has 9 axioms, out of which 7 are single axioms, and 2 are actually schemas (comprehension and replacement). The schemas are based on textual substitiution of a formula in the language to get an instance of an axiom. You're not permitted to extend the language with additional symbols and then substitute a formula with additional symbols in one of the two schemas. That would be an axiom which is not an axiom of ZFC, as the axioms of ZFC only include axioms in the above language.
Now let's consider the unparameterized recursion theorem. The theorem is actually a schema that accepts as input a formula $\psi(x,y)$ whose free variables are among $x,y$. From $\psi$ the theorem constructs another formula $\psi_R(x,y)$ which expresses a recursive function built from $\psi$ in any model of ZF/ZFC in which $\psi$ is 'functional'. Here functional means a model of ZF/ZFC in which for every assignment of $x$, there is at most one assignment of $y$ for which $\psi(x,y)$ is true in the model.
The proof of the recursion theorem takes $\psi$ and builds various formulas from it. One of them is of course $\psi_R$, but there are other auxilary formulas built from $\psi$ used during the proof. One of these is a formula $\chi$ that is substituted into the replacement axiom schema to get an instance of the axiom of replacement.
Where is replacement necessary? I'll describe it informally. In the proof you want the recursive function $G$ to satisfy $G(\beta)=F(G\rvert_\beta)$. This requires showing that $G\rvert_\beta$ is a set. At this point you've already defined sets $G(\gamma)$ for all $\gamma<\beta$. Now, $G\rvert_\beta=\{\langle \gamma,G(\gamma)\rangle:\gamma<\beta\}$ where $\langle,\rangle$ denotes ordered pair. To show this is a set, you need the axiom of replacement. You're replacing every $\beta\in\gamma$ (which is equivalent to saying $\beta<\gamma$) with $\langle \gamma,G(\gamma)\rangle$ and you want to know this is a set.
The way I described it informally it may sound like you need to use the replacement schema once for every ordinal $\beta$, but that is actually not the case. You can do it all with one instance of replacement for all $\beta$. This is tricky. I suggest you look up the proof in Kunen's Set Theory to see how. When using replacement for this, you need to substitute a formula $\chi$ derived from $\psi$ (that describes $F$) into the replacement schema. This is only permitted if $\chi$ is a formula in the base language of set theory without additional symbols.
Now, for your suggestion that you're asking about. What you do there is start with a formula $\psi(z, x, y)$ (For $F$) that has an additional parameter $z$. Then you introduce a new constant symbol into the language $a$, make the substitution $\psi(a, x, y)$, and use the substituted formula in the unparameterized recursion theorem. This is not permitted because the formula is not in the base language of set theory. It leads to an instance of the replacement axiom in the proof that uses this new constant $a$ - and that's not allowed.
Normally when doing proofs, you're allowed to introduce new constants like $a$. It is even necessary for applying rules of inference for quantifiers (universal instantiation, universal generalization, existential instantiation, existential generalization.) But you're not allowed to substitute formulas with additional constants introduced during a proof into an axiom schema. That is where the limitation comes from. It is a special limitation resulting from the use of schemas.