A frankly remarkable set of circumstances lead to a really , really beautiful answer in this situation.
Let me start by saying this : the answer is motivated by the question itself, in that we may use the expression in the question to provide further bounds on $F$ which lead to the expression for the question : which means that nowhere in the working below am I assuming that $F$ is unique in any way. In fact, were I to insist that $F$ not be continuous or something, I'd probably think there's way more than just one solution to this problem.
It remains to be seen, for example, what conditions on $F$ would ensure uniqueness from merely the functional equation, but anyway, the proof is here.
So the point is simple : yes , $F$ satisfies the equation, but what gives linear bounds on $F$ is not the functional equation, but rather its origin : the function which $F$ is supposed to represent.
Let's see how. This is an extension of Problem number 18, A problem seminar, D.J.Newman.
So, we have :
$$
F(x):=\sqrt{ax+(n+a)^2+x\sqrt{a(x+n)+(n+a)^2+(x+n)\sqrt{\cdots}}}
$$
First, we focus on at least providing linear bounds for $F$.
Task : To find constants $C_1<1,C_2>1$ such that $C_1(x+n+a)\leq F(x) \leq C_2(x+n+a)$ for all $x \geq 1$.
For the lower bound,we use the obvious monotonicity of the square root to write :
$$
F(x) \geq \sqrt{x\sqrt{x\sqrt{x\ldots}}} = x
$$
(convert the square-roots to exponents and use the geometric series for the equality, or use a sequence-based definition) and now, for all $x \geq 1$ we have $x+n+a \leq (n+a+1)x$, so we write $F(x) \geq \frac{x+n+a}{n+a+1}$ for all $x \geq 1$.
The other way is going to be very twisty-turny : we use the inequality $a+xb \leq (a+x)b$ for all $a \geq 0,b \geq 1$ , and the monotonicity of the square root to obtain :
$$
F(x) \leq \sqrt{(x(a+1)+(n+a)^2)\sqrt{((x+n)(a+1)+(n+a)^2)\sqrt{\ldots}}} \\
\leq \sqrt{((x+n+a)(a+1)+(n+a)^2)\sqrt{((x+2n+a)(a+1)+(n+a)^2)\sqrt{\ldots}}}
$$
now use the inequality again : $ax+b \leq x(a+b)$ for all $x \geq 1,b \geq 0$ so :
$$
\sqrt{((x+n+a)(a+1)+(n+a)^2)\sqrt{((x+n+a)(a+1)+n(a+1)+(n+a)^2)\sqrt{\ldots}}} \leq \sqrt{(x+n+a)((a+1)+(n+a)^2)\sqrt{(x+n+a)((a+1)(n+1)+(n+a)^2) \sqrt{\ldots}}} \\ \leq (x+n+a)\sqrt{[(a+1)+(n+a)^2] \sqrt{[(a+1)(n+1)+(n+a)^2] \sqrt{\ldots}}}
$$
Using our linear crude bounds again, we get that $(a+1)(n+1)+(n+a)^2 \leq (n+1)((a+1)+(n+a)^2)$, and extremely crudely $(a+1)(kn+1) + (n+a)^2 \leq (n+1)^{2^k}((a+1)+(n+a)^2)$.
Why are we doing all this? It's basically to keep stuff linear, and to make sure that the constant we get on the right hand side is explicit. For this we need to ensure that the square roots simplify to something, and the inequalities reflect this structure.
Once we do that, we get :
$$
(x+n+a)\sqrt{[(a+1)+(n+a)^2] \sqrt{[(a+1)(n+1)+(n+a)^2] \sqrt{\ldots}}} \leq (x+n+a) \sqrt{[(a+1)+(n+a)^2] \sqrt{(n+1)[(a+1)+(n+a)^2]}\sqrt{(n+1)^2[(a+1)+(n+a)^2]\sqrt{\ldots}}} \\ \leq (x+n+a)[(a+1)+(n+a)^2]\sqrt{1\sqrt{(n+1)\sqrt{(n+1)^2 \sqrt{(n+1)^4 \sqrt{\ldots}}}}} \leq (x+n+a)[(a+1)+(n+a)^2]\sqrt{n+1}
$$
And, breathe.
Thus, we have constants $C_1<1,C_2 >1$ such that for all $x\geq 1$, $C_1 (x+n+a) \leq F(x) \leq C_2(x+n+a)$. Let's use a bootstrap argument to bring $C_1,C_2$ arbitrarily close to $1$.
Well, let's start with this : $C_1(x+n+a) \leq F(x) \leq C_2(x+n+a)$ from definition. Now, by definition, $C_1(x+2n+a) \leq F(x+n) \leq C_2(x+2n+a)$, therefore we have :
$$
ax+(n+a)^2 +C_1x(x+2n+a) \leq F(x)^2 = ax+(n+a)^2 + xF(x+n) \leq ax+(n+a)^2 + C_2x(x+2n+a)
$$
Now, we note that $ax+(n+a)^2 +C_1x(x+2n+a) \geq C_1(x+n+a)^2$, and we also have $ax+(n+a)^2 + C_2x(x+2n+a)\leq C_2(x+n+a)^2$. (verifying this step was deeply satisfying, and you have to use the fact that $C_1<1,C_2>1$).
Therefore, following the square root, we get $$
\sqrt{C_1} (x+n+a) \leq F(x) \leq \sqrt{C_2} (x+n+a)
$$
repeating this procedure and taking the limit leads to $F(x)=x+n+a$ for all $n \geq 1$.
Salient features :
Notes :
To rigorously justify that $F$ is continuous, you will have to define a sequence of functions $F_m(x) = \sqrt{ax+(n+a)^2 +xF_{m-1}(x+n)}$ with an appropriate $F_1$. Show that the $F_i$ converge uniformly to a function $F$ (perhaps on some bounded interval) : then , by taking the limit, $F$ satisfies the above recurrence and will be continuous.
To make all the inequalities super-rigorous, you'll have to make sure they hold for the $F_i$, and then take the limit. I haven't done this, but I'm going to leave that to someone else to do.
I believe that I cannot force $F$ to be linearly bounded if I merely assume that it is continuous and satisfies the difference equation (i.e. that it never came from the Ramanujan problem), since I think having only this condition, along with the fact that the control of the functional equation is only from $x$ to $x+n$ (that gap of $n$ is maintained and can't be shortened) tells me that there can be wild misbehaviour of candidate solutions unless differentiability or additional conditions are present. In fact ,even with these conditions, I think the equation will have many solutions.
