What Rudin is after is to choose $q=p+\epsilon$, such that $\epsilon >0$, $q^2<2$ and $q$ is rational. This way he will have a $q>p$ and also have $q \in A$ (similar treatment can be done for $B$).
So he wants $q^2-2 <0$. This gives a quadratic inequality $\epsilon^2+2p\epsilon +(p^2-2)<0$. The quadratic equation $\epsilon^2+2p\epsilon +(p^2-2)=0$ has two roots namely, $-p+\sqrt{2}$ and $-p-\sqrt{2}$, with former being a positive root.
So Rudin's job was to pick a number $\epsilon \in (0,-p+\sqrt{2})$ range that satisfies all the requirements stated above. The biggest hurdle is to get $\epsilon$ such that $q$ is rational. So he went for a simple trick (which I believe he would have borrowed from continued fraction expansion for $\sqrt{2}$) to write this $\epsilon$ in the form of $\epsilon=(2-p^2)x$ and then choose $x$ wisely.
He went for a simple choice for $x$ in the form of $x=\frac{1}{p+k}$, where $k$ is any integer greater than or equal to $2$. And Rudin picked $k=2$!!
I hope this gives you some insight as to how that magical expression was chosen.
With Rudin's book you will encounter more such situations, where expressions will seem to be pulled from thin air, but the fun part is to unravel those.