Let $ f $ be a polynomial of degree $ 3 $ and having rational coefficients. Prove that, if there exist two distinct nonzero rational numbers $ a,b $ and two roots $ x_1,y_1 $ of $ f $ such that $ ax_1+by_1 $ is rational, then all roots of $ f $ are rational.
Let $f(x)=a_3(x-x_1)(x-y_1)(x-\alpha)$. Notice that if we get $x_1,y_1$ to be rational then $\alpha$ too is rational. Now, assume $x_1$ is irrational, then by the given conditions, we get $y_1$ too is irrational. How do i proceed from here?
First, it's given that
$$f(x) = r_3x^3 + r_2x^2 + r_1x + r_0 \tag{1}\label{eq1A}$$
where $r_i$ are all rational and $r_3 \neq 0$. Also, for some non-zero and non-equal rational $a$ and $b$, there's a rational $r_4$ where
$$ax_1 + by_1 = r_4 \;\;\to\;\; y_1 = \frac{r_4 - ax_1}{b} \tag{2}\label{eq2A}$$
Since $f(x)$ is an odd-degree polynomial, it has at least one real root. The complex conjugate root theorem states that if $f(x)$ has any non-real complex roots, they are of the form $d\pm ei$ for real $d$ and $e$. From the first part of \eqref{eq2A}, if one or both of $x_1$ or $y_1$ were these roots, then the LHS would be a non-real complex number. Thus, all $3$ roots must be real.
As Mark Bennet's comment indicates, WLOG we can have $r_3 = 1$ (since dividing by the leading non-zero coefficient doesn't change the roots), and that Vieta's formulas should be used. This gives that, if $\alpha$ is the third root, then
$$x_1 + y_1 + \alpha = -r_2 \;\;\to\;\; \alpha = -r_2 - x_1 - y_1 \tag{3}\label{eq3A}$$
$$x_{1}y_{1} + x_{1}\alpha + y_{1}\alpha = r_1 \;\;\to\;\; x_1(y_1 + \alpha) + y_1\alpha = r_1 \tag{4}\label{eq4A}$$
$$x_{1}y_{1}\alpha = -r_0 \tag{5}\label{eq5A}$$
From \eqref{eq2A}, as you've already stated, we have that $x_1$ and $y_1$ are either both rational or irrational. From \eqref{eq3A}, if both are rational, so is $\alpha$, so all $3$ are rational. Thus, assume that both $x_1$ and $y_1$ are irrational.
Using \eqref{eq2A} and \eqref{eq3A} in the second part of \eqref{eq4A} results in
$$\begin{equation}\begin{aligned} x_1(-r_2 - x_1) + \left(\frac{r_4 - ax_1}{b}\right)\left(-r_2 - x_1 - \frac{r_4 - ax_1}{b}\right) & = r_1 \\ b^{2}x_1(-r_2 - x_1) + (r_4 - ax_1)(-br_2 - bx_1 - (r_4 - ax_1)) & = b^{2}r_1 \\ (-b^{2} + ab - a^{2})x_1^{2} + ((2a - b)r_4 + abr_2)x_1 + (-br_{2}r_{4} - r_{4}^{2} - b^{2}r_1) & = 0 \end{aligned}\end{equation}\tag{6}\label{eq6A}$$
This is a quadratic equation with rational coefficients, with the first one being non-zero. Thus, multiplying by the resulting common denominator will convert it into one with integers coefficients. Next, using the quadratic formula, we get that
$$x_1 = c_1 + d_1\sqrt{e_1} \tag{7}\label{eq7A}$$
where $c_1$ and $d_1$ are rationals, with $d_1 \neq 0$, and $e_1$ is a square-free integer $\gt 1$. Similarly, using \eqref{eq7A} in \eqref{eq2A}, we also have
$$y_1 = c_2 + d_2\sqrt{e_1} \tag{8}\label{eq8A}$$
where $c_2$ and $d_2$ are rationals, with $d_2 \neq 0$.
Note that \eqref{eq2A} minus $a$ times \eqref{eq3A} gives $(b - a)y_1 - a\alpha = r_4 + ar_2 \;\to\; y_1 = \frac{a\alpha + r_4 + ar_2}{b - a}$. If $\alpha$ were rational, then $y_1$ would also be rational, so $\alpha$ must be irrational. Also, from the RHS of \eqref{eq3A}, along with the results above, we have that
$$\alpha = c_3 + d_3\sqrt{e_1} \tag{9}\label{eq9A}$$
where $c_3$ and $d_3$ are rationals, with $d_3 \neq 0$. Thus, using \eqref{eq7A}, \eqref{eq8A} and \eqref{eq9A} in \eqref{eq3A} we get the coefficient of $\sqrt{e_1}$ must be $0$, i.e.,
$$d_1 + d_2 + d_3 = 0 \;\;\to\;\; d_3 = -d_1 - d_2 \tag{10}\label{eq10A}$$
Similarly, from \eqref{eq2A}, we have
$$ad_1 + bd_2 = 0 \tag{11}\label{eq11A}$$
As Mark Bennet's answer is possibly alluding to, we can use what's asked about and proven in The radical conjugate roots theorem, e.g., from \eqref{eq7A}, with $x_1 = c_1 + d_1\sqrt{e_1}$ being a root, then $c_1 - d_1\sqrt{e_1}$ is also a root. This can also be seen since, from \eqref{eq1A}, we get $f(x_1) = r_5 + r_6\sqrt{e_1}$ for some rational $r_5$ and $r_6$. Since $x_1$ is a root, we get $r_5 = r_6 = 0$. Note that we then also have $f(c_1 - d_1\sqrt{e_1}) = r_5 - r_6\sqrt{e_1} = 0$. Thus, $c_1 - d_1\sqrt{e_1}$ is another root. However, it can't be $y_1$ since $d_2 = -d_1$ in \eqref{eq11A} gives $a = b$, which isn't allowed. Thus, the other root must be $\alpha$ instead, so $d_3 = -d_1$. However, \eqref{eq10A} then gives that $d_2 = 0$, with \eqref{eq11A} showing that $d_1 = 0$ also, so $d_3 = 0$ as well. Thus, \eqref{eq7A}, \eqref{eq8A} and \eqref{eq9A} give that $x_1$, $y_1$ and $\alpha$ are all rational.
This is a faster and simpler way to finish the solution than what I originally did, as shown below. Nonetheless, for posterity and anybody who's interested, I'm leaving this longer and more convoluted technique here.
Next, \eqref{eq11A} minus $a$ times \eqref{eq10A} gives
$$(b - a)d_2 - ad_3 = 0 \;\;\to\;\; d_2 = \left(\frac{a}{b - a}\right)d_3 \tag{12}\label{eq12A}$$
Similarly, we get
$$d_1 = \left(\frac{-b}{b - a}\right)d_3 \tag{13}\label{eq13A}$$
Squaring both sides of the first part of \eqref{eq3A} gives
$$x_1^2 + y_1^2 + \alpha^2 + 2(x_{1}y_{1} + x_{1}\alpha + y_{1}\alpha) = r_2^2 \tag{14}\label{eq14A}$$
Using \eqref{eq4A} shows that $x_1^2 + y_1^2 + \alpha^2$ is rational. Thus, substituting \eqref{eq7A}, \eqref{eq8A} and \eqref{eq9A}, the resulting coefficient of $\sqrt{e_1}$ must be $0$, so by also using \eqref{eq12A} and \eqref{eq13A}, we get
$$\begin{equation}\begin{aligned} c_{1}d_{1} + c_{2}d_{2} + c_{3}d_{3} & = 0 \\ c_{1}\left(\frac{-b}{b - a}\right)d_3 + c_2\left(\frac{a}{b - a}\right)d_3 + c_{3}d_{3} & = 0 \\ c_{1}(-b) + c_2(a) + c_3(b - a) & = 0 \\ (-c_{1} + c_{3})b + (c_2 - c_3)a & = 0 \\ (c_3 - c_2)a & = (c_{3} - c_{1})b \end{aligned}\end{equation}\tag{15}\label{eq15A}$$
Substituting each of the $3$ roots into \eqref{eq1A}, summing the results and collecting the powers of each root, then using that the sum of roots and the sum of the squares of the roots are both rational, we also have the sum of the cubes of the roots (i.e., $x_{1}^3 + y_{1}^3 + \alpha^3$) is rational. Once again, substituting \eqref{eq7A}, \eqref{eq8A} and \eqref{eq9A}, the resulting coefficient of $\sqrt{e_1}$ must be $0$, so along with \eqref{eq10A}, we get
$$\begin{equation}\begin{aligned} 3c_1^{2}d_1 + 3c_2^{2}d_2 + 3c_3^{2}d_3 + d_1^3 + d_2^3 + d_3^3 & = 0 \\ 3c_1^{2}d_1 + 3c_2^{2}d_2 + 3c_3^{2}(-d_1 - d_2) + d_1^3 + d_2^3 + (-d_1 - d_2)^3 & = 0 \\ 3d_1(c_1^2 - c_3^2) + 3d_2(c_2^2 - c_3^2) - 3d_{1}^{2}d_{2} - 3d_{1}d_{2}^2 & = 0 \\ (c_1 - c_3)(c_1 + c_3)d_1 + (c_2 - c_3)(c_2 + c_3)d_2 - d_{1}d_{2}(d_1 + d_2) & = 0 \end{aligned}\end{equation}\tag{16}\label{eq16A}$$
From \eqref{eq10A} in the first line of \eqref{eq15A} gives
$$c_{1}d_{1} + c_{2}d_{2} + c_{3}(-d_1 - d_2) = 0 \;\;\to\;\; (c_1 - c_3)d_1 = (c_3 - c_2)d_2 \tag{17}\label{eq17A}$$
Next, with \eqref{eq17A} and \eqref{eq13A} in \eqref{eq16A},
$$\begin{equation}\begin{aligned} (c_3 - c_2)(c_1 + c_3)d_2 + (c_2 - c_3)(c_2 + c_3)d_2 - d_{1}d_{2}(d_1 + d_2) & = 0 \\ (c_3 - c_2)(c_1 + c_3) - (c_3 - c_2)(c_2 + c_3) - d_1(d_1 + d_2) & = 0 \\ (c_3 - c_2)(c_1 + c_3 - c_2 - c_3) & = d_1(d_1 + d_2) \\ (c_3 - c_2)(c_1 - c_2) & = -d_{1}(d_3) \\ (c_3 - c_2)(c_1 - c_2) & = \left(\frac{b}{b - a}\right)d_3^2 \end{aligned}\end{equation}\tag{18}\label{eq18A}$$
Multiplying both sides by $a$ and using \eqref{eq15A} gives that
$$\begin{equation}\begin{aligned} (c_3 - c_2)a(c_1 - c_2) & = \left(\frac{ab}{b - a}\right)d_3^2 \\ (c_3 - c_1)b(c_1 - c_2) & = \left(\frac{ab}{b - a}\right)d_3^2 \\ (c_3 - c_1)(c_1 - c_2) & = \left(\frac{a}{b - a}\right)d_3^2 \\ \end{aligned}\end{equation}\tag{19}\label{eq19A}$$
Now, \eqref{eq18A} minus \eqref{eq19A} gives
$$(c_3 - c_2 - (c_3 - c_1))(c_1 - c_2) = d_3^2 \;\;\to\;\; (c_1 - c_2)^2 = d_3^2 \;\;\to\;\; c_1 - c_2 = \pm d_3 \tag{20}\label{eq20A}$$
If $c_1 - c_2 = d_3$, then using this in \eqref{eq18A}, and with \eqref{eq13A}, we have
$$(c_3 - c_2)d_3 = \left(\frac{b}{b - a}\right)d_3^2 \;\;\to\;\; c_3 - c_2 = -d_1 \tag{21}\label{eq21A}$$
Similarly, if $c_1 - c_2 = -d_3$, we'd get $c_3 - c_2 = d_1$, so in either case, this results in
$$(c_1 - c_2)(c_3 - c_2) = -d_{1}d_{3} \tag{22}\label{eq22A}$$
For the last time, substituting \eqref{eq7A}, \eqref{eq8A} and \eqref{eq9A} into \eqref{eq5A}, the resulting coefficient of $\sqrt{e_1}$ must be $0$, so along with \eqref{eq10A}, \eqref{eq17A} and \eqref{eq22A}, we get
$$\begin{equation}\begin{aligned} c_{1}c_{3}d_{2} + c_{2}c_{3}d_{1} + c_{1}c_{2}d_{3} + d_{1}d_{2}d_{3}e_1 & = 0 \\ c_{1}c_{3}d_{2} + c_{2}c_{3}d_{1} + c_{1}c_{2}(-d_1 - d_2) & = -d_{1}d_{2}d_{3}e_1 \\ c_{2}(c_3 - c_1)d_{1} + c_{1}(c_3 - c_2)d_{2} & = -d_{1}d_{2}d_{3}e_1 \\ c_{2}(-(c_3 - c_2))d_{2} + c_{1}(c_3 - c_2)d_{2} & = -d_{1}d_{2}d_{3}e_1 \\ -c_{2}(c_3 - c_2) + c_{1}(c_3 - c_2) & = -d_{1}d_{3}e_1 \\ (c_{1} - c_{2})(c_3 - c_2) & = -d_{1}d_{3}e_1 \\ -d_{1}d_{3} & = -d_{1}d_{3}e_1 \\ 1 & = e_1 \end{aligned}\end{equation}\tag{23}\label{eq23A}$$
This contradicts that $x_1$ and $y_1$ are both irrational. In fact, it explicitly shows from \eqref{eq7A}, \eqref{eq8A} and \eqref{eq9A} that $x_1$, $y_1$ and $\alpha$, i.e., all $3$ roots, are rational.
Since a full answer has been proposed, here is something simpler.
Note that if $x_1$ and $y_1$ are roots of a rational quadratic $x^2-px+q=0$ then $x+y=p$ is a rational relation between them but has (in the terms of the question) $a=b=1$ and the only rational relations between $x_1$ and $y_1$ would then have $a=b$ (this condition is therefore important).
Since $ax_1+by_1$ is rational (not sure why the subscripts are necessary - they lead to overly complicated notation) we have that $y_1=cx_1+d$ for $c$ and $d$ rational. The conditions give $c\neq 0,-1; d\neq 0$.
By Vieta relations we know that the sum of the roots is rational so that the third root $z_1=ex_1+f$ for $e$ and $f$ rational. Also $e\neq 0$.
Then the product of the roots is rational so that $x_1(cx_1+d)(ex_1+f)=r$ and the sum of the pairs of products of the roots is rational $x_1(cx_1+d)+x_1(ex_1+f)+(cx_1+d)(ex_1+f)=s$ with $r$ and $s$ rational. The first of these is a cubic with rational coefficients (why can it not reduce to a quadratic?). The second of these equations is at most a quadratic for $x_1$ having rational coefficients.
That should give you enough of a start - what can you say about the quadratic?
$\color{#0a0}{\rm \bf{Another \thinspace\thinspace proof \thinspace\thinspace .}}$
Let $P(x):=x^3+px^2+qx+r$ where $(p,q,r)\in\mathbb Q^{3}$ and $x,y,z$ be the roots of $P(x)$, then define the rational number $c$ as $c:=ax+by$ or equivalently $n:=x+my$, where $n=\dfrac ca$ and $m=\dfrac ba\in\mathbb Q\setminus \{0,1\}$ .
Note that if $x,y\in\mathbb C\setminus \mathbb R$ , then we have $x=\alpha + \beta i$ and $y=\alpha - \beta i$, where $\alpha\in\mathbb R \wedge \beta\in\mathbb R\setminus\{0\}$ . This leads to :
$$ \begin{align}n:&=\alpha +\beta i + m \left(\alpha-\beta i\right)\\ &=\small{\alpha(m+1)+i\beta\underbrace{(1-m)}_{\color{#c00}{\thinspace ≠\thinspace 0}}\not\in\mathbb R}\end{align} $$
which yields a contradiction . Also note that, the case $x\in\mathbb R \wedge y\in\mathbb C\setminus\mathbb R$ is trivially impossible . Therefore, we conclude that $(x,y)\in\mathbb R^{2}$ .
So if $(x,y)\in\mathbb Q^{2}$, then Vieta's formulas tell $z=-p-x-y\in\mathbb Q$ . Wlog, suppose that $x\in\mathbb Q$ and $y\not\in\mathbb Q$ . Since $c\in\mathbb Q$, then there does not exist such $(a,b)\in\mathbb Q^{2}$ in this case too .
Finally, the last case is $(x,y)\not\in\mathbb Q^{2}$. We have :
$$ \begin{align}&\begin{cases}x+my=n\\ x+y=-p-z\end{cases}\\\\ \implies &y(m-1)=(n+p)+z\end{align} $$
which implies $z\not\in\mathbb Q$ . This means that, $P(x)$ is irreducible over $\mathbb Q[x]$. Then since $x,y$ are the roots of $P(x)$, we have $P(n-my)=0$, which is a cubic equation respect to $y$ . Finally, without using any algebraic expansions, we see that putting $y^3=-\left(py^2+qy+r\right)$ in $P(n-my)=0$ leads to the quadratic equation with the rational coefficients respect to $y$, which contradicts the irreducibility of $P(y)$ and this completes the proof .
So we have WLOG $f(x)$ is monic and $x_1,y_1$ are both irrational.
►If $\alpha$ is irrational then its degree $d(\alpha)$ is $2$ or $3$.
If $d(\alpha)=2$ then $x_1$ or $y_1$ is quadratic conjugate of $\alpha$; let it be $x_1$ so $f(x)=(x^2+rx+s)(x-y_1)$ and $y_1$ is rational. Contradiction.
►►If $d(\alpha)=3$ then $x_1$ and $y_1$ are cubic conjugates of $\alpha$ so $ax_1+by_1$ cannot be rational for all distinct rationals $a,b$.
(In fact one has $\alpha, x_1, y_1\in\mathbb Q(\theta)$ where $\theta$ is certain irrrational cubic and $$x_1=r_1\theta^2+r_2\theta+r_3\\y_1=r_4\theta^2+r_5\theta+r_6$$ where $r_i;\space\space1\ge i\ge6$ is rational so $ax_1+by_1$ cannot be rational. Contradiction)
.
