How to symbolically solve the minimization problem in simple linear regression

Question

I want to use mathematica to symbolically solve the minimization problem in simple linear regression:

$${\text{Find }}\text{arg}\min _{\alpha ,\,\beta }Q(\alpha ,\beta ),\qquad {\text{for }}Q(\alpha ,\beta ) =\sum _{i=1}^{n}(y_{i}-\alpha -\beta x_{i})^{2},$$

where $y_i$, $x_i$, and $n$ are symbolic but not specific numbers.The expected answer would be something like

$$\begin{align} \hat {\beta }&={\frac {\sum _{i=1}^{n}(x_{i}-{\bar {x}})(y_{i}-{\bar {y}})}{\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}}},\\ {\hat {\alpha }}&={\bar {y}}-{\hat {\beta }}\,{\bar {x}} \end{align}$$

Can someone show some code example for doing this? Thanks in advance!

Answer 1

The following is an adaptation of my answer to this question, which focused on the 3D linear least-squares problem.

(* Rules to get constants out of sums (or integrals etc) *)
outrules = {
   Sum[f_ + g_, it : {x_Symbol, __}] :> Sum[f, it] + Sum[g, it],
   Sum[c_ f_, it : {x_Symbol, __}] :> c Sum[f, it] /; FreeQ[c, x],
   Sum[c_, it : {x_Symbol, __}] :> c Sum[1, it] /; FreeQ[c, x]
   };

(*Generate the sum of squares*)
Sum[Expand[(y[i] - a x[i] - b)^2], {i, 1, n}];

(*Calculate the derivatives*)
Grad[%, {a, b}];

(*Use the linearity property of sums*)
Distribute /@ %;

(*Pull out any constants from summations*)
% //. outrules;

(*Set the derivatives equal to zero to generate a system of equations*)
Simplify[Thread[% == 0]];

(*Solve for the a, b parameters*)
Solve[%, {a, b}] // FullSimplify

These sums are not yet expressed as a function of $(\bar{x},\bar{y})$, i.e. the average values of the $(x_i,y_i)$, respectively, as you have them in your question, but that should be a question of some algebraic transformations. Some such transformations are reported e.g. in this MathWorld document of least-squares fitting.