0

2.2 Tikhonov Reqularization

Tikhonov regularization, named for Russian mathematician Andrey Tikhonov, attempts to fix the issue that arises when the least squares method is used with an ill-posed inverse problem by adding an additional constraint on the minimization: $$\min_x\{\Vert Ax-b \Vert ^2_2 + \lambda^2\Vert x\Vert^2_2\}$$ In the new minimization, the additional constraint is the squared norm of $x.$ $\lambda$ is a non-negative constant decided in advance, acting as a weight on the strength of the additional constraint. Some alternate ways to write Tikhonov regularization include $$\bbox[lightgreen]{(A^T A + \lambda^2 I)x=A^{T}b}$$ and $$\min \left\Vert\genfrac{[}{]}{0pt}{}{A}{\lambda I}x - \genfrac{[}{]}{0pt}{}{b}{0}\right\Vert$$

Can someone explain the alternate formulation in green. I am not able to convert this formulation to standard one. I get one Atransposedinverse extra. The second formulation can be easily converted to original formulation, though. Thanks.

Olli Niemitalo
  • 13,491
  • 1
  • 33
  • 61
Sukuya
  • 23
  • 3

1 Answers1

1

Just expand the term to be minimized, take the derivative with respect to $x$ and set it equal to zero:

$$\begin{align}||Ax-b||_2^2+\lambda^2||x||_2^2&=(Ax-b)^T(Ax-b)+\lambda^2x^Tx\\&=x^TA^TAx-2b^TAx+b^Tb+\lambda^2x^Tx\tag{1}\end{align}$$

Taking the derivative w.r.t. $x$ and setting it equal to zero gives

$$2A^TAx-2A^Tb+2\lambda^2x=0\tag{2}$$

from which the given equation follows.

Matt L.
  • 89,963
  • 9
  • 79
  • 179
  • 1
    I didn't notice that there is no min in that formulation unlike others, so effectively it was closed form solution. Thanks Matt. – Sukuya Feb 22 '18 at 12:24