$\begingroup$

Can anyone please explain step-by-step why the x,A,b position as such? For example in step 2 'Expand the error', why isn't is (Ax)^2*x -2(Ax)(b)+b^2? In step 3 why setting derivative = 0, the 2x^T(A^Tb) in previous step & ||b||^2 disappear? Thx.

$\endgroup$ 2

1 Answer

$\begingroup$

By definition, $$\lVert Ax-b\rVert_2^2 = \langle Ax-b, Ax-b\rangle$$which you can expand by (bi)linearity of the inner product:

$$\begin{align}\lVert Ax-b\rVert_2^2 &= \langle Ax-b, Ax\rangle - \langle Ax-b, b\rangle\\ &=\langle Ax, Ax\rangle - \langle b, Ax\rangle - \langle Ax, b\rangle+\langle b, b\rangle\\ &=\langle Ax, Ax\rangle - 2\langle Ax, b\rangle+\langle b, b\rangle\\ &=(Ax)^TAx - 2(Ax)^Tb +\lVert b\rVert_2^2\\ &=x^TA^TAx - 2x^TA^Tb +\lVert b\rVert_2^2\end{align}$$using that (i) $\langle Ax, b\rangle=\langle b, Ax\rangle$ and (ii) $(Ax)^T = x^T A^T$.

When you set the derivative to 0, terms do not "disappear". $- 2x^TA^Tb$ gives you a $- 2A^Tb$, and $x^TA^TAx$ gives you a $2A^TAx$. The constant term $\lVert b\rVert_2^2$, indeed, does disappear (it's a constant, so when you differentiate with regard to $x$...).

$\endgroup$ 0

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy