Hilbert projection theorem explained
In mathematics, the Hilbert projection theorem is a famous result of convex analysis that says that for every vector
in a
Hilbert space
and every nonempty closed convex
there exists a unique vector
for which
is minimized over the vectors
; that is, such that
for every
Finite dimensional case
Some intuition for the theorem can be obtained by considering the first order condition of the optimization problem.
Consider a finite dimensional real Hilbert space
with a subspace
and a point
If
is a or of the function
defined by
(which is the same as the minimum point of
), then derivative must be zero at
In matrix derivative notation[1] Since
is a vector in
that represents an arbitrary tangent direction, it follows that
must be orthogonal to every vector in
Statement
Proof by reduction to a special case
It suffices to prove the theorem in the case of
because the general case follows from the statement below by replacing
with
Consequences
If
then
0=\langlec,c\rangle=\|c\|2,
which implies
Let
where
is the underlying scalar field of
and define
which is continuous and linear because this is true of each of its coordinates
h\mapsto\langleh,c\rangle.
The set
C\bot=L-1(0)=L-1\left(\{0\}\right)
is closed in
because
is closed in
and
is continuous. The kernel of any linear map is a vector subspace of its domain, which is why
is a vector subspace of
Let
The Hilbert projection theorem guarantees the existence of a unique
such that
\|x-m\|\leq\|x-c\|forallc\inC
(or equivalently, for all
). Let
so that
and it remains to show that
The inequality above can be rewritten as:
Because
and
is a vector space,
and
which implies that
The previous inequality thus becomes
or equivalently,
But this last statement is true if and only if
every
Thus
Properties
Expression as a global minimum
The statement and conclusion of the Hilbert projection theorem can be expressed in terms of global minimums of the followings functions. Their notation will also be used to simplify certain statements.
Given a non-empty subset
and some
define a function