In mathematics, low-rank approximation is a minimization problem, in which the cost function measures the fit between a given matrix (the data) and an approximating matrix (the optimization variable), subject to a constraint that the approximating matrix has reduced rank. The problem is used for mathematical modeling and data compression. The rank constraint is related to a constraint on the complexity of a model that fits the data. In applications, often there are other constraints on the approximating matrix apart from the rank constraint, e.g., non-negativity and Hankel structure.
Contents
- Definition
- Applications
- Basic low rank approximation problem
- Proof of EckartYoungMirsky theorem for spectral norm
- Proof of EckartYoungMirsky theorem for Frobenius norm
- Weighted low rank approximation problems
- Image and kernel representations of the rank constraints
- Alternating projections algorithm
- Variable projections algorithm
- A Variant convex restricted low rank approximation
- References
Low-rank approximation is closely related to:
Definition
Given
Applications
Basic low-rank approximation problem
The unstructured problem with fit measured by the Frobenius norm, i.e.,
has analytic solution in terms of the singular value decomposition of the data matrix. The result is referred to as the matrix approximation lemma or Eckart–Young–Mirsky theorem. Let
be the singular value decomposition of
where
is such that
The minimizer
Proof of Eckart–Young–Mirsky theorem (for spectral norm)
Let
is the singular value decomposition of
We claim that the best rank
where
First, note that we have
Therefore, we need to show that if
Since
such that
The result follows by taking the square root of both sides of the above inequality.
Proof of Eckart–Young–Mirsky theorem (for Frobenius norm)
Let
is the singular value decomposition of
We claim that the best rank
where
First, note that we have
Therefore, we need to show that if
By the triangle inequality with the spectral norm, if
Since
Therefore,
as required.
Weighted low-rank approximation problems
The Frobenius norm weights uniformly all elements of the approximation error
where
The general weighted low-rank approximation problem does not admit an analytic solution in terms of the singular value decomposition and is solved by local optimization methods, which provide no guarantee that a globally optimal solution is found.
Image and kernel representations of the rank constraints
Using the equivalences
and
the weighted low-rank approximation problem becomes equivalent to the parameter optimization problems
and
where
Alternating projections algorithm
The image representation of the rank constraint suggests a parameter optimization methods, in which the cost function is minimized alternatively over one of the variables (
The resulting optimization algorithm (called alternating projections) is globally convergent with a linear convergence rate to a locally optimal solution of the weighted low-rank approximation problem. Starting value for the
Matlab implementation of the alternating projections algorithm for weighted low-rank approximation:
Variable projections algorithm
The alternating projections algorithm exploits the fact that the low rank approximation problem, parameterized in the image form, is bilinear in the variables
Consider again the weighted low rank approximation problem, parameterized in the image form. Minimization with respect to the
The original problem is therefore equivalent to the nonlinear least squares problem of minimizing
Matlab implementation of the variable projections algorithm for weighted low-rank approximation:
The variable projections approach can be applied also to low rank approximation problems parameterized in the kernel form. The method is effective when the number of eliminated variables is much larger than the number of optimization variables left at the stage of the nonlinear least squares minimization. Such problems occur in system identification, parameterized in the kernel form, where the eliminated variables are the approximating trajectory and the remaining variables are the model parameters. In the context of linear time-invariant systems, the elimination step is equivalent to Kalman smoothing.
A Variant: convex-restricted low rank approximation
Usually, we want our new solution not only to be of low rank, but also satisfy other convex constraints due to application requirements. Our interested problem would be as follows,
This problem can find tons of real applications, including to recover a good solution from a inexact (semidefinite programming) relaxation. If additional constraint
This problem is helpful in solving many problems. However, it is challenging due to the combination of the convex and nonconvex (low-rank) constraints. Different techniques were developed based on different realizations of