0

I have the following minimization problem:

$$ \arg \min_{z} \max_{U} \left\{ \frac{1}{2} {\left\| b - z \right\|}_{2}^{2} + \frac{\lambda}{2} \sum_{i = 1}^{n} {\left\| {U}^{T} {z}_{i} \right\|}_{2}^{2} \right\} \; \text{s. t.} \; {U}^{T} U = I $$

Where $ {z}_{i} \in {\mathbb{R}}^{d} $, $ z = {\left[ {z}_{1}^{T}, {z}_{2}^{T}, \ldots, {z}_{n}^{T} \right]}^{T} \in {\mathbb{R}}^{nd} $ and $ U \in {\mathbb{R}}^{d \times k}, \; k < n $.

For fixed $ U $ this is a simple Least Squares problem with respect to $ z $.
For fixed $ z $ this is PCA / Eigenvector / SVD problem with respect to $ U $ (The data is given by the set of $ {z}_{i} $).

The questions are:

  1. How could one solve this (Is there a unique minimizer at all?)?
  2. Will solve it in iterative way (Alternating between $ U $ and $ z $ as fixed) as described above guaranteed to achieve something? Could it be shown to fail?

Thank You.

P. S.
I think it is not trivial as the function of $ U $ isn't concave and the constraints aren't convex. Hence it is not a convex optimization problem and alternation isn't guaranteed to produce something.

Royi
  • 8,711
  • @Exodd, It should be like in the SVD. Namely $ U $ isn't necessarily square matrix. But I was mistaken by writing $ \min $ with respect to $ U $. – Royi Dec 27 '16 at 13:50
  • @Exodd, I was mistaken and replaced $ \min $ with $ \max $ with respect to $ U $. – Royi Dec 27 '16 at 13:57
  • So the subtlety is that $| U^T z |2^2 = z^T U U^T z$ and $U U^T$ might not be $I$ even though $U^T U$ is? Perhaps it would be clearer to write the sizes of these matrices/vectors. Anyway, presumably $F(z)=\max{U : U^T U = I} \frac{| b - z|^2 + \lambda | U^T z |^2}{2}$ is convex, which would mean that you could do an alternating iterative method: apply any black box convex optimization routine to $F$ and evaluate $F$ by an SVD method or similar. – Ian Dec 27 '16 at 14:09
  • @Ian, have a look here: http://math.stackexchange.com/questions/1902421/prove-that-the-trace-of-the-matrix-product-uau-is-maximized-by-setting-us You may think $ \Sigma = z {z}^{T} $ and you need to find $ U $. Just classic PCA problem. Also can be seen here: http://math.stackexchange.com/questions/1678404/pca-interpretation/1678432. – Royi Dec 27 '16 at 14:16
  • I'm pretty sure I know what you're doing, but it would still be clearer with the dimensions explicitly stated. – Ian Dec 27 '16 at 14:22
  • 1
    @Ian, It seems I missed something I will edit it. By the way, I think it is not a convex optimization problem as the function is convex yet it is maximized and the constraints are not convex. – Royi Dec 27 '16 at 15:05

0 Answers0