1

I have the linear underdetermined system $$Ax=b$$ and I need to find $x$ constrained by the maximization of a score function $g(x)$.

I could find the minimum of a function like $$|Ax-b|^2+1/g(x)$$ but I wanted to know if there exists a canonical way to solve this problem, exploiting the linear part to have a more regular convergence.

I'm using python with scikit-learn, but I can also try to convert MATLAB code to solve it.

EDIT. The $g(x)$ function is not convex and has many maximums not necessarily where $ Ax=b$.

N74
  • 231
  • 1
  • 6
  • 1
    How does $g(x)$ look like? What you are wanint to do, of course, is maximize $g(x)$ over all of the solutions of $Ax=b$; whether this is simple or not depends on the structure of $g(x)$. I'm going to note, however, that that's not the formulation you have -- in your minimization problem, the solution does not necessarily solve $Ax=b$. – Wolfgang Bangerth Oct 30 '17 at 23:24
  • What assumptions can be mde about the function $1/g(x)$? Is it convex? differentiable? – Brian Borchers Oct 30 '17 at 23:49
  • @wolfgangbangerth you are right. You can use $Ax=b$ as a sort of equality constraint for the maximization problem, but I can't find an optimization function that has the constraints expressed as a linear combination of the solution vector. I would need to give $|Ax-B|^2$ as an equality constraint function and manually evaluate the Jacobian of this function to speed up the convergence. I'd like to know if there is an easier way to express this problem. – N74 Oct 31 '17 at 03:14
  • 1
    Posing $Ax=b$ as a (linear) constraint does not make the problem any more complicated and seems like the way to go. In general, linear constraints are trivial to deal with and don't add much complexity. Can you state the actual form of $g(x)$ and why you chose that one? – Wolfgang Bangerth Oct 31 '17 at 04:02
  • @wolfgangbangerth I don't have the analytical expression of $g(x)$ as it is evaluated by a piece of code I import. I know it expresses the likelihood that the solution vector belongs to a class. I suppose it is something like a gaussian mixture but I can't be sure. – N74 Oct 31 '17 at 06:18
  • I'm not sure anyone can help you then. If the null space of $A$ is small, then maximizing $g(x)$ becomes a low-dimensional problem, which makes the problem relatively simple. But without further knowledge of $g$, I don't think there are any good suggestions to be gotten. – Wolfgang Bangerth Nov 01 '17 at 01:19

1 Answers1

2

Right name for your problem is maximization of g(x) subject to linear constraint Ax = b. I.e. you have requirement that Ax=b exactly, and you want to maximize g(x) subject to this requirement.

The difference from your approach is that such formulation explicitly states that we tolerate no deviations from Ax=b, whilst your formulation allows complicated trade-offs between accuracy of Ax=b and maximization of g(x).

There exist numerous libraries for constrained nonlinear optimization. As for Python, I'd recommend scipy.optimize.minimize(), SLSQP solver.

UPDATE: if you are unsatisfied with SciPy, another option is to try ALGLIB. This library is mature and free and has Python wrapper (actually, it is C++ and C# library). It has linearly constrained optimizer called MinBLEIC, which allows to specify entire constraining matrix A as one array. The only drawback is that library has no support for numpy's ndarray, so if you prefer numpy for arrays, you have to perform conversion of ndarray to list-of-lists (see this question). Disclosure: I am one of the developers :)

Sergey
  • 21
  • 2
  • I tried minimize... but it is annoying that I must create a lambda function for each row of the $Ax=b$ constraint. I looked into the SLSQP solver source code and it seems that the Fortran program called by minimize actually needs $A_{eq}$ and $b_{eq}$, but I did not find a way to pass them directly. – N74 Nov 02 '17 at 12:53
  • Anyway, just as info, optimize gives me an answer with a worse score than the initial vector I provide (that I just find with a bounded linear programming). – N74 Nov 02 '17 at 12:55
  • I added one more link to another library, see "UPDATE" in my post – Sergey Nov 03 '17 at 13:38