## ORF523: Mirror Prox

Today I’m going to talk about a variant of Mirror Descent invented by Nemirovski in 2004 (see this paper). This new algorithm is called Mirror Prox and it is designed to minimize a smooth convex function over a compact convex set . We assume that the smoothness is measured in some arbitrary norm , precisely we assume that .

Let be a mirror map on (see this post for a definition) and let . Mirror Prox is described by the following equations:

In words the algorithm first makes a step of Mirror Descent to go from to , and then it makes a similar step to obtain , starting again from but this time using the gradient of evaluated at (instead of ). The following result justifies the procedure.

Theorem. Let be a mirror map -strongly convex on with respect to . Let and be convex and -smooth w.r.t. . Then Mirror Prox with satisfies (with )

Proof: Let . We write

We will now bound separately these three terms. For the first one we have (by definition of the method, first-order optimality condition of the projection, and the definition of the Bregman divergence)

For the second term using the same properties than above and the strong-convexity of the mirror map one has

Finally for the last term, using H\”older’s inequality, -smoothness, and the fact one has

Thus summing up these three terms and using that one gets

and this inequality is also true for (see the bound for the first term). The proof is then concluded as usual.

Mirror Prox for Saddle Points

Following the generalization of Mirror Descent for Saddle Points (MD-SP, see this post) one can propose a version of Mirror Prox for Saddle Points (MP-SP). In fact the importance of Mirror Prox lies primarily in its application to compute the saddle point of a smooth convex-convave function. Indeed, the key observation is that many non-smooth convex functions admit a smooth saddle-point representation (see examples below). With such a representation one can then use MP-SP to obtain a rate of convergence of order , despite the fact that the original function is non-smooth! Recall that we proved that for non-smooth convex functions the optimal black-box rate is of order . However MP-SP is not a black-box method in the sense that it makes use of the structure of the function (by representing it as a saddle-point). Nonetheless MP-SP still enjoys the nice time-complexity property of black-box methods since it is a first-order method. It is a win-win situation!

Let us now describe more precisely MP-SP. Let be a compact and convex set endowed with some norm and a mirror map (defined on ), and let be a compact and convex set endowed with some norm and a mirror map (defined on ). Let be a continuously differentiable function such that is convex and is concave. We assume that is -smooth in the sense that:

We consider the mirror map on (defined on ). Let . Then for , let and be defined by the following equations (with ):

Theorem. Assume that is -strongly convex on with respect to and let (with similar definition for ). Let . Then SP Mirror Descent with satisfies

Proof: First we endow with the norm defined by

It is immediate that is -strongly convex with respect to on . Furthermore one can easily check that

and thus the vector fields used in MP-SP satisfy:

It suffices now to use the exact same proof than for the standard Mirror Prox starting with:

Smooth saddle-point representation of non-smooth functions

Assume that the function to be optimized is of the form:

where the functions are convex, -smooth and -Lipschitz in some norm . Minimizing over is equivalent to finding a saddle point over since one has:

Thus we are looking for the saddle point of which is convex in , and concave (in fact linear!) in . Furthermore is smooth and it is an easy exercise to verify that in this case one has , , , and . Thus a rate of convergence (to the minimizer of ) of order can be obtained by applying MP-SP with appropriate mirror maps (in particular the update in the -space has to be done with the negative entropy). Note that is rate is particularly striking since the oracle lower bound of was proved precisely for this type of function! Here we are able to get around this lower bound by making use of the structure of the function. Remark also that if you think about what this algorithm is doing to optimize the function you get a very very elegant procedure for which it is highly non-trivial a priori to prove something about its rate of convergence.

Finally note that if and are of different order of magnitudes one can obtain a better result by considering the mirror map and optimizing over the values of and .

Another interesting example arises when with , and . With the -norm on both spaces it is easy to see that (the entrywise max norm), and of course . Thus we obtain here a convergence rate of order . Again if and are of different order of magnitudes one can improve the result to obtain a rate of order by considering the mirror map and optimizing over the values of and .

Many more interesting examples can be derived and we refer the reader to the original paper of Nemirovski as well as this book chapter by Juditsky and Nemirovski.

Posted in Optimization | 1 Comment

## COLT 2013 accepted papers

The accepted papers for COLT 2013 have just been revealed. You can also find the list below with a link to the arxiv version when I could find one (if you want me to add a link just send me an email).

I also take the opportunity of this post to remind my student readers that you can apply for a ‘student travel award’ to come participate in COLT 2013. The instructions can be found here. So far I received very few applications so I highly encourage you to give it a try! You can check out this blog post that gives a few reasons to make the effort to travel and participate in conferences.

Update with a message from Philippe Rigollet: Sebastien Bubeck and I are local organizers of this year’s edition of the Conference On Learning Theory (COLT 2013) on June 12-14.
This means that it will take place at Princeton (in Friend 101), and of course, that it’ll be awesome.
The list of accepted papers is very exciting and has been posted on the official COLT 2013 website:

Thanks to generous sponsors, we have been able to lower the registration costs to a historic minimum (regular: $300, students:$150).
These costs include:
-a cocktail party followed by a banquet on Thursday night
-coffee breaks (with poster presentations)
-lunches (with poster presentation)
-47 amazing presentations
-3 Keynote lectures by Sanjeev Arora (Princeton), Ralf Herbrich (amazon.com) and Yann LeCun (NYU) respectively

The deadline for early registration is May 13 (even if you’re local, help us support the conference by registering and not just crashing the party).
To register, go to:
http://orfe.princeton.edu/conferences/colt2013/?q=for-participants/registration

Please help us spread the word by forwarding this email to anyone interested in Learning Theory and by using your favorite social media.

COLT 2013 accepted papers

## ORF523: Mirror Descent, part II/II

We start with some of the standard setups for the Mirror Descent algorithm we described in the previous post.

Standard setups for Mirror Descent

‘Ball’ setup. The simplest version of Mirror Descent is obtained by taking on The function is a mirror map strongly convex w.r.t. , and furthermore the associated Bregman Divergence is given by . Thus in that case Mirror Descent is exactly equivalent to Projected Subgradient Descent, and the rate of convergence obtained in the previous post recovers our earlier result on Projected Subgradient Descent.

‘Simplex’ setup. A more interesting choice of a mirror map is given by the negative entropy

on . In that case the gradient update can be written equivalently as

The Bregman divergence of this mirror map is given by (also known as the generalized Kullback-Leibler divergence). It is easy to verify that the projection with respect to this Bregman divergence on the simplex amounts to a simple renormalization . Furthermore it is also easy to verify that is -strongly convex w.r.t. on (this result is known as Pinsker’s inequality). Note also that one has and when

The above observations imply that when minimizing on the simplex a function with subgradients bounded in -norm, Mirror Descent with the negative entropy achieves a rate of convergence of order . On the other the regular Subgradient Descent achieves only a rate of order in this case!

‘Spectrahedron’ setup. We consider here functions defined on matrices, and we are interested in minimizing a function on the spectrahedron defined as:

In this setting we consider the mirror map on given by the negative von Neumann entropy:

where are the eigenvalues of . It can be shown that the gradient update can be written equivalently as

where the matrix exponential and matrix logarithm are defined as usual. Furthermore the projection on is a simple trace renormalization.

With highly non-trivial computation one can show that is -strongly convex with respect to the Schatten -norm defined as

It is easy to see that one has and for . In other words the rate of convergence for optimization on the spectrahedron is the same than on the simplex!

Mirror Descent for Saddle Points

We consider now the following saddle-point problem: Let be compact convex sets, and be a continuous function such that is convex and is concave. We are interested in finding such that

(Note that the above equality uses Sion’s minimax theorem from this lecture.) We measure the quality of a candidate solution through the primal/dual gap

The key observation now is that for any , one has with and ,

and

In particular, denoting we just proved

for some This inequality suggests that one can solve the saddle-point problem with a Mirror Descent in the space using the ‘vector field’ given by . Precisely this gives the following SP Mirror Descent:

Let be a mirror map on (defined on ) and let be a mirror map on (defined on ). We consider now the mirror map on (defined on ).

Let . Then for , let such that

where with and . Finally let

The following result describes the performances of SP Mirror Descent.

Theorem. Assume that is -strongly convex on with respect to , and let . Assume that is -Lipschitz w.r.t. . Define similarly . Then SP Mirror Descent with satisfies

Proof: First we endow with the norm defined by

It is immediate that is -strongly convex with respect to on . Furthermore one can easily check that

and thus the vector field used in the SP mirror descent satisfies:

It suffices now to use the exact same proof than for the standard Mirror Descent starting with:

Posted in Optimization | 3 Comments

## ORF523: Mirror Descent, part I/II

In this post we are interested in minimizing a convex function over a compact convex set under the assumption that is -Lipschitz on with respect to some arbitrary norm (precisely we assume that ). As we have seen in this post, using the correct norm to measure the variation of a function can make an enormous difference. We describe now a method invented by Nemirovski and Yudin to deal with this situation.

Let us for a moment abstract the situation a little bit and forget that we are trying to do optimization in finite dimension. We already observed that Projected Subgradient Descent works in an arbitrary Hilbert space (think of ). However we are now interested in the more general situation where we are optimizing the function in some Banach space (for example ). In that case the Gradient Descent strategy does not even make sense: indeed the gradients (more formally the Fréchet derivative) are elements of the dual space and thus one cannot perform the computation (it simply does not make sense). We did not have this problem for optimization in an Hilbert space since by Riesz representation theorem is isometric to . The great insight of Nemirovski and Yudin is that one can still do a gradient descent by first mapping the point into the dual space , then performing the gradient update in the dual space, and finally mapping back the resulting point to the primal space . Of course the new point in the primal space might lie outside of the constraint set and thus we need a way to project back the point on the constraint set . The rest of this lecture will make this idea more precise.

Preliminaries

We consider a non-empty convex open set such that the constraint set is included in its closure, that is , and such that .

Now we consider a continuously differentiable and strictly convex function defined on . This function will play the role of the mirror map between the primal and dual space. Precisely a point in the primal is mapped to in the ‘dual’ (note that here all points lie in and the notions of ‘primal’ and ‘dual’ space only have an intuitive meaning). In other words the gradient update of Mirror Descent with the mirror map will take the form . Next this dual point needs to be mapped back to the primal space, which is always possible under the condition that (weaker assumptions are possible but we will not need them), that is one can find such that

Now this point might lie outside of the constraint set . To project back on the set of constraint we use the Bregman divergence associated to defined by

The projection of onto then takes the following form:

To ensure the existence of this projection we assume now that satisfies

With this assumption is locally increasing on the boundary of , which together with the fact that is compact, and is stricly convex implies the existence and uniqueness of the projection defined above.

We say that is a mirror map if it satisfies all the above assumptions.

Mirror Descent

We can now describe the Mirror Descent strategy based on a mirror map . Let . Then for , let such that

and

Theorem Let be a mirror map. Assume also that is strongly convex on with respect to , that is

Let and be convex and -Lipschitz w.r.t. , then Mirror Descent with satisfies

Proof: Let . The claimed bounded will be obtained by taking a limit . Now by convexity of , the definition of the Mirror Descent, and the definition of the Bregman divergence one has

Now the key observation is that by writing down the first order optimality condition for one obtains

which can be written equivalently as

Thus the term will lead to a telescopic sum when summing over to , and it remains to bound the other term as follows:

We proved

which concludes the proof up to trivial computation.

Mirror Descent as a proximal method

One can rewrite Mirror Descent as follows:

Nowadays this last expression is usually taken as the definition of Mirror Descent (see this paper by Beck and Teboulle). It gives a proximal point of view on Mirror Descent: the algorithm is trying to minimize the local linearization of the function while not moving too far away from the previous point (with distances measured via the Bregman divergence of the mirror map).

Next time we will see some of the ‘standard setups’ for Mirror Descent, that is pairs of (constraint set, mirror map) that ‘go well together’.

Posted in Optimization | 2 Comments

## ORF523: ISTA and FISTA

As we said in the previous lecture it seems stupid to consider that we are in a black-box situation when in fact we know entirely the function to be optimized. Consider for instance the LASSO objective where one wants to minimize over . By resorting to black-box procedures one would solve this problem with Subgradient Descent and the rate of convergence would be of order as this function is non-smooth (and potentially non-strongly convex). However we will see now that one can take advantage of the form of the LASSO objective which is a sum of a smooth part and a simple non-smooth part to obtain rates as fast as .

In this lecture (which follows this paper by Beck and Teboulle) we consider the unconstrained minimization of a sum of two functions and that satisfy the following requirements:

(i)  admits a minimizer on .

(ii)  and are convex, and is -smooth.

(iii)  is known and is accessible with a -order oracle.

As we will see next, for the proposed algorithm to be computationally efficient also needs to be ‘simple’. For instance a separable function (i.e., ) will be considered as a simple function. Our prime example will be .

ISTA (Iterative Shrinkage-Thresholding Algorithm)

Recall that the Gradient Descent algorithm to optimize the smooth function is simply given by

which can be written in the proximal form as

Now here one wants to minimize , and is assumed to be known and ‘simple’. It seems very natural to consider the following iterative procedure known as ISTA:

In terms of convergence rate it is not too hard to show that ISTA has the same convergence rate on than Gradient Descent on , more precisely with one has

This improved convergence rate over Subgradient Descent comes at a price: computing may be a difficult optimization problem by itself in general, and this is why one needs to assume that is ‘simple’. For instance if can be written as then one can compute by solving convex problems in dimension . In the case where this one-dimensional problem is given by:

Elementary computations show that this problem has an analytical solution given by , where is the shrinkage operator defined by

FISTA (Fast ISTA)

As we have seen in this lecture, the optimal rate of convergence for smooth functions can be obtained with Nesterov’s Accelerated Gradient Descent. Combining this idea with ISTA one gets FISTA which is described as follows. Let

Let an arbitrary initial point, and

Again it is not hard to show that the rate of FISTA is similar to the one of Nesterov’s Accelerated Gradient Descent, more precisely:

Posted in Optimization | 2 Comments

## ORF523: Conditional Gradient Descent and Structured Sparsity

In the following table we summarize our findings of previous lectures in terms of oracle convergence rates (we denote ).

Note that in the last two lines the upper bounds and lower bounds are not matching. In both cases one can show that in fact the lower bound is tight (using respectively Nesterov’s Accelerated Gradient Descent, and the Center of Gravity Method).

Thus in terms of oracle complexity the picture is now complete. However this is not quite the end of the story and the algorithms can be improved in various ways:

1. The projection step in Projected Subgradient Descent can be expensive in terms of computational complexity. We will see an algorithm that avoids this projection, called Conditional Gradient Descent.
2. The black-box model is essentially an abstraction (note that this is true for -order oracles, but the situation is different for -order oracles). In practice one often knows the function to be optimized entirely. In that context it seems very wasteful to drop all this information and focus on black-box procedures. However by doing so we were able to derive linear time algorithm while the ‘structural’ Interior Point Methods (which use the form of the function to be optimized by deriving an appropriate self-concordant barrier) are not linear time. We will see one example of an algorithm that makes a partial use of the structural information on the function to be optimized, while still being essentially a gradient-based procedure with linear time complexity. More specifically we will study the FISTA algorithm that can optimize functions of the form , where is known and can be accessed with a -order oracle. Another very interesting idea in that direction is the Nesterov’s smoothing but unfortunately we will not have time to cover it.
3. The rates of convergence we proved did not explicitely involve the dimension . However in specific instances some parameters may involve a dependency on the dimension. For instance consider the case where the function is such that the magnitude of its subgradients are controlled coordinate-wise, say . In that case the best one can say is that which leads to a rate of convergence for Projected Subgradient Descent of order . We will see that this situation can be dramatically improved and that one can obtain in this setting a rate of order . More generally we will describe the Mirror Descent algorithm which is adapted to convex functions which are -Lipschtiz in an arbitrary norm  (not necessarily ).

We start this program with a projection-free algorithm for black-box constrained optimization.

Conditional Gradient Descent (a.k.a. Frank-Wolfe algorithm)

Conditional Gradient Descent is an algorithm designed to minimize a smooth convex function over a compact convex set . It performs the following update for :

This algorithm goes back to Frank and Wolfe (1956), and the following result is extracted from Jaggi (2013). We consider here functions which are -smooth in some arbitrary norm , that is

where the dual norm is defined as .

Theorem. Let be a -smooth function w.r.t. some norm , , and . Then for any , one has

Proof: The following sequence of inequalities holds true, using respectively -smoothness (which implies ), the definition of , the definition of , and the convexity of :

Rewriting this inequality in terms of one obtains

A simple induction using that finishes the proof (note that the initialization is done at step with the above inequality yielding ).

The rate of convergence we just proved for Conditional Gradient Descent is no better than the rate of Projected Gradient Descent in the same setting. However, on the contrary to the latter algorithm where a quadratic programming problem over has to be solved (the projection step), here we only need to solve a linear problem over , and in some cases this can be much simpler.

While being projection-free is an important property of Conditional Gradient Descent, a perhaps even more important property is that it produces sparse iterates in the following sense: Assume that is a polytope with sparse vertices, that is the number of non-zero coordinates (which we denote by the -norm which is not a norm despite the name) in a fixed vertex is small, say of order . Then at each iteration, the new iterate will increase its number of non-zero coordinates by at most . In particular one has . We will see now an example of application where this property is critical to obtain a computationally tractable algorithm.

Note that the property described above proves in particular that for a smooth function on the simplex there always exist sparse approximate solutions. More precisely there must exist a point with only non-zero coordinates and such that . Clearly this is the best one can hope for in general, as it can be seen with the function (since by Cauchy-Schwarz one has which implies on the simplex ).

An application of Conditional Gradient Descent: Least-squares regression with structured sparsity

This section is inspired by an open problem from this paper by Gabor Lugosi (what is described below solves the open problem). Consider the problem of approximating a signal by a ‘small’ combination of elementary dictionary elements . One way to do this is to consider a LASSO type problem in dimension of the following form (we denote here for the coordinate of a vector )

Let be the dictionary matrix with column given by . Instead of considering the penalized version of the problem one could look at the following constrained problem on which we will focus now:

We will now make some assumptions on the dictionary. We are interested in cases where the size of the dictionary can be very large, potentially exponential in the ambient dimension . Nonetheless we want to restrict our attention to algorithms that run in reasonable time with respect to the ambient dimension , that is we want polynomial time algorithms in . Of course in general this is impossible, and we need to assume that the dictionary has some structure that can be exploited. Here we make the assumption that one can do linear optimization over the dictionary in polynomial time in . More precisely we assume that we can solve in time (where is polynomial) the following problem for any :

This assumption is met for many combinatorial dictionaries. For instance the dictionary elements could be vector of incidence of spanning trees in some fixed graph, and then the linear optimization problem can be solved with a greedy algorithm.

Finally we assume that the -norm of the dictionary elements are controlled by some , that is .

Recall that one wants to minimize the function on the -ball of in polynomial time in . Note that at first sight this task may seem completely impossible as we are not even allowed to write down entirely a vector (since this would take time linear in ). The key property that will save us is that this function admits sparse minimizers as we discussed in the previous section, and this will be exploited by the Conditional Gradient Descent. First note that

This form for the gradient together with our computational assumption on the dictionary imply that Conditional Gradient Descent can be run in polynomial time. Indeed, assume that is already computed, then the first step of Conditional Gradient Descent is to find the coordinate that maximizes which can be done by maximizing and . Thus this first step takes time . Computing from and takes time since , and computing from and takes time . Thus the overall time complexity of steps is .

To derive a rate of convergence it remains to study the smoothness of . This can be done as follows:

which means that is -smooth with respect to the norm. Thus we get the following rate of convergence:

In other words we proved that one can get an -optimal solution to our original problem with a computational effort of using the Conditional Gradient Descent.

Posted in Optimization | 1 Comment

## A new book on Concentration Inequalities by Boucheron, Lugosi and Massart

If you work in a field related to probability theory you most certainly have already come across the concentration of measure phenomenon: ‘a function that depends on many independent variables, but not too much on any one of them, is almost constant’. This extremely strong statement can be made formal in various ways, the most basic one being the Hoeffding inequality (and its well-known generalization the Mc Diarmid’s inequality). However in many applications Mc Diarmid’s inequality is too weak: the resulting bound does not account for the variance, and the assumption on the underlying function is very stringent. Fortunately one can go far beyond Mc Diarmid’s inequality, and this is explored in depth in the new book (published a few weeks ago!) ‘Concentration Inequalities‘, by Stéphane Boucheron, Gábor Lugosi and Pascal Massart.

I have read an early draft of the book during my postdoc with Gábor, and I strongly recommend it to anyone who is even just remotely interested in probabilistic phenomenons. The first 100 pages form a gentle introduction to the subject and they contain indispensable results for Machine Learners and alike. Then Chapter 5-6-7  are (in my opinion) the core of the book: they masterfully describe results that were always a bit daunting to me, including the geometrical point of view on the concentration of measure. This is where I stopped my first reading two years ago, but now that I have the book in my hands I realize that this is only the first half of it!!! I have started to read the second half, and right now I am having a lot of fun with influences and monotone sets ;).

Posted in Uncategorized | 3 Comments

## Guest post by Amir Ali Ahmadi: Sum of Squares (SOS) Techniques: An Introduction, Part II/II

The reader is encouraged to read Part I of this series before reading this post. For ease of reference, we recall the polynomial optimization problem from our previous post:

(1)

This is the task of minimizing a multivariate polynomial over a basic semialgebraic set defined by the polynomials .

Sum of squares and semidefinite programming

If a polynomial is nonnegative, can we write it in a way that its nonnegativity becomes obvious? This is the meta-question behind Hilbert’s 17th problem. As the title of this post suggests, one way to achieve this goal is to try to write the polynomial as a sum of squares of polynomials. We say that a polynomial is a sum of squares (sos), if it can be written as for some polynomials . Existence of an sos decomposition is an algebraic certificate for nonnegativity. Remarkably, it can be decided by solving a single semidefinite program.

Theorem 1 A multivariate polynomial in variables and of degree  is a sum of squares if and only if there exists a positive semidefinite matrix (often called the Gram matrix) such that

(2)

where is the vector of monomials of degree up to

The proof is straightforward and is left to the reader. Note that the feasible set defined by these constraints is the intersection of an affine subspace (arising from the equality constraints matching the coefficients of with the entries of in (2)) with the cone of positive semidefinite matrices. This is precisely the semidefinite programming (SDP) problem. The size of the Gram matrix is , which for fixed is polynomial in . Depending on the structure of , there are well-documented techniques for further reducing the size of the Gram matrix and the monomial vector . We do not pursue this direction here but state as an example that if is homogeneous of degree , then it suffices to place in the vector only monomials of degree exactly .

Example: Consider the task proving nonnegativity of the polynomial

Since this is a form (i.e., a homogeneous polynomial), we take

One feasible solution to the SDP in (2) is given by

Upon a decomposition , with , , , one obtains the sos decomposition

(3)

You are probably asking yourself right now whether every nonnegative polynomial can be written as a sum of squares. Did we just get lucky on the above example? Well, from complexity considerations alone (see previous post), we know that we should expect a gap between nonnegative and sos polynomials, at least for large .

In a seminal 1888 paper, Hilbert was the first to show that there exist nonnegative polynomials that are not sos. In fact, for each combination of degree and dimension, he showed whether such polynomials do or do not exist. For example, one famous theorem of Hilbert from that paper is that all nonnegative ternary quartic forms are sos. So we did not really use any luck in our example above! The same would have happened for any other nonnegative quartic form in three variables. Hilbert’s proof of existence of nonnegative polynomials that are not sos was not constructive. The first explicit example interestingly appeared nearly years later and is due to Motzkin:

(4)

Nonnegativity of follows from the arithmetic-geometric inequality (why?). Non-existence of an sos decomposition can be shown by some algebraic manipulations or by the systematic tools we have today for showing infeasibility of a semidefinite program (separating hyperplane theorems). From an application viewpoint, the good news for sum of squares optimization is that constructing polynomials of the type in (4) is not a trivial task. This is especially true if additional structure is required on the polynomial. For example, the following problem is still open.

Open problem. Construct an explicit example of a convex, nonnegative polynomial that is not a sum of squares.

This question is due to Parrilo. The motivation behind it is quite clear: we would like to understand whether sos optimization is exact for the special case of convex polynomial optimization problems. Blekherman has shown with non-constructive arguments that such “bad” convex polynomials must exist when the degree is four or larger and the number of variables goes to infinity. However, we do not know the smallest (or any reasonable) dimension for which this is possible and lack any explicit examples. For the reader interested in tackling this problem, it is known that any such convex polynomial must necessarily be not “sos-convex”. Roughly speaking, sos-convex polynomials are convex polynomials whose convexity is certified by an appropriately defined sum of squares identity. Examples of convex but not sos-convex polynomials have recently appeared and a characterization of the degrees and dimensions where they exist is now available. Interestingly, this characterization coincides with that of Hilbert, for reasons that are also not well-understood; see Chapter 3 of our thesis.

Having shown that not every nonnegative polynomial is a sum of squares of polynomials, Hilbert asked in his 17th problem whether every such polynomial can be written as a sum of squares of rational functions. Artin answered the question in the affirmative in 1927. As we will see next, such results allow for a hierarchy of semidefinite programs that approximate the set of nonnegative polynomials better and better.

Positivstellensatz and the SOS hierarchy

Consider proving a statement that we all learned in high school:

Just for the sake of illustration, let us pull an algebraic identity out of our hat which certifies this claim:

(5)

Think for a second why this constitutes a proof. The Positivstellensatz is a very powerful algebraic proof system that vastly generalizes what we just did here. It gives a systematic way of certifying infeasibility of any system of polynomial equalities and inequalities over the reals. Sum of squares representations play a central role in it. (They already did in our toy example above if you think about the role of the first term on the right hand side of (5)). Modern optimization theory adds a wonderfully useful aspect to this proof system: we can now use semidefinite programming to automatically find suitable algebraic certificates of the type in (5).

The Positivstellensatz is an example of a theorem of the alternative. The reader may already be familiar with similar fundamental theorems, such as the Hilbert’s (weak) Nullstellensatz (1893),

a system of polynomial equations is infeasible over the complex numbers

there exist polynomials such that ,

or the Farkas Lemma (1902) of linear programming,

a system of linear (in)equalities is infeasible over the reals

there exist such that .

These theorems typically have an “easy” (well, trivial) direction and a “hard” direction. The same is true for the Positivstellensatz.

Theorem 2  (Positivstellensatz — Stengle (1974)) The basic semialgebraic set

is empty

there exist polynomials and sum of squares polynomials , such that

(6)

The number of terms in this expression is finite since we never raise any polynomial to a power larger than one. The sum of squares polynomials are of course allowed to be the zero polynomial, and in practice many of them often are. There are bounds in the literature on the degree of the polynomials , , but of exponential size as one would expect for complexity reasons. There is substantial numerical evidence, however, from diverse application areas, indicating that in practice (whatever that means) the degrees of these polynomials are usually quite low. We remark that the Positivstellensatz is a very powerful result. For example, it is straightforward to show that the solution to Hilbert’s 17th problem follows as a corollary of this theorem (how?).

Under minor additional assumptions, refined versions of the Positivstellensatz we presented are available. The two most well-known are perhaps due to Schmüdgen and Putinar. For example, Putinar’s Positivstellensatz states that if the set satisfies the so-called Archimedean property (a property slightly stronger than compactness), then emptiness of guarantees a representation of the type (6), where the second and third line are scratched out; i.e., there is no need to take products of the constraints . While this may look like a simplification at first, there is a tradeoff: the degree of the sos multipliers may need to be higher in Putinar’s representation than in Stengle’s. This makes intuitive sense as the proof system needs to additionally prove statements of the type , while in Stengle’s representation this is taken as an axiom.

SOS hierarchies. Positivstellensatz results form the basis of sos hierarchies of Parrilo and Lasserre for solving the polynomial optimization problem (1). The two approaches only differ in the version of the Positivstellensatz they use (originally, Parrilo’s paper follows Stengle’s version and Lasserre’s follows Putinar’s), and the fact that Lasserre presents the methodology from the dual (but equivalent) viewpoint of moment sequences. In either case though, the basic idea is pretty simple. We try to obtain the largest lower bound for problem (1), by finding the largest for which the set is empty. We certify this emptiness by finding Positivstellensatz certificates. In level of the hierarchy, the degree of the polynomials and the sos polynomails in (6) is bounded by . As increases, the quality of the lower bound monotonically increases, and for each fixed , the search for the optimal , and the polynomials is a semidefinite optimization problem (possibly with some bisection over ).

Revisiting MAXCUT

Recall from Seb’s earlier lecture that we saw a beautiful algorithm for MAXCUT due to Goemans and Williamson which produced an approximation ratio of . The algorithm had two steps: first we solved a semidefinite program, then we performed some clever randomized rounding. In this section we focus only on the first step. We show that even low degree Positivstellensatz refutations can produce stronger bounds than the standard SDP relaxation.

Consider the 5-cycle with all edge weights equal to one. It is easy to see that the MAXCUT value of this graph is equal to . However, the standard SDP relaxation (i.e. the one used in the Goemans and Williamson algorithm) produces an upper bound of .

The MAXCUT value of the 5-cycle is equal to minus the optimal value of the quadratic program

(7)

We will find the largest constant such that the objective function minus is algebraically certified to be nonnegative on the feasible set. To do this, we solve the sos optimization problem

The decision variables of this problem are the constant and the coefficients of the polynomials , which in this case we parametrize to be quadratic functions. This sos program results in a polynomially sized semidefinite optimization problem via Theorem 1. The optimal value of the program is ; i.e., we have solved the MAXCUT instance exactly.

You may be wondering, “can we show that a certain level of the sos hierarchy combined with an appropriately designed rounding procedure produces an approximation ratio of better than ?” Let’s just say that if you did this, you would probably become an overnight celebrity.

Software

There are very nice implementations of sum of squares optimization solvers that automate the process of setting up the resulting semidefinite programs. The interested reader may want to play around with SOSTOOLS, YALMIP, or GloptiPoly.

Impact

While we focused in this post on the polynomial optimization problem, the impact of sum of squares optimization goes much beyond this area. In dynamics and control, sos optimization has enabled a paradigm shift from classical linear control to an efficient framework for design of nonlinear controllers that are provably safer, more agile, and more robust. Papers on applications of sos optimization have appeared in areas as diverse as quantum information theory, robotics, geometric theorem proving, formal verification, derivative pricing, stochastic optimization, and game theory, among others. In theoretical computer science, sos techniques are currently a subject of intense study. Not too long ago, I attended a plenary presentation at the CCC by Subhash Khot, where one of the last things he said was, “the sos hierarchies are currently the greatest threat to the unique games conjecture.” If you are an ORFE student and still not convinced that this SOS business is actually useful, you may find relief in knowing that SOS is in fact the core discipline of ORFE ;)

Posted in Optimization | 1 Comment

## ORF523: Strong convexity

Today we will talk about another property of convex functions that can significantly speed-up the convergence of first-order methods: strong convexity. We say that is -strongly convex if it satisfies

(1)

Of course this definition does not require differentiability of the function , and one can replace in the inequality above by . It is immediate to verify that a function is -strongly convex if and only if is convex.

Note that (1) can be interpreted as follows: at any point one can find a (convex) quadratic lower bound to the function , i.e. (and ). Thus in some sense strong convexity is a dual assumption to the smoothness assumption from previous lectures. Indeed recall that smoothness can be defined via the inequality:

which implies that at any point one can find a (convex) quadratic upper bound to the function , i.e. (and ). In fact we will see later a precise sense in which smoothness and strong convexity are dual notions (via Fenchel duality). Remark also that clearly one always has .

Projected Subgradient Descent for strongly convex and Lipschitz functions

In this section we investigate the setting where is strongly convex but potentially non-smooth. As we have already seen in a previous lecture, in the case of non-smooth functions we have to project back on the set where we control the norm of the gradients. Precisely let us assume that is a compact and convex set such that . We consider the Projected Subgradient Descent algorithm with time-varying step size, that is

The following result is extracted from a recent paper of Simon Lacoste-Julien, Mark Schmidt, and Francis Bach.

Theorem Let , then Projected Subgradient Descent satisfies for

\noindent ,

Note that one can immediately see from the analysis in this lecture that the rate is optimal. Indeed, one can always find a function and set (an ball) that satisfies the above assumptions and such that no black-box procedure can go at a rate faster than for (in fact the constant can be improved to ).

Proof: Let . Coming back to our original analysis of Projected Subgradient Descent and using the strong convexity assumption one immediately obtains

Multiplying this inequality by yields

Now sum the resulting inequality over to , and apply Jensen’s inequality to obtain the claimed statement.

Gradient Descent for strongly convex and smooth functions

As will see now, having both strong convexity and smoothness allows for a drastic improvement in the convergence rate. The key observation is the following lemma.

Lemma Let be -smooth and -strongly convex. Then for all , one has

Proof: Using the definitions it is easy to prove that is convex and -smooth, and thus using a result from the previous lecture one has

which gives the claimed result with straightforward computations. (Note that if then one just has to apply directly the above inequality to .)

Theorem Let be -smooth and -strongly convex, and let be the condition number of . Then Gradient Descent with satisfies

Proof: First note that by -smoothness one has

Now using the previous lemma one obtains

which concludes the proof.

Oracle lower bound for strongly convex and smooth functions

To simplify the proof of the next theorem we will consider the case of where . More precisely we assume now that we are working in rather than in . Note that in fact everything that we have done so far for first-order methods could have been done in an arbitrary Hilbert space rather than in , and we chose to work in only for the clarity of exposition.

Theorem For any black-box optimization procedure satisfying

there exists a -smooth and -strongly convex function such that for any one has

A consequence of this result is that the bound we derived in the previous section for Gradient Descent is not optimal, as the rate had a similar form than the above lower bound but with instead (which can be an important difference for large values of ). One can obtain a matching upper bound by resorting to Nesterov’s Accelerated Gradient Descent instead of the plain Gradient Descent, and we refer the interested reader to the book by Nesterov for more details on this.

Proof: The overall argument is similar to what we did in the lower bound for smooth functions, but the details are different. As usual we assume without loss of generality that . Let be the linear operator that corresponds to the infinite tridiagonal matrix with on the diagonal and on the upper and lower diagonals. Let . We consider now the following function:

We already proved that which easily implies that is -strongly convex and -smooth. Now as always the key observation is that for this function, thanks to our assumption on the black-box procedure, one necessarily has . This implies in particular:

Furthermore since is -strongly convex, one has

Thus it only remains to compute . This can be done by differentiating and setting the gradient to , which gives the following infinite set of equations

It is easy to verify that defined by satisfy this infinite set of equations, and the conclusion of the theorem then follows by straightforward computations.

Posted in Optimization | 2 Comments

## Guest post by Amir Ali Ahmadi: Sum of Squares (SOS) Techniques: An Introduction, Part I/II

Sum of squares optimization is an active area of research at the interface of algorithmic algebra and convex optimization. Over the last decade, it has made significant impact on both discrete and continuous optimization, as well as several other disciplines, notably control theory. A particularly exciting aspect of this research area is that it leverages classical results from real algebraic geometry, some dating back to prominent mathematicians like Hilbert. Yet, it offers a modern, algorithmic viewpoint on these concepts, which is amenable to computation and deeply rooted in semidefinite programming. In this post, we give an introduction to sum of squares optimization focusing as much as possible on aspects relevant to ORF523, namely, complexity and interplay with convex optimization. A presentation of this length is naturally incomplete. The interested reader is referred to a very nice and recent edited volume by Blekherman, Parrilo, and Thomas, the PhD thesis of Parrilo or his original paper, the independent papers by Lasserre and by Nesterov, the paper by Shor (translated from Russian), and the survey papers by Laurent and by Reznick. Much of the material below can be found in these references.

Polynomial Optimization

For the purposes of this post, we motivate the sum of squares machinery through the polynomial optimization problem:

(1)

where , , and are multivariate polynomials. A set defined by a finite number of polynomial inequalities (such as the set above) is called basic semialgebraic. Of course, we can write with polynomial inequalities only (by replacing with and ), or (unlike the case of linear programming) with polynomial equalities only (by replacing with , for some new variables ). We prefer, however, to keep the general form above since we will later treat polynomial inequalities and equalities slightly differently.

It follows from the quantifier elimination theory of Tarski and Seidenberg that (the decision version of) problem (1) is decidable in its greatest generality. In other words, we can test in finite time (on a Turing Machine) whether a set of polynomial inequalities and equalities are feasible. This is in sharp contrast to problem (1), appended with integer constraints, which is undecidable even when the polynomials have fixed degree and dimension.

The special case of problem (1) where the polynomials all have degree one is of course linear programming, which we can solve in polynomial time. Unfortunately though, as we will see in the complexity section of this post below, the problem quickly becomes intractable when the degrees increase from one ever so slightly. For example, unconstrained minimization of a quartic polynomial, minimization of a cubic polynomial over the sphere, or minimization of a quadratic polynomial over the simplex are all NP-hard.

The sum of squares methodology offers a hierarchy of polynomially sized semidefinite programming relaxations to cope with this computational intractability. It is quite different in philosophy from the approach taken by, say, the descent methods in nonlinear optimization. In particular, it makes absolutely no assumptions about convexity of the objective function , or the constraint set . Nevertheless, the hierarchy has a proof of asymptotic convergence to a globally optimal solution and in practice often the first few levels of the hierarchy suffice to solve the problem globally.

If we could optimize over nonnegative polynomials…

A point of departure for the sum of squares methodology is the observation that if we could optimize over the set of polynomials that take nonnegative values over given basic semialgebraic sets, then we could solve problem (1) globally. To see this, note that the optimal value of problem (1) is equal to the optimal value of the following problem:

(2)

Here, we are trying to find the largest constant such that is nonnegative on the set . This formulation suggests the need to think about a few fundamental questions: given a basic semialgebraic set as in (1), what is the structure of the set of polynomials (of, say, some fixed degree) that take only nonnegative values on ? Can we efficiently optimize a linear functional over the set of such polynomials? Can we even test membership to this set efficiently?

Observe that independent of the convexity of the set , the set of polynomials that take nonnegative values on it form a convex set! Albeit, as we see next, this convex set is not quite tractable to work with.

Complexity considerations

We first show that testing membership to the set of polynomials that take nonnegative values over a basic semialgebraic set is NP-hard, even when . In order to give a very simple reduction “from scratch”, we first prove this claim with the word “nonnegative” replaced by “positive”.

Theorem
Given a polynomial of degree , it is strongly NP-hard to decide if it is positive definite, i.e., if for all .

Proof: Recall the NP-complete problem of PARTITION problem from Seb’s earlier lecture: Given integers , decide if there is a way to split them into two bags such that the sum of the integers in one bag equals the sum of the integers in the other. It is easy to see that a PARTITION instance is infeasible if and only if the quartic polynomial

is positive definite (why?).

This already implies weak NP-hardness. Recall also from Seb’s lecture that PARTITION admits a simple pseudo-polynomial time algorithm based on dynamic programming. To obtain a strong NP-hardness result and rule out the possibility of a pseudo-polynomial time algorithm (unless P=NP), we give a slightly more involved reduction from ONE-IN-THREE 3SAT, taken from this paper. ONE-IN-THREE 3SAT is a close variant of 3SAT, which just like 3SAT is NP-hard in the strong sense. (The reason why we pick this problem over the more familiar 3SAT is that an equally straightforward reduction from the latter problem would only prove hardness of positivity testing for polynomials of degree .) In ONE-IN-THREE 3SAT, we are given a 3SAT instance (i.e., a collection of clauses, where each clause consists of exactly three literals, and each literal is either a variable or its negation) and we are asked to decide whether there exists a  assignment to the variables that makes the expression true with the additional property that each clause has exactly one true literal.

To avoid introducing unnecessary notation, we present the reduction on a specific instance. The pattern will make it obvious that the general construction is no different. Given an instance of ONE-IN-THREE 3SAT, such as the following

(3)

we define the quartic polynomial as follows:

(4)

Having done so, our claim is that for all (or generally for all ) if and only if the ONE-IN-THREE 3SAT instance is not satisfiable. Note that is a sum of squares and therefore nonnegative. The only possible locations for zeros of are by construction among the points in . If there is a satisfying Boolean assignment  to (3) with exactly one true literal per clause, then will vanish at point . Conversely, if there are no such satisfying assignments, then for any point in , at least one of the terms in (4) will be positive and hence will have no zeros.

Deciding if a polynomial is nonnegative—i.e., if for all —is also strongly NP-hard if we consider polynomials of degree or higher even degree. A simple reduction is from the matrix copositivity problem: Given a symmetric matrix , decide if for all . (Note the similarity to testing matrix positive semidefiniteness, yet the drastic difference in complexity.) To see the connection to polynomial nonnegativity, observe that the quartic homogeneous polynomial

with , is nonnegative if and only if is a copositive matrix.

Matrix copositivity arises in surprisingly many areas of optimization and more broadly computational mathematics. Its strong NP-hardness follows e.g. via a reduction from the STABLE SET problem (a.k.a. the INDSET problem), which can be found here. The main ingredient is a beautiful theorem of Motzkin and Straus (1965): The stability number of a graph with adjacency matrix satisfies

A quadratic programming formulation makes sum of squares techniques directly applicable to the STABLE SET problem, and in a similar vein, applicable to any NP-complete problem. We end our complexity discussion with a few remarks.

1.  The set of nonnegative polynomials and the set of copositive matrices are both examples of convex sets for which optimizing a linear functional, or even testing membership, is strongly NP-hard. This observation deserves some attention in view of the common misconception about “convex problems being easy.” Seb already placed a word of caution about this in his very first lecture, where he showed that practically any optimization problem can be “written” as a convex problem. So the algebraic/geometric structure of the set, beyond convexity, cannot be ignored.
2. Back to the polynomial optimization problem in (1), the reductions we gave above already imply that unconstrained minimization of a quartic polynomial is NP-hard. The aforementioned hardness of minimizing a quadratic form over the standard simplex follows e.g. from the Motzkin-Straus theorem above. Unlike the case of the simplex, minimizing a quadratic form over the unit sphere is easy. We have seen in a previous lecture already that this problem (although non-convex in this formulation!) is simply an eigenvalue problem. On the other hand, minimizing forms of degree over the unit sphere is NP-hard, due to a result of Nesterov.
3. Finally, we remark that for neither the nonnegativity problem nor the positivity problem did we claim membership in the class NP or co-NP. This is because these problems are still open! One may think at first glance that both problems should be in co-NP: If a polynomial has a zero or goes negative, simply present the vector at which this happens as a certificate. The problem with this approach is that there are quartic polynomials, such as the following,

for which the only zero takes bits to write down. Membership of these two problems in the class NP is much more unlikely. Afterall, how would you give a certificate that a polynomial is nonnegative?

To be continued…

Posted in Optimization | 1 Comment