CONVEX OPTIMIZATION CHEAT SEET

MARK SCHMIDT

See Nesterovs book for proofs of the below.

We say that a function f is convex if for all x and y on its domain and all 0 1 we have

f (x + (1 )y) f (x) + (1 )f (y).

If f is differentiable, equivalent definitions are that

f (y) f (x) + hf 0 (x), y xi,

hf 0 (x) f 0 (y), x y)i 0.

If f is twice-differentiable, an equivalent definition is that

2 f (x)  0.

For a differentiable convex f , the following conditions are equivalent to the condition that the gradient f 0 is
L-Lipschitz continuous:

kf 0 (x) f 0 (y)k Lkx yk

L
f (y) f (x) + hf 0 (x), y xi + kx yk2
2
1
f (y) f (x) + hf 0 (x), y xi + kf 0 (x) f 0 (y)k2
2L
hf 0 (x) f 0 (y), x yi Lkx yk2
1
hf 0 (x) f 0 (y), x yi kf 0 (x) f 0 (y)k2
L
(1 )L
f (x + (1 y)) f (x) + (1 )f (y) kx yk2
2
(1 ) 0
f (x + (1 )y) f (x) + (1 )f (y) kf (x) f 0 (y)k2 .
2L
You can define Lipschitz continuity under a different norm, and in this case the first condition becomes
kf 0 (x) f 0 (y)kq Lkx ykp where k kp and k kq are dual norms. For all the other inequalities, you replace
all instances of kx yk with kx ykp and kf 0 (x) f 0 (y)k with kf 0 (x) f 0 (y)kq .
For twice-differentiable f , any of the above are equivalent (under the Euclidean norm) to

2 f (x)  LI.

The following conditions are equivalent to the condition that a differentiable f is -strongly convex:

x 7 f (x) kxk2 is convex
2

f (y) f (x) + hf 0 (x), y xi + kx yk2
2
hf 0 (x) f 0 (y), x yi kx yk2
(1 )
f (x + (1 )y)) f (x) + (1 )f (y) kx yk2 .
2
1
The following are not equivalent to -strong convexity but are implied by it:
kf 0 (x) f 0 (y)k kx yk
1
f (y) f (x) + hf 0 (x), y xi + kf 0 (x) f 0 (y)k2
2
1 0
hf 0 (x) f 0 (y), x yi
kf (x) f 0 (y)k2 .

For a twice-differentiable f strong-convexity is equivalent to:
2 f (x)  I.
If f is -strongly convex and f 0 is L-Lipschitz continuous then we have
L 1
hf 0 (x) f 0 (y), x yi kx yk2 + kf 0 (x) f 0 (y)k2 .
+L +L