ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 1

ECE580 Solution to Problem Set 2

These problems are from the textbook by Chong and Zak, 4th edition, which is the
textbook for the ECE580 Fall 2015 semester. As such, many of the problem statements
are taken verbatim from the text; however, others have been reworded for reasons of
efficiency or instruction. Solutions are mine. Any errors are mine and should be reported
to me, skoskie@iupui.edu, rather than to the textbook authors.

5.9 Level Sets

The functions f1 : R2 → R and f2 : R2 → R are defined by

f1 (x) = x21 − x22 , (1)

f2 (x) = 2x1 x2 . (2)

Draw the level sets C1 = {x : f1 (x) = 12} and C2 = {x : f2 (x) = 16} on the same
set of axes. Identify on the diagram the set of x such that f (x) = [12, 16]T .
Solution: The p solution is obtained using Matlab. The set C1 is the set of x such
that x2 = ± x21 − 12. The set C2 is the set of x such that x2 = 8/x1 . For x1 6= 0 the
intersection of these two level sets is the set of points x such that x21 −12−64/x21 = 0.
Let y = x21 . Then the set of x such that x21 − 12 − 64/x21 = 0 corresponds to

x41 − 12x21 − 64 = (x21 − 16)(x21 + 4) = 0.

The imaginary roots are x1 = ±2j. The real roots are x1 = ±4. The corresponding
real values of x2 are ±2.
The intersecting level sets are shown in Figure 1.
Figure 1 was generated by the following script:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%
%%%%% ECE569 Fall 2015, IUPUI
%%%%% Plot of Intersecting Level Sets for Problem 5.9 of Chong
%%%%% and Zak 4th ed.
%%%%% c. 2015 S. Koskie
%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

clf
clear x1a x1b x1c x2f1p1 x2f1p2 x2f1n1 x2f1n2 x2f2
x1a = -8:.01:-3.46;
x2f1p1 = real(sqrt(x1a.^2-12));
x2f1n1 = -real(sqrt(x1a.^2-12));
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 2

Problem 5.9 Level Sets

8

f1(x1,x2) = x21 − x22 = 12

2
x2

−2

−4
f (x ,x ) = 2x x = 16
2 1 2 1 2

−6

 2015 S. Koskie
−8
−8 −6 −4 −2 0 2 4 6 8
x1

Figure 1: Intersecting Level Sets for Problem 5.9

x1b = 3.46:.01:8;
x2f1n2 = -real(sqrt(x1b.^2-12));
x2f1p2 = real(sqrt(x1b.^2-12));
x1c = -8:.01:8;
x2f2 = 16./(2*x1c);
figure(5)
plot(x1a,x2f1p1,’b’)
hold
plot(x1a,x2f1n1,’b’)
plot(x1b,x2f1n2,’b’)
plot(x1b,x2f1p2,’b’)
plot(x1c,x2f2,’r’)
axis([-8,8,-8,8])
xlabel(’x_1’)
ylabel(’x_2’)
title(’Problem 5.9 Intersecting Level Sets’)
text(-5,5,’\color{blue}f_1(x_1,x_2) = x_1^2 - x_2^2 = 12’);
text(-1,-5,’\color{red}f_2(x_1,x_2) = 2x_1x_2 = 16’)
text(4,-7.5,’\copyright 2015 S. Koskie’)
grid
print -depsc p5p9.eps
hold

5.10 Taylor Series Expansions

Find the Taylor series expansion of f (x) about x0 neglecting terms of order three
and higher.
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 3

T
(a) f (x) = x1 e−x2 + x2 + 1 and x0 = 1 0 .


Solution:
We need the following factors that occur in the terms of the zeroth to second
order terms of the Taylor series.

f (x0 ) = 1 + 0 + 1 = 2.

h i
∂f ∂f
e−x2 (−x1 e−x2 + 1)
 
Df (x) = ∂x1 ∂x2 =
so
   
Df (x0 ) = 1 (−1 + 1) = 1 0 .

∂2f ∂2f
" #
−e−x2
 
2 ∂x21 ∂x1 ∂x2 0
F (x) = D f (x) = ∂2f ∂2f =
−e−x2 x1 e−x2
∂x1 ∂x2 ∂x22
so
 
0 −1
F (x0 ) = .
−1 1
Thus near x0

f (x) ≈ f (x0 ) + Df (x0 )(x − x0 ) + (x − x0 )T F (x0 )(x − x0 )/2 (3)

  
0 −1 x1 − x1,0 1
= 2 + x1 − x1,0 + [x1 − x1,0 x2 − x2,0 ]
−1 1 x2 − x2,0 2
= 2 + x1 − 1 + (x1 − 1)(−x2 )/2 + x2 (−x1 + x2 + 1)/2
= 1 + x1 − x1 x2 + x2 + x22 /2.

We check that the right hand side (rhs) equals f (x0 ) at x0 . The value of the
rhs is 1 + 1 − 0 + 0 + 0 = 2 as expected.
 T
(b) f (x) = x41 + 2x21 x22 + x42 and x0 = 1 1 .
Solution: We simplify this by noting that f (x) = (x21 + x22 )2 . Thus

f (x0 ) = 22 = 4.
 
Df (x) = 22 x1 (x21 + x22 ) 22 x2 (x21 + x22 )
so  
Df (x0 ) = 8 8 .
 
4(3x21 + x22 ) 8x1 x2
F (x) =
8x1 x2 4(x21 + 3x22 )
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 4

so  
16 8
F (x0 ) = .
8 16
Thus near x0

f (x) ≈ f (x0 ) + Df (x0 )(x − x0 ) + (x − x0 )T F (x0 )(x − x0 )/2

  
16 8 x1 − 1 1
= 4 + 8(x1 − 1) + 8(x2 − 1) + [x1 − 1 x2 − 1]
8 16 x2 − 1 2
= −12 + 8x1 + 8x2 + (16(x1 − 1)2 + 16(x1 − 1)(x2 − 1) + 16(x2 − 1)2 )/2
= −12 + 8((x1 + x2 ) + (x21 − 2x1 + 1) + (x1 x2 − (x1 + x2 ) + 1) + (x22 − 2x2 + 1))
= 12 − 16(x1 + x2 ) + 8(x21 + x1 x2 + x22 )

We check that the right hand side (rhs) equals f (x0 ) at x0 . The value of the
rhs is 12 − 32 + 24 = 4 as expected.
T
(c) f (x) = ex1 −x2 + ex1 +x2 + x1 + x2 + 1 and x0 = 1 0 .


Solution: We simplify this by noting that f (x) = ex1 (e−x2 + ex2 ) + x1 + x2 + 1.

Thus
f (x0 ) = e(2) + 1 + 0 + 1 = 2(e + 1).
Df (x) = ex1 (e−x2 + ex2 ) + 1 ex1 (−e−x2 + ex2 ) + 1
 

so  
Df (x0 ) = 2e + 1 1 .
ex1 (e−x2 + ex2 ) ex1 (−e−x2 + ex2 )
 
F (x) =
ex1 (−e−x2 + ex2 ) ex1 (e−x2 + ex2 )
so  
2e 0
F (x0 ) = .
0 2e
Thus near x0

f (x) ≈ f (x0 ) + Df (x0 )(x − x0 ) + (x − x0 )T F (x0 )(x − x0 )/2

  
  2e 0 x1 − 1 1
= 2(e + 1) + (2e + 1)(x1 − 1) + x2 + x1 − 1 x2
0 2e x2 2
2 2
= 2(e + 1) + (2e + 1)x1 + x2 − (2e + 1) + e(x1 − 1) + ex2
= 1 + (2e + 1)x1 + x2 + e(x21 − 2x1 + 1 + x22 )
= 1 + e + x1 + x2 + ex21 + ex22

We check that the right hand side (rhs) equals f (x0 ) at x0 . The value of the
rhs is 1 + e + 1 + 0 + e + 0 = 2(e + 1) as expected.
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 5

6.1 Consider the problem

minimize f (x)
subject to x ∈ ω,
where f ∈ C 2 . For each of the following definitions of Ω, x∗ , and f , determine
whether x∗ is, is not, or might be a local minimizer.
T T
(a) f : R2 → R, Ω = {x : x1 ≥ 1}, x∗ = 1 2 , ∇f (x∗ ) = 1 1
 
.
∗ T ∗
Solution: x is on the boundary so we must verify that d ∇f (x ) ≥ 0 for
all feasible directions d. To be feasible, d1 must be nonnegative, and d2 is
arbitrary, so long as d 6= 0.

dT ∇f (x∗ ) = d1 + d2 .

Since d2 is arbitrary, this quantity may be negative, thus x∗ is not a local

minimizer.
T T
(b) f : R2 → R, Ω = {x : x1 ≥ 1, x2 ≥ 2}, x∗ = 1 2 , ∇f (x∗ ) = 1 0
 
.
∗
Solution: In this case, x is on the boundary at a corner. Thus, to be feasible,
d must have both d1 and d2 nonnegative.

dT ∇f (x∗ ) = d1 ≥ 0,

so the FONC is satisfied and x∗ could be a local minimizer.

T T
(c) f : R2 → R, Ω = {x : x1 ≥ 0, x2 ≥ 0}, x∗ = 1 2 , ∇f (x∗ ) = 0 0
 
,
∗
F (x ) = I.
Solution: x∗ is an interior point. Since the gradient is zero at x∗ , the FONC
is satisfied. Since the Hessian is positive definite, the SOSC is satisfied so x∗
is a local minimizer.
T T
(d) f : R2 → R, Ω = {x : x1 ≥ 1, x2 ≥ 2}, x∗ = 1 2 , ∇f (x∗ ) = 1 0
 
,
 T
1 0
F (x∗ ) = .
0 −1
Solution: x∗ is on the boundary at a corner. Both d1 and d2 must be non-
negative as before, and not both zero. The Hessian is not positive definite,
and
dT F (x∗ )d = d21 − d22 .
The SONC is not satisfied for any d having d2 > d1 . Thus, regardless of
whether the FONC is satisfied, x∗ is not a local minimizer.
6.8 Gradients, Directional Derivatives, and Hessians, FONC and SONC
Consider the following function f : R2 → R:
   
T 1 2 T 3
f (x) = x x+x + 6.
4 7 5
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 6

To simplify the calculations, we symmetrize to obtain

   
T 1 2 T 1 3
x x=x x,
4 7 3 7
and rewrite the function as
   
T 1 3 T 3
f (x) = x x+x + 6.
3 7 5
 T
(a) Find the gradient and Hessian of f at x0 = 1 1 .
Solution: The gradient is
   
1 3 3
∇f (x) = 2 x+ ,
3 7 5
which, evaluated at x0 is
      
1 3 1 3 11
∇f (x0 ) = 2 + = .
3 7 1 5 25

The Hessian is  
2 1 3
D f =2 ,
3 7
which is a constant, so independent of x0 .
(b) Find the directional derivative of f at x0 with respect to a unit vector in the
direction of maximal rate of increase.
Solution: The direction of maximum increase of f at a point is the value of
the gradient at that point, in this case ∇f (x0 ). The directional derivative with
along the direction d is
∂f
(x) = dT ∇f (x)
∂d
. Normalizing, we obtain
   
ˆ
p 1 11 1 11
d = ∇f (x0 )/ (∇f (x0 )) ∇f (x0 ) = √
T =√ .
121 + 625 25 746 25
Thus the directional derivative with respect to a unit vector along the maximal
rate of increase is
(∇f (x))T ∇f (x) p
p = (∇f (x))T ∇f (x). (4)
(∇f (x))T ∇f (x)
At x0 we have
∂f √
(x0 ) = 746
∂ dˆ
.
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 7

(c) Find a point that satisfies the FONC and determine whether it satisfies the
SONC.
Solution: Such a point must satisfy ∇f (x) = 0, so we need
  ∗   
1 3 x1 3
∇f (x0 ) = 2 ∗ + = 0.
3 7 x2 5
T
One such x∗ =

3/2 −1 .
 
2 6
To determine whether it satisfies the SONC we must determine whether
6 14
is positive definite. Solving for the eigenvalues we obtain
 
 s−2 −6  2
 −6 s − 14  = (s − 2)(s − 14) − 36 = s − 16s − 12.
 

Using the quadratic formula we find that

√ √
s = (16 ± 176 + 48)/2 = 8 ± 4 14,

so the matrix is indefinite and the point does not satisfy the SONC. A more
efficient way to determine this for a two by two matrix is to take the deter-
minant. The determinant is the product of the eigenvalues, so being negative
indicates that the two eigenvalues have opposite signs.

6.10 Finding Minimizers

Consider the function f : R2 → R
   
T2 5 T 3
f (x) = x x+x +7
−1 1 4
   
T 2 2 T 3
= x x+x +7
2 1 4
   
0 1
(a) Find the directional derivative of f at x0 = in the direction d = .
1 0
Solution    
2 2 3
∇f (x) = 2 x+ .
2 1 4
The value of the requested directional derivative is thus
     
T
  4 4 0 3
d ∇f (x0 ) = 1 0 +
4 2 1 4
= 7.
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 8

(b) Find all points that satisfy the FONC.

Solution Points that satisfy the FONC are points at which the gradient is
zero.    
4 4 ∗ 3
∇f (x) = x +
4 2 4
has solution
 −1 

∗ 4 4 −3
x =
4 2 −4
  
−1/4 1/2 −3
=
1/2 −1/2 −4
 
−5/4
= .
1/2

Because the inverse exists, the result is unique. The SONC is not satisfied
because the determinant of the Hessian is negative, so the two by two matrix
has one positive and one negative eigenvalue, hence is indefinite. Accordingly,
f does not have a minimizer.

6.16 Constrained Feasible Set

Consider the problem
minimize f (x)
subject to x ∈ Ω
where f (x) = 4x21 − x22 , and Ω = {x : x21 + 2x1 − x2 ≥ 0, x1 ≥ 0, x2 ≥ 0}.

(a) Does the point x∗ = 0 satisfy the FONC?

Solution The origin is on the boundary of Ω, in fact on a corner. Thus we
must test all feasible directions to determine whether dT ∇f (x∗ ) ≥ 0 for all of
them. The slope of the curve x2 = x21 + 2x1 is 2x1 + 2. This yields a slope of 2
 T
at the origin, so we must consider all directions d1 d2 for which d2 ≥ 2d1
down to the horizontal. The condition for the FONC is thus
 
T 8x1
d ≥ 0.
4x2
In fact, since we are looking at x∗ , the gradient is zero so the FONC is satisfied
for all feasible directions.
(b) Does the point x∗ = 0 satisfy the SONC?
 
8 0
Solution: The Hessian is , independent of the value of mathbf x∗ .
0 −2
 
T 8 0
d d = 8d21 − 2d22 .
0 −2
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 9

 
T 8 0
Now so long as d2 ≤ 2d1 , we will have d d ≥ 0 so the SONC is
0 −2
satisfied.
(c) Is the point x∗ = 0 a local minimizer of f over Ω?
A local minimizer of f over Ω, could occur anywhere on the boundary or the
interior of Ω. Because the Hessian is indefinite everywhere, the minimizer does
not occur in the interior of Ω. If we check the section of the boundary along
the x1 axis, the FONC is satisfied for any direction vector that has d2 = 0.
However, this does not cover all feasible directions, so the FONC is not satisfied
for nonzero points on the positive x1 axis. The last section of the boundary is
the curve x2 = x21 + 2x1 . (x21 + 2x1 )2 = x41 + 4x31 + 4x21 , so on this boundary,
f (x) = 4x21 − 2(x21 + 2x1 )2 = −2x41 − 8x31 , which is negative for all positive x1
as one approaches the origin along the curve. Thus the origin is not a local
minimizer of f over Ω.

7.8 Application of Newton’s Method of Tangents to Root Finding

x−1
Let g(x) = eex +1 , x ∈ R. Notice that the denominator is never zero and that the
unique zero of the numerator, hence g is x = 0.
(a) Apply Newton’s Method of Tangents
Solution: The algorithm is
g(x(k) )
x(k+1) = x(k) − (5)
g 0 (x(k) )
where g 0 := dg/dx.
dg d x
(x) = (e − 1)(ex + 1)−1
dx dx
= ex (ex + 1)−1 + (ex − 1)(−1)(ex )(ex + 1)−2
ex (ex + 1) − ex (ex − 1)
=
(ex + 1)2
2ex
= .
(ex + 1)2
We will be dividing by this quantity, so we must verify that it is never zero.
The numerator is indeed never zero. Substituting into (5), we obtain
! !
x(k) x(k) 2
e − 1 (e + 1)
x(k+1) = x(k) −
ex(k) +1 2ex(k)
(k)
(k)1 − e2x
= x +
2ex(k)
(k) (k)
(k) e−x − ex
= x +
2
= x(k) − sinh x(k) .
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 10

(b) Find an initial condition x(0) such that the algorithm cycles, specifically such
that
x(0) = x(2) = x(4) = · · · .
Solution The graph of sinh x is odd, i.e. sinh(−x) = − sinh(x), see Figure 2.
Thus for the algorithm to cycle such that x(2k) = x(0) for all nonnegative
integers k, we need x(k+1) = −x(k) . This is true when

x(k+1) = −x(k) = x(k) − sinh x(k) .

Equivalently, we need 2x(k) = sinh xk , and, in particular

2x(0) = sinh x(0)

(0)
, which, according to Matlab occurs when x(0) = x∗ ≈ ±2.1773.
4 Hyperbolic Sine Function
x 10
1.5

0.5
sinh(x)

−0.5

−1

−1.5
−10 −8 −6 −4 −2 0 2 4 6 8 10
x

Figure 2: Graph of the odd function sinh(x)

(0)
(c) If kx0 k > x∗ , then kx(k+1) k > kx(k) k for all nonnegative integers k and the
sequence generated by the algorithm diverges. Examples of convergence, cy-
cling, and divergence are shown in Figures 3 through 5. This is a good example
showing why a picture is not a proof. The value of x(0) used for the cycling
example is an approximation, chosen carefully so that the algorithm would ap-
pear to cycle for the ten iterations shown. It converges or diverges later. It is
(0)
likely that the actual value of x∗ is irrational so can never be captured with
finite precision.

7.10
ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 11

(0)
Newton Algorithm Converging for x =2
2

1.5

0.5
x(k)

−0.5

−1

−1.5

−2
1 2 3 4 5 6 7 8 9 10 11
k

Figure 3: ’Fast convergence of Newton Algorithm for Good x(0) ’

(0)
Newton Algorithm Cycling for x = 2.177319
2.5

1.5

0.5
x(k)

−0.5

−1

−1.5

−2

−2.5
1 2 3 4 5 6 7 8 9 10 11
k

Figure 4: Apparent Cycling of Newton Algorithm’

ECE580 Fall 2015 Solution to Problem Set 2 October 5, 2015 12

(0)
Newton Algorithm Diverging for x = 2.177348
250

200

150
x(k)

100

50

−50
1 2 3 4 5 6 7 8 9 10 11
k

Figure 5: Divergence of Newton Algorithm for Bad x(∗) ’