You are on page 1of 6

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395 -0056

Volume: 02 Issue: 04 | July-2015

p-ISSN: 2395-0072

www.irjet.net

A New Non-monotonic Self-adaptive Trust Region Algorithm


with Non-monotonic Line Search
Xiao Wu1, Qinghua Zhou1,*, Changyuan Li1, Xiaodian Sun2
1College

of Mathematics and Information Science, Hebei University, Baoding, Hebei Province, China, 071002.

2Laboratory

for Statistical Genomics and Systems Biology, Department of Environmental Health, University of

Cincinnati College of Medicine, 3223 Eden Ave. ML56, Cincinnati OH 45267-0056, USA
(wuxiao616@163.com, qinghua.zhou@gmail.com(*corresponding author), mbydlcy@126.com, sunxd@uc.edu)
---------------------------------------------------------------------***---------------------------------------------------------------------

Abstract. We consider an efficient trust-region


framework which employs a new non-monotone
search

technique

problems.

Unlike

for
the

unconstrained
traditional

line

solution. At the

k th step, it calculates a trial step d k by

solving the following trust region sub-problem

min f k (d ) = f k + g kT d +

optimization

1 T
d Bk d
2

(2)

s.t P d P D k .

non-monotonic

gk = f ( xk )

trust-region method, the new point is given by the

where

non-monotonic Wolfe line search at each iteration, and

function value and the gradient vector at the current

the trust region radius is updated at a variable rate.


The new algorithm solves the trust region sub-problem
only once at each iteration. Under certain conditions,
the global convergence of the algorithm is proved.

Key words: non-monotonic, self-adaptive, trust region,

f k = f ( xk )

and

are

the

xk respectively, Bk is an n n
symmetric matrix which may be the exact Hessian H ( xk )
or the quasi-Newton approximation and D k > 0 is the
trust region radius. In this paper, the notation PP
n
denotes the Euclidean norm on R . After obtaining d k ,
the trust region method computes the ratio rk between
approximation iterate

the actual reduction and the predicted reduction of the


objective function to check if

d k is acceptable. Then the

D k is also updated according to the


value of rk , since rk reflect the extent to which the
quadratic model f k (d ) approximates the objective
function f ( xk + d ) . Hei [1] proposed a self-adaptive
trust region algorithm, in which D k is updated by
R -function at a variable rate.
trust region radius

line search

1. INTRODUCTION
We consider the following unconstrained optimization
problem

minn f ( x).
x R

where

(1)

f ( x) is twice continuously differentiable.

Trust region method is a very effective and robust


technique for solving problem (1) to find the global optimal
2015, IRJET.NET- All Rights Reserved

Non-monotonic line search technique for unconstrained


optimization is first proposed by Grippo et al. [2]. Due to its
high

efficiency,

many

authors

generalized

the

non-monotone technique to trust region method and


proposed non-monotone trust region method, see [3,4].
Yang and Sun [5] proposed a new region algorithm where

Page 60

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395 -0056

Volume: 02 Issue: 04 | July-2015

p-ISSN: 2395-0072

www.irjet.net

the trust region radius is updated at a variable rate.

Rh (t ) defined
R = (- , + ) with the parameter h (0,1) is
R -function if and only if it satisfies:
Rh (t ) is non-decreasing in (- , + ) ;
lim Rh (t ) = b1 , where b1 (0,1) is a small

Moreover, the new algorithm performed a backtracking

Definition 1 Any one-dimensional function

line search from the failed point instead of resolving the

in

trust region sub-problem.

an
1)

Wang [6] presented a non-monotonic trust region

2)

algorithm

constant;

with

line

search.

Unlike

traditional

t -

Rh (t ) 1- g1 , for all t < h where g1 (0,1- b1 ) is

non-monotonic trust region algorithms, the new point is

3)

given by the non-monotonic Wolfe line search at each

a constant;

iteration, and trust region radius is adjusted by

4)

sub-problem approximate solution and line search step

Rh (t ) = 1 + g 2 , where g 2 (0, + ) is a constant;


5) lim Rh (t ) = b 2 , where b 2 (1 + g 2 , + ) is a

length.

constant.

Mosoud

Ahookhosh

[7]

introduced

variant

non-monotone strategy and incorporate it into trust-region

t +

Theorem 2 An
satisfies

framework to construct more reliable approach. The new

0 < b1 Rh (t ) 1- g1 < 1, " t (- , h).

non-monotone strategy is a convex combination of the


maximum of function value of some prior successful

a k by

sub-sequent Wolfe line search

Now, we give a description of our new algorithm.


Algorithm 1

f ( xk + a k dk ) Rk + ba k g dk .
T
k

g ( xk + a k dk )T dk g gkT dk .
where

(3)

1 < 1+ g 2 Rh (t ) b 2 < + , " t [h, + ). (4)

iterates and the current function value.


In this paper, we determine the step-length

R -function Rh (t ) (where h (0,1) )

0 < b < g < 1 and

Rk = hk fl ( k ) + (1- hk ) f k

hk [hmin , hmax ] for


hmin [0,1)
and
hmax [hmin ,1] .And we improve the self-adaptive trust

x0 , B0 Rn n , D 0 > 0, e 0,0 < b1 < 1,


0 < g1 < 1- b1 , g 2 > 0, b 2 > 1+ g 2 ,0 < c1 < 1, N > 0,
0 < b < g < 1 . Set k = 0, m(0) = 0.
Step 2 Compute g ( xk ) . If P g ( xk ) P e , stop.
Step 3 Solve the sub-problem (2) for d k satisfying
predk t P gk P min{D k ,P gk P P Bk P}. (5)
Step 1 Given

gkT dk - t P gk P min{D k ,P gk P P Bk P}. (6)

where

region algorithm proposed in [1] by adopting the above


ideas: backtracking line search and non-monotonic

fl ( k ) , Rk and r k = ( Rk - f ( xk + dk )) /
pred k ,find the step-length a k satisfying in

Step 4 Compute

f ( xk + a k dk ) Rk + ba k gkT dk .

technique. Now, we can outline our new non-monotone

(7)

self-adaptive trust region algorithm with non-monotonic

g ( xk + a k dk )T dk g gkT dk .

line search.

2. NEW ALGORITHM
Before describing the new algorithm, we need the following
definition and conclusion ([5]).
2015, IRJET.NET- All Rights Reserved

(8)

xk + 1 = xk + a k dk , go to Step 5.
Step 5 D k + 1 = Rc1 (r k )D k . Update the matrix Bk + 1 by a
Set

quasi-Newton formula, set

m(k ) = min{m(k - 1) + 1, N}, k = k + 1.


Page 61

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395 -0056

Volume: 02 Issue: 04 | July-2015

p-ISSN: 2395-0072

www.irjet.net

go to Step 1.

This show that the sequence

{ fl ( k ) } is a decreasing

sequence.
Lemma 4 Suppose that (H1)-(H4) hold and the sequence

3. CONVERGENCE ANALYSIS

{xk } be generate by Algorithm 1, then we have


lim f ( xl ( k ) ) = lim f ( xk ).

To analyze the new algorithm, we make the following

(15)

assumptions.
Proof. Using (7) and (11), we obtain

Assumption

f ( xl ( k ) ) = f ( xl ( k )- 1 + a l ( k )- 1dl ( k )- 1 )

f ( x) is continuously differentiable and has a lower


bound on the level set L( x) = {x | f ( x) f ( x0 )} is
(H1)

Rl ( k )- 1 + ba l ( k )- 1 g Tl ( k )- 1 dl ( k )- 1

compact.

Bk is an uniformly bounded matrix, i.e.


there exist b > 0 such that P Bk P b for all k N .
(H3) f ( x) is uniformly continuous, there exist a
constant L > 0 such that
P f ( x) - f ( y) P L P x - y P, " x, y R n . (9)
(H4) There exists a constant e such that the trial step
d k satisfies
(10)
P dk P e P gk P.

fl (l ( k )- 1) + ba l ( k )- 1 g Tl ( k )- 1 dl ( k )- 1.

(H2) The matrix

Preceding inequality, along with

b > 0 and Corollary 2.1

in [8] implies that

lim a l ( k )- 1 g Tl ( k )- 1 dl ( k )- 1 = 0.

Thus using (10), we can conclude that

lim a l ( k )- 1 P dl ( k )- 1 P= 0.

The rest of the proof is similar to a theorem in [2].

{xk } be generated
by Algorithm 1, then the sequence { f l ( k ) } is a decreasing

Corollary 5 Suppose that the sequence

sequence.

generated by Algorithm 1, then we have

Lemma 3 Suppose that the sequence

Proof.

Using definition of

Rk and fl ( k ) , we observe

lim Rk = lim f ( xk ).

that

Rk = hk fl ( k ) + (1- hk ) f k
hk fl ( k ) + (1- hk ) fl ( k )

(16)

Proof. The proof is similar to that of Corollary 8 in [7].


(11)

a k be
sequence generated by Algorithm 1 and {D k } are
bounded upper, if g ( x) is uniformly continuous satisfies
P gk P e > 0 " k.
where e > 0 , then there exists a constant a > 0 such
Lemma 6 Suppose that (H2) and (H3) hold. Let

= fl ( k )

f ( xk + a k d k ) Rk + ba k g kT d k
Rk .

(12)

that

These two inequalities show that

f k + 1 Rk fl ( k ) .
(13)
We have m(k + 1) m(k ) + 1 , thus from the definition
of f l ( k + 1) and (13), we can write
fl ( k + 1) = max { f k - j + 1}
0 j m ( k + 1)

{xk } be

max { f k -

0 j m ( k )+ 1

j+ 1

= max{ fl ( k ) , f k + 1}
fl ( k ) .
2015, IRJET.NET- All Rights Reserved

(14)

a k a " k.

(17)

Proof. By the same way as in the proof of Lemma 5.1 in [9],


we have the conclusion.
Lemma 7 If
sequence

g ( x) is uniformly continuous and the

{xk } be generated by Algorithm 1 satisfies


P gk P e > 0.
(18)
Page 62

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395 -0056

Volume: 02 Issue: 04 | July-2015

p-ISSN: 2395-0072

www.irjet.net

where e > 0 is a constant, then there exists a constant

c > 0 such that

c
Dk
, k = 0,1,L
Mk
where

f ( xk + d k ) - Rk
f ( xk + d k ) - f ( xk )

(19)

M k is defined by M k = 1 + max P Bk P.

d kT [ f ( xk + qd k ) - f ( xk )]d q + f ( xk )T d k

< g kT d k +

1 i k

f ( x) is uniformly continuous, there


exist a positive number s such that
d kT [ f ( x + d ) - f ( x)]

t e (1- c1 )
P d k P.
2

Proof. Because

(20)
1
t (1- c1 )e P d k P.
2
is satisfied for all P d k P< s . We show by induction that
(19) holds with

c = min{D 0 M 0 , b1s M 0 , b1e, t (1- c1 )b1e}. (21)


when k = 0 , it is obvious that D 0 c M 0 , so (19)
holds for

k = 0 . Assume that (19) is true for k , we will

prove that (19) holds for


Since

k + 1.

M k is non-decreasing, to prove (19) for k + 1, it

From (23) and (24), we have

(1- c1 )( gkT dk + t e P dk P 2) > c1dkT Bk dk 2. (25)


Moreover, we have by (5) and (18) that

- gkT dk - dkT Bk dk 2 t e min{D k , e P Bk P}. (26)


Multiplying (26) by

(1- c1 ) and combining it with (25),

we get

D 2k P Bk P
- d kT Bk d k
> 2(1- c1 )t e min{D k , e P Bk P} - (1- c1 )t e P d k P

suffices to establish the relation

D k+ 1

(24)

c
.
Mk

(22)

(1- c1 )t e{2 min{D k , e P Bk P} - D k }


= (1- c1 )t e min{D k , 2e P Bk P- D k }.

D k + 1 = Rc1 (r k )D k , it follow form Theorem 2 that


when r k c1 , D k + 1 D k c M k . In this case, relation
Since

(22) is trivial. Therefore, in the remainder of proof,

r k < c1 , thus D k+ 1 is in the range b1D k D k + 1


(1- g1 )D k .
If P d k P s , then
D k + 1 b1D k

Using the above condition, we can give a constant lower


bound on the product

D k P Bk P> min{(1- c1 )t e, e} c b1 .
always holds. So

D k + 1 b1D k > c P Bk P> c M k .


Thus, (19) holds for k + 1 . Therefore, by induction, (19)
holds for all

k.

b1 P d k P
b1s

Theorem 8 Assume that Assumption holds. Let

b1s M 0 M k

the sequence generated by Algorithm 1, then

c Mk .
If

lim inf P gk P= 0.

P dk P< s , it follows from Algorithm 1 that

f ( xk + d k ) - Rk > - c1 (f k (0) - f k (d k ))
= c1 ( g kT d k + d kT Bk d k 2).

{xk } be
(27)

Proof. We prove it by contradiction. Assume that (27) is


not true, that is, there exists a constant e > 0 such that
(23)

P gk P e, " k .
It follows from (6), (7), (11) and Lemma 6 that

From (20) and Lemma 2.2. in [8], we know that

2015, IRJET.NET- All Rights Reserved

Page 63

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395 -0056

Volume: 02 Issue: 04 | July-2015

p-ISSN: 2395-0072

www.irjet.net

[2]Grippo L., Lampariello F., Lucidi S., (1986).

fl ( k ) = f ( xl ( k )- 1 + a l ( k )- 1dl ( k )- 1 )
Rl ( k )- 1 + ba l ( k )- 1 glT( k )- 1dl ( k )- 1
min{D
Rl ( k )- 1 - t b ae
l ( k )- 1 , e b}

nonmonotonic line search technique for Newtons


(28)

trust-region for unconstrained optimization problems.


Applied Mathematics and Computation. 163, 707-716.

Using Corollary 2.1 in [8] , from (28), we have

lim D l ( k )- 1 = 0.

(29)

[4]Sun W., (2004). Non-monotonic trust region method for


solving optimization problems. Applied Mathematics

For k > M , we have

k - N k - m(k ) l (k ) k

and hence

0 k - l (k ) N .

and Computation. 156, 159-174.


[5]Yang Yang, Wenyu Sun., (2007). A New Non-monotonic

(30)

D k and Theorem 2,
j
j
b1 D k D k + j b 2 D k holds for all j . Thus, it follows

By the updating formula of


from (30) that

Self-adaptive Trust Region Algorithm with Line Search.


Chinese Journal of Engineering Mathrmatics. 24,
788-794.
[6] X. Wang, C.L. Wang., (2014). A new nonmonotone

b1M + 1D l ( k )- 1 D k b 2M + 1D l ( k )- 1.
Then, from (29), we have lim D k
in Lemma 7.

method. SIAM J.Numer.Anal. 23, 707-716.


[3]Fu J H, Sun W., (2005). Non-monotone adaptive

min{D
fl (l ( k )- 1) - t b ae
l ( k )- 1 , e b}.

= 0 . It contradicts (19)

trust-region method with nonmonotone line search(in


Chinese).

Science

and

Mathematics.

34,

179-186.
[7]M.

4. CONCLUSIONS

System

Ahookhosh,

K.

Amini.,

(2002).

An

efficient

nonmonotone trust-region method for unconstrained


optimization. Numer Algor. 59, 523-540.

In this paper, we proposed a new non-monotone

[8]M. Ahookhosh, K. Amini and M.R. Peyghami., (2012). A

self-adaptive trust region algorithm with non-monotone

nonmonotone trust-region line search method for

line search. After we analyzed the properties of the new

large-scale

algorithm, the global convergence theory is proved. In the


near future, we would like to investigate some new type of
combinations composed of different functions values in
order to sufficiently use the information which the
algorithm has already derived.

unconstrained

optimization.

Applied

Mathematical Modelling. 36, 478-487.


[9] S.M. Pang, L.P Chen., (2011). A trust-region method with
nonmonotone line search(in Chinese). Mathematica
numerica sinca. 33, 48-56.

ACKNOWLEDGEMENTS
This work is supported by the National Natural Science
Foundation of China (61473111) and the National Science
Foundation of Hebei Province (Grant No. A2014201003,
A2014201100).

REFERENCES
[1]Hei L., (2003). A self-adaptive trust region algorithm.
Journal of Computational Mathematics. 21, 229-236.
2015, IRJET.NET- All Rights Reserved

Page 64

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395 -0056

Volume: 02 Issue: 04 | July-2015

p-ISSN: 2395-0072

www.irjet.net

BIOGRAPHIES
Xiao Wu:

Postgraduate student

at College of Mathematics and


Information

Science,

Hebei

University, China. Her research


interest

mainly

concerns

non-monotone line search and


trust region methods .

Qinghua Zhou: Professor in HeBei


University. Major interest includes
Optimization

Theory

and

its

Applications.

Changyuan Li: Postgraduate student at


the

College

of

Mathematics

and

Information Science, Hebei University,


China. His research interest includes
non-monotone trust region methods
and derivative-free methods.

Xiaodian Sun: Research Scientist at


University of Cincinnati College of
Medicine.
includes
inference,

Her

research

dynamic
time

interest

modeling
series,

and

statistical

learning.

2015, IRJET.NET- All Rights Reserved

Page 65