Nonlinear Regression

Sujantoko
Mathematic Engineering
Ocean Engineering FTK ITS

Nonlinear Regression
Some popular nonlinear regression models:
1. Exponential model:

( y = aebx )

2. Power model:

( y = axb )
ax

=
y

b
+
x

3. Saturation growth model:

4. Polynomial model:

( y = a0 + a1x + ... + amx m )

Nonlinear Regression
Given n data points ( x1, y1), ( x 2, y 2), ... , ( xn, yn) best fit y = f (x )
to the data, where f (x ) is a nonlinear function of x .
( xn , yn )

(x , y )
2

y = f (x)

(x , y )
i

(x , y )
1

y f ((xx )
i

Figure Nonlinear regression model for discrete y vs.

Figure.
vs x data

Regression
g

Exponential Model

Exponential Model
bx
Given ( x1 , y1 ), ( x 2 , y 2 ), ... , ( x n , y n ) best fit y = ae to the data.

(x , y )
n

( x2 , y2 )
( xi , yi )

(x , y )
1

y = aebx
y f (x )
i

Figure. Exponential model of nonlinear regression for y vs. x data

Fi di Constants
Finding
C
off Exponential
E
i l Model
M d l
Th sum off th
The
the square off th
the residuals
id l is
i defined
d fi d as
n

Sr = y ae
i =1

bx 2

Differentiate with respect to a and b

n
S r
b i
b i
= 2 y i ae bx
e bx
=0
a
i =1

)(

)(

n
S r
bxi
bxi
= 2 y i ae
axi e = 0
b
i =1

Finding Constants of Exponential Model

Rewriting the equations, we obtain

yi e

bxi

i =1

y i xi e

i =1

bxi

+ ae

2bxi

=0

2bxi

=0

i =1

a xi e
i =1

Finding constants of Exponential Model

Solving the first equation for a yields
n

a=

yi e

bxi

i =1
n

2bxi
e

i =1

Substituting
g a back into the previous equation
n

y i xi e

i =1

bxi

yi e

i =1
n

bxi

2bxi
e

2bxi
x
e
=0
i

i =1

i =1

The constant b can be found through numerical

methods such bisection method.
8

Example 1-Exponential
1 Exponential Model
Many patients get concerned when a test involves injection of a
radioactive material. For example for scanning a gallbladder, a
few drops of Technetium-99m isotope is used. Half of the
techritium-99m would be gone in about 6 hours. It, however,
takes about 24 hours for the radiation levels to reach what we
are exposed to in day-to-day activities. Below is given the
relative intensity of radiation as a function of time.

Table Relative intensity of radiation as a function of time.

Table.
time
t(hrs)

1 000
1.000

0 891
0.891

0 708
0.708

0 562
0.562

0 447
0.447

0 355
0.355

Example 1-Exponential Model cont.

The relative intensity is related to time by the equation
t

= Ae

Find:
a) The value of the regression constants A and
b) The
h half-life
h lf l f off Technium-99m
h
99
c) Radiation intensity after 24 hours

10

Plot of data

11

Constants of the Model

= Ae t

The
h value
l off is found
f
db
by solving
l
the
h nonlinear
l
equation
n

f ( ) = i t i e

ti

i =1

ti

e
i

i =1
n

2ti
e

i =1
n

A=

e
i
i =1
n

2 ti
e

i =1

12

2ti
t
e
=0
i

i =1

Setting up the Equation in MATLAB

n

f ( ) = i t i e
i =1

ti

ie

i =1
n

ti

ti e

2ti i =1

2ti

=0

i =1

t (h
(hrs))
0
1
3
5
7
9

1.000 0.891 0.708 0.562 0.447 0.355

13

Setting
g up
p the Equation
q
in MATLAB
n

f ( ) = i t i e
i =1

ti

ie

i =1
n

ti

2ti
t
e
=0
i

2ti i =1

= 0.1151

i =1

t=[0 1 3 5 7 9]
gamma=[1
ga
a [ 0.891
0 89 0.708
0 08 0.562
0 56 0.447
0
0.355]
0 355]

syms lamda
sum1=sum(gamma.*t.*exp(lamda*t));
sum2=sum(gamma.*exp(lamda*t));
sum3=sum(exp(2*lamda*t));
sum4=sum(t *exp(2*lamda*t));
sum4=sum(t.*exp(2*lamda*t));
f=sum1-sum2/sum3*sum4;
14

Calculating the Other Constant

The value of A can now be calculated
6

A=

e
i =1
6

ti

2 ti
e

= 0.9998

i =1

The exponential regression model then is

= 0.9998 e
15

0.1151t

Plot of data and regression curve

= 0.9998 e 0.1151t

16

Relative Intensity After 24 hrs

The relative intensity of radiation after 24 hours

= 0.9998 e

0.1151( 24 )

= 6.3160 10

This result implies that only

6.316 102
100 = 6.317%
0.9998
radioactive intensity is left after 24 hours.
17

Homework

18

What is the half-life of technetium 99m

isotope?
Compare the constants of this regression
model
d l with
i h the
h one where
h
the
h data
d
is
i
transformed.
Write a program in the language of your
choice to find the constants of the model
(fortran, matlab, matchad, maple)

P l
Polynomial
i l Model
M d l
Given

( x1, y1), ( x 2, y 2), ... , ( xn, yn)

best fit y = a0 + a1 x + ... + am x

( m n 2) to a given data set.

( xn , yn )

(x , y )
2

( xi , yi )

(x1, y1)

y = a + a x + K+ a x m
0

y f (x )
i

Figure. Polynomial model for nonlinear regression of y vs. x data

19

Polynomial Model cont.

The residual at each data point is given by

E i = y i a 0 a1 xi . . . a m xim
Th sum off th
The
the square off th
the residuals
id l then
th is
i
n

S r = Ei2
i =1

= y i a 0 a1 xi . . . a m xim
i =1

20

Polynomial Model cont.

To find the constants of the polynomial model, we set the derivatives
with respect to ai where i = 1,K m, equal to zero.
n
S r
= 2. yi a0 a1 xi . . . am xim (1) = 0
a0 i =1

n
S r
= 2. yi a0 a1 xi . . . am xim ( xi ) = 0
a1 i =1

n
S r
= 2. yi a0 a1 xi . . . am xim ( xim ) = 0
am i =1

21

Polynomial Model cont.

These equations in matrix form are given by

n
xi
i =1
. . .

n m
xi
i =1

xi
i =1
n 2
xi
i =1
. . .

. .
. .
. .

n m+1
xi . .
i =1

n m
. xi a
i =1 0

n m+1 a1
. xi
i =1
. .
. . . a
m
n

. xi2 m
i =1

yi
ni =1
=
xi yi

. i =1
. . .

n
xim yi
i =1

The above equations are then solved for a0 , a1 , K , am

22

Example 2-Polynomial Model

Regress the thermal expansion coefficient vs. temperature data to
a second order polynomial.

23

Coefficient of
thermal
expansion,
(i /i /oF))
(in/in/

80

6.47106

40

6.24106

40

5.72106

120

5.09106

200

4.30106

280

3.33106

340

2.451066

6.00E-06
5.00E-06
(in/in/oF)

Temperature, T
(oF)

7.00E-06
Therm
mal expansion c
coefficient,

Table. Data points for

temperature vs

4.00E-06
3.00E-06
2.00E-06

-400

-300

-200

1.00E-06
-100
0

100

200

Temperature oF
Temperature,

Figure. Data points for thermal expansion coefficient vs

temperature.

Example 2-Polynomial Model cont.

We are to fit the data to the polynomial regression model
= a0 + a1T + a 2 T 2

a1 , a2 are found by differentiating the sum of the

The coefficients a0 ,a
square of the residuals with respect to each variable and setting the
values equal to zero to obtain

n
T

i
i =n1
T 2
i

i =1

24

Ti
i =1
n 2
Ti
i =1
n 3
Ti
i =1

n 2
n
Ti
i

i =1 a i =1
0
n 3 n
Ti a1 = Ti i
i =1 i =1
a
n
n 4 2 T 2
Ti

i
i

i
=
1

i =1

Example 2-Polynomial Model cont.

The necessary summations are as follows
Table. Data points for temperature vs.
Temperature,
T
t
T
(oF)

C ffi i t off
Coefficient
thermal expansion,
(in/in/oF)

80

6.47106

40

6.2410
6.24
106

40

5.72106

120

5.09106

200

4.30106

280

3.33106

340

2.45106

T
i =1
7

T
i =1

= 7.0472 10 7

= 2.1363 1010

i =1

= 3.3600 10 5

T
i =1

T
i =1

25

= 2.5580 105

i =1

= 2.6978 10 3

i =8.5013 10 1

Example 2-Polynomial Model cont.

Using these summations, we can now calculate a0 ,a1 , a2
7.0000

2
8.600 10
2.5800 10 5

8.6000 10 2
2.5800 10 5
7.0472 10 7

2.5800 10 5 a 0 3.3600 10 5

7.0472 10 7 a1 = 2.6978 10 3
2.1363 1010 a 2 8.5013 10 1

Solving the above system of simultaneous linear equations we have

6
a 0 6.0217 10
a = 6.2782 10 9

1
11

a 2 1.2218 10

The polynomial regression model is then

= a 0 + a1T + a 2T 2
= 6.0217 10 6 + 6.2782 10 9 T 1.2218 10 11 T 2
26

Linearization of Data
To find the constants of many nonlinear models, it results in solving
simultaneous nonlinear equations. For mathematical convenience,
some of the data for such models can be linearized. For example,
p , the
data for an exponential model can be linearized.
As shown in the previous example, many chemical and physical processes
are governed by the equation,

y = aebx
Taking the natural log of both sides yields,

ln y = ln a + bx
Let z = ln y and a 0 = ln a
We now have a linear regression model where z = a 0 + a1 x
(implying) a = e ao with
27

a1 = b

Linearization of data cont.

Using linear model regression methods,

a1 =

i =1

i =1

i =1

n xi z i xi z i

n xi2 xi
i =1
i =1
n

a 0 = z a1 x
Once ao , a1 are found, the original constants of the model are found as
b = a1

a = e a0

28

Example 3-Linearization of data

Many patients get concerned when a test involves injection of a radioactive
material. For example for scanning a gallbladder, a few drops of Technetium99m isotope is used. Half of the technetium-99m would be gone in about 6
hours. It, however, takes about 24 hours for the radiation levels to reach what
we are exposed to in day-to-day activities. Below is given the relative intensity
of radiation as a function of time.

t(hrs)

1.000

0.891

0.708

0.562

0.447

0.355

1
Relative inte
ensity of radiation,,

Table. Relative intensity of radiation as a function

of time

0.5

0
0

5
Time t, (hours)

10

Figure. Data points of relative radiation intensity

vs. time
29

Example 3-Linearization of data cont.

Find:
a) The value of the regression constants A and
b) The
Th half-life
h lf lif off Technium-99m
T h i
99
c) Radiation intensity after 24 hours
The relative intensity is related to time by the equation

= Ae t

30

Example 3-Linearization of data cont.

Exponential model given as,

= Ae t
l ( ) = ln
ln
l ( A) + t
Assuming z = ln , a o = ln ( A) and a1 = we obtain
z =a +at
0
1
z and
This
h is a linear
l
relationship
l
h between
b
d t

31

Example 3-Linearization of data cont.

Using this linear relationship, we can calculate a0 , a1 where

a =

i =1

i =1

i =1

n ti zi ti zi

and

n t12 ti
i =1
i =1
n

a0 = z a1t
= a1
A=e

32

a0

Example 3-Linearization of Data cont.

Summations for data linearization are as follows
Table. Summation data for linearization of data model

ti

i
1
2
3
4
5
6

0
1
3
5
7
9

25.000

z = ln

1
0 891
0.891
0.708
0.562
0.447
0.355

0.00000
00.11541
11541
0.34531
0.57625
0.80520
1.0356

0.0000
0
0.11541
11541
1.0359
2.8813
5.6364
9.3207

0.0000
1 0000
1.0000
9.0000
25.000
49.000
81.000

2.8778

18.990

165.00

tz

i i

2
i

With n = 6
6

t
i =1
6

z
i =1

= 2.8778

t z
i =1
6

t
i =1

33

= 25.000

i i

2
i

= 18.990

= 165.00

Example 3-Linearization of Data cont.

Calculating a0 , a1
a1 =

6( 18.990 ) (25)( 2.8778)

2
6(165.00 ) (25)

= 0.11505

a0 =

2.8778
25
( 0.11505 )
6
6

= 2.6150 10 4

Since
a0 = ln ( A)

A = e a0

=e

2.6150104

= 0.99974

also

= a1 = 0.11505
34

Example 3-Linearization of Data cont.

0 . 11505 t
Resulting model is = 0 .99974 e

= 0.99974 e 0.11505t
Relative
Intensity
0.5
off
Radiation,

0
0

10

Time, t (hrs)
Figure. Relative intensity of radiation as a function of
temperature using linearization of data model.
35

Example 3-Linearization of Data cont.

The regression formula is then
= 0 .99974 e 0.11505 t
1

b) Half life of Technetium 99 is when

2
1
0 .99974 e 0 .11505 t = (0 .99974 )e 0 .11505 (0 )
2
e 0 .11508 t = 0 .5
0 .11505 t = ln (0 .5 )
t = 6 .0248 hours

36

t =0

Example 3-Linearization of Data cont.

c) The relative intensity of radiation after 24 hours is then
= 0.99974e 0.11505(24 )

= 0.063200

6.3200 10 2
100 = 6.3216%
This implies that only
0.99983

material is left after 24 hours.

37

of the radioactive

Comparison
Comparison of exponential model with and without data linearization:
Table. Comparison for exponential model with and without data
linearization.

With data linearization

(Example 3)

Without data linearization

(Example 1)

0.99974

0.99983

0.11505
0 11505

0.11508
0 11508

Half-Life (hrs)

6.0248

6.0232

Relative intensity
after 24 hrs.

6.3200102

6.3160102

The values are very similar so data linearization was suitable to

find the constants of the nonlinear exponential model in this
case.
case
38