Cs1100Lec27 30 Numeric

8/2/2019 Cs1100Lec27 30 Numeric

1/67

CS-110Computational Engineering

Numerical Methods

byA. Avudainayagam

Department of Mathematics

revised byTimothy Gonsalves Dept of Computer Science & Engg

8/2/2019 Cs1100Lec27 30 Numeric

2/67

Lecture 27: Root Finding

Tutors:

Shailesh Vaya Anurag Mittal [email protected] [email protected]

8/2/2019 Cs1100Lec27 30 Numeric

3/67

Numerical Methods

Used for: Solution of algebraic equations Approximation of functions Differentiation and integration of functions Solution of differential equations

Statistical analysis of data

8/2/2019 Cs1100Lec27 30 Numeric

4/67

Numerical Methods

Used because: Analytical solution extremely difficult for acomplex function

Analytical solution may require evaluation of esoteric functions

Mathematical functions may not be analytical Function may be in the form of pairs of data

Given experimental data for pillar design:

Pillar radius (m) 1.2 1.5 1.8 2.0 3.0

Max Load (tons) 10.3 15.6 20.3 28.7 43.5

What is pillar radius for 35 tons?

8/2/2019 Cs1100Lec27 30 Numeric

5/67

Numerical Errors

Source of Errors Approximate evaluation of functions = 22/7 = 3.14285714

Representation of numbers in a finite number of bits

Round-off error, eg correct to 3 decimals: 2.000 + 0.77 10 -6 = 2.00000077 = 2.000

Reduction of Error Iterative solutions -- repeat until error <

Efficiency of convergence

Stability --- may never converge

8/2/2019 Cs1100Lec27 30 Numeric

6/67

Fundamental Motifs

Several techniques for a given problem Technique of choice depends on nature of

the data One technique with modifications may be

used to solve several different problems Error analysis essential to determine

reliability of the computed results

8/2/2019 Cs1100Lec27 30 Numeric

7/67

Root Finding: f(x)=0

Method 1: The Bisection methodTh: If f(x) is continuous in [a,b] and if f(a)f(b) 0

Multiple roots in (a,b)

ax1

x2x3

x4 x5 b

y

x

y

x

8/2/2019 Cs1100Lec27 30 Numeric

8/67

The Method Find an interval [x 0, x1] such that f(x 0)f(x 1) < 0

This may not be easy. Use your knowledge of the physical phenomenon which the equation represents.

In each iteration, cut the interval into half Examine the sign of the function at the mid point

If f(m) = 0, x is the root If f(m) 0 and f(x 0)f(x) < 0, root lies in [x 0, m]

Otherwise root lies in [m, x 1] Repeat the process until convergence (length of interval < )

m = x0 + x12

8/2/2019 Cs1100Lec27 30 Numeric

9/67

Number of Iterations and Error Tolerance

Length of the interval (where the root lies) after n iterations

n01

n2

xxe

=

We can fix the number of iterations so that the root lies withinan interval of chosen length (error tolerance).

ne n ln( x1 x0) ln ln2 If n satisfies this, root lies within a distance of /2 of the

actual root

8/2/2019 Cs1100Lec27 30 Numeric

10/67

Though the root lies in a small interval, |f(x)| may not besmall if f(x) has a large slope.

Conversely if |f(x)| small, x may not be close to the root if

f(x) has a small slope.

So, we use both these facts for the termination criterion.We first choose an error tolerance on f(x): |f(x)| < and K the maximum number of iterations.

8/2/2019 Cs1100Lec27 30 Numeric

11/67

Pseudo code (Bisection Method)1. Input > 0 , K > 0, x 1 > x 0 so that f(x 0) f(x 1) < 0.

Compute f 0 = f(x 0) .k = 1 (iteration count)

2. Do{

(a) Compute m = , and f = f(m)

(b) If f f 0 < 0, set x 1= m otherwise set x 0 = m

(c) Set k = k+1

} while |f| > and k K 3. Set root = m

+

210 x x

8/2/2019 Cs1100Lec27 30 Numeric

12/67

Bisection Method Examplef(x) = x 3-2 = 0, = 10 -4x0 = 1, x 1 = 2

k x 0 x1 m f(m)

1 1 2 1.5 1.375

2 1 1.5 1.25 -0.4688

3 1.25 1.5 1.375 0.5996

4 1.25 1.375 1.3125 0.2610

5 1.25 1.3125 1.2813 0.1033

After 13 iterations, m = 1.2599 (to 4 decimal places)

Using , n 12.29n ln( x1 x0) ln

ln2

8/2/2019 Cs1100Lec27 30 Numeric

13/67

False Position Method (Regula Falsi)

Root may lie near end of interval with smaller value

of |f|

Instead of bisecting the interval [x 0,x1], we choose

the point where the straight line through the end

points meet the x-axis as w and bracket the root with

[x0, w] or [w, x 1] depending on sign of f(w)

8/2/2019 Cs1100Lec27 30 Numeric

14/67

False Position Method

(x1,f 1)

x0

(x0,f 0)

w x 1

(w,f 2)

y=f(x)

Straight line through (x 0,f 0) ,(x 1,f 1) :

)( 10101

1 x x x x

f f

f y

+=

New end point w:

y

x

w = x1 x1 x0

f 1 f 0

f 1

8/2/2019 Cs1100Lec27 30 Numeric

15/67

2. do {a. Compute , and f = f(w)

b. If f 0 f < 0 set x 1 = w, f 1 = f

else set x 0 = w, f 0 = f

c. k = k+1

} while (|f| ) and (k K)

False Position Method (Pseudo Code)

w = x1 x1 x0 f 1

f 0

f 1

1. Choose > 0 (tolerance on |f(x)| )K > 0 (maximum number of iterations )k = 1 (iteration count)x0 ,x1 (so that f 0 ,f 1 < 0)

4. x = w, the root

8/2/2019 Cs1100Lec27 30 Numeric

16/67

Regula Falsi Examplef(x) = x 3-2 = 0, = 10 -4x0 = 1, x 1 = 2

k x 0 x1 f(x 0) f(x 1) w f(w)

0 1.0 2.0 -1.0 6.0 1.1429 -0.5071

1 1.1429 2.0 -0.5071 6.0 1.2097 -0.22982 1.2097 2.0 -0.2298 6.0 1.2389 -0.0987

3 1.2389 2.0 -0.0987 6.0 1.2512 -0.0412

4 1.2512 2.0 -0.0412 6.0 1.2563 -0.0172

9 1.2598 2.0 -0.0003 6.0 1.2607 0.0039

10 1.2598 1.2607 -0.0003 0.0039 1.2599 -0.0003

11 1.2599 1.2607 -0.0003 0.0039 1.2600 0.0002

8/2/2019 Cs1100Lec27 30 Numeric

17/67

Newton-Raphson or Newtons Method

At an approximation xk

to the root, the curve is approximated by thetangent to the curve at x k and the next approximation x k+1 is the

point where the tangent meets the x-axis.

y = f(x)

root

x0

xx

1x

2

y

8/2/2019 Cs1100Lec27 30 Numeric

18/67

)()(

1k

k k k x f

x f x x =+

Warning : If f(x k ) is very small, method fails.

Two function Evaluations per iteration

This tangent cuts the x-axis at x k+1

Tangent at (x k , f k ) :y = f(x k ) + f (x k )(x-x k )

8/2/2019 Cs1100Lec27 30 Numeric

19/67

Newtons Method - Pseudo code

4. x = x 1 the root.

1. Choose > 0 (function tolerance |f(x)| < )m > 0 (Maximum number of iterations)x0 - initial approximationk - iteration count

Compute f(x 0)2. Do { q = f (x0) (evaluate derivative at x 0)

x1= x 0 - f 0/qx0 = x 1

f 0 = f(x 0)k = k+1

}3. While (|f 0| ) and (k m)

8/2/2019 Cs1100Lec27 30 Numeric

20/67

Getting caught in a cycle of Newtons Method

Alternate iterations fall at the same point .

No Convergence.

xk+1xk x

y

8/2/2019 Cs1100Lec27 30 Numeric

21/67

Newtons Method for finding the squareroot of a number x = a

k

k k k x

a x x x

2

22

1=+

Example : a = 5 , initial approximation x 0 = 2.

x1 = 2.25

x2 = 2.236111111

x3 = 2.236067978

x4 = 2.236067978

f(x) = x 2 - a2 = 0

8/2/2019 Cs1100Lec27 30 Numeric

22/67

CS110 Lecture 28:

Root Finding Secant MethodTutors:


8/2/2019 Cs1100Lec27 30 Numeric

23/67

Problems with Newtons Method

If |f (x)| is very small, accuracy is difficultto obtain

Depending on the initial estimate, any oneof the roots may be found (answer may nothave physical significance)

Use bisection to get close to desired root, then Newtons method for fast convergence

May get caught in an infinite cycle

8/2/2019 Cs1100Lec27 30 Numeric

24/67

The secant Method

Newtons Method requires 2 function evaluations (f, f ).

The Secant Method requires only 1 function evaluation andconverges as fast as Newtons Method at a simple root.

Start with two points x 0,x1 near the root (no need for bracketing the root as in Bisection Method or Regula FalsiMethod) .

xk-1 is dropped once x k+1 is obtained.

h h d

8/2/2019 Cs1100Lec27 30 Numeric

25/67

The Secant Method(Geometrical Construction)

Two initial points x 0, x1 are chosen

The next approximation x 2 is the point where the straight line joining (x 0,f 0) and (x 1,f 1) meet the x-axis

Take (x 1,x2) and repeat.

y = f(x 1)

x2 x0 x1

(x0,f 0)(x1,f 1)

x

y

8/2/2019 Cs1100Lec27 30 Numeric

26/67

The secant Method (Pseudo Code)1. Choose > 0 (function tolerance |f(x)| )

m > 0 (Maximum number of iterations)

x0 , x1 (Two initial points near the root )f 0 = f(x 0)f 1 = f(x 1)k = 1 (iteration count)

2. Do {

x0 = x 1 f 0 = f 1

x1= x 2f 1 = f(x 2)k = k+1 }

3. while (|f 1| ) and (m k)

x2 = x1 x1 x0 f 1 f 0

f 1

8/2/2019 Cs1100Lec27 30 Numeric

27/67

On Convergence

# The false position method in general converges faster than

the bisection method.# But not always, shown by counter examples

# The bisection method and the false position method areguaranteed to converge

# The secant method and the Newton-Raphson method arenot guaranteed to converge

8/2/2019 Cs1100Lec27 30 Numeric

28/67

Order of Convergence

# A measure of how fast an algorithm converges

Let be the actual root: f( ) = 0

Let x k be the approximate root at the kth iteration . Error at the kth iteration, e k = |x k - |

The algorithm converges with order p if there exists

such that pk 1k ee =+

8/2/2019 Cs1100Lec27 30 Numeric

29/67

Order of Convergence of

# Bisection method p = 1 (linear convergence) # False position - generally super linear (1 < p < 2)

# Secant method (super linear)618.12

51=

+

# Newton Raphson method p = 2 quadratic

8/2/2019 Cs1100Lec27 30 Numeric

30/67

Machine Precision

# The smallest positive float M that can be added toone and produce a sum that is greater than one.

8/2/2019 Cs1100Lec27 30 Numeric

31/67

Pseudo code to find Machine Epsilon

1. Set M = 1

2. Do

{

M =M / 2

x = 1 + M

}

3. While ( x > 1 )

4. M = 2 M

8/2/2019 Cs1100Lec27 30 Numeric

32/67

CS110 Lecture 29:

Approximation & InterpolationTutors:


8/2/2019 Cs1100Lec27 30 Numeric

33/67

Approximation & Interpolation

Reasons to approximate value of a function: Difficult or impossible to evaluate the function

analytically, eg. sine, log, etc. Have only a table of values and must interpolate Faster to compute approx function than original Function defined implicitly rather than by an

equation

8/2/2019 Cs1100Lec27 30 Numeric

34/67

Interpolation

P1

P2P3

P4y

x

Given the data:

(xk , yk ) , k = 1, 2, 3, ... , n,

find a function f which we can use to predict the value of y at pointsother than the samples

1. f(x) may pass through all the data points:

f(xk ) = y k , 1 k n

2. f(x) need not pass through any of the data points:

Need to control error |f(x k ) y k |

8/2/2019 Cs1100Lec27 30 Numeric

35/67

1. Piecewise Linear InterpolationA straight line segment is used between each adjacent pair of

data points

x1 x2 x3 x4 x5

( ) ( )k k k k

k k k y y x x

x x y x f

+= ++

11

1 k n

Simple and computationally efficient

l l l

8/2/2019 Cs1100Lec27 30 Numeric

36/67

1. Polynomial Interpolation

For the data set (x k , yk ), k = 1, ... , n,

we find the one polynomial of degree (n - 1) subject to the ninterpolation constraints f(x k ) = y k

( ) =

=n

1k

1k k xaxf

=

n

2

1

n

2

1

1nn

2nn

1n2

222

1n1

211

y

yy

a

aa

xxx1

xxx1xxx1

Not feasible for large data sets, since the condition number increases rapidly with increasing n.

8/2/2019 Cs1100Lec27 30 Numeric

37/67

Lagrange Interpolating PolynomialThe Lagrange interpolating polynomial of degree k , f ( x) is

constructed as follows:1. Caculate the Lagrangian multipliers Qk ( x) each of which is a

polynomial of degree n-1 that is non-zero at only the one base point xk

Normalise by Qk ( xk )

Qk x( ) =x x1( ) ( x x2)...( x xk 1)( x xk +1)...( x xn)

( xk x1)( xk x3)...( xk xk 1)( xk xk +1)( xk xn )

Qk ( x) = ( x xi )i= 0, i k

n

/ ( xk xi )i= 0, i k

n

8/2/2019 Cs1100Lec27 30 Numeric

38/67

Each Qk ( x) is a polynomial of degree ( n - 1)

Qk ( x j) = 1 , j = k

= 0 , j k

The polynomial curve that passes through the data set ( xk , yk ),k = 1, 2, ..., n is

f ( x) = y1Q1( x) + y2Q2( x) + ... + yn Qn( x)

Polynomial is written directly without having to solve asystem of equations

L i l i (P d C d )

8/2/2019 Cs1100Lec27 30 Numeric

39/67

Lagrange interpolations (Pseudo Code)Choose x, the point where the function value is required

y = 0for i = 1 to n

p = y 1

for j = 1 to n

if ( i j )

p = p * (x - x j) / (x i - x j)

end for

y = y + product

End for

8/2/2019 Cs1100Lec27 30 Numeric

40/67

Lagrange Interpolation Example

Given the following base points, estimate sin 23 tofive decimal places:

i xi yi

0 20 0.34202

1 22 0.37641

2 24 0.40674

3 26 0.43837Qk ( x) = ( x xi )i= 0, i k

n

/ ( xk xi )i= 0, i k

n

Q0( x) = -0.0625

Q1( x) = -0.5625

Q2( x) = -0.5625

Q3( x) = -0.0625

f ( x) = y1Q1( x) + y2Q2( x) + ... + yn Qn( x)

= (0.34202)(-0.0625) +

(0.37461)(0.5625) +(0.40674)(0.5625) +(0.43837)(-0.0625)

= 0.39074True value: 0.39073, discrepancy due to

accumulation of round-off error.

8/2/2019 Cs1100Lec27 30 Numeric

41/67

CS110 Lecture 30:

Curve FittingTutors:


8/2/2019 Cs1100Lec27 30 Numeric

42/67

2. Least Squares FitIf the number of samples is large or if the dependantvariable contains measurement noise, it is often better tofind a function which minimizes an error criterion suchas

A function that minimizes E is called the Least Squares Fit

Depending on the nature of the function we have:linear regression

polynomial regression (quadratic, cubic, )exponential regression, etc.

E = f ( xk ) yk [ ]2

k = 1

n

8/2/2019 Cs1100Lec27 30 Numeric

43/67

2. Minimax

Least squares approximation gives good fit overall, butmay have large deviation from one point

Minimize the maximum deviation from y k

Also known as Chebyshev or optimal polynomial approximation

|)(|max k k k

y x f E =

8/2/2019 Cs1100Lec27 30 Numeric

44/67

Straight Line Fit or Linear Regression To fit a straight line through the n points (x 1, y1), (x 2, y2), ... ,(x n, yn)

Assume f(x) = a 1 + a 2x

Error [ ]2n

1k k k y)x(f E

=

=

[ ]

2n

1k k k 21 yxaa= +=

Find a 1, a2 which minimize E

0x)yxaa(2aE n

1k k k k 21

2

=+/=

=

E a1 =

/2 ( a1 + a 2 xk yk )k =1

n

= 0

8/2/2019 Cs1100Lec27 30 Numeric

45/67

=

k k

k

k k

k

y x

y

a

a

x x

xn

2

1

2

0x)yxaa(2aE n

1k k k k 21

2

=+/=

=

E a1

= /2 ( a1 + a 2 xk yk )

k =1

n

= 0

Solve:

St i ht Li Fit ( l )

8/2/2019 Cs1100Lec27 30 Numeric

46/67

Straight Line Fit (example)Fit a straight line through the five points

(0, 2.10), (1, 2.85), (2, 1.10), (3, 3.20), (4, 3.90)a11 = n = 5

=+++== 3016941xa 2k 22

043210xa k 12 =++++==

=+++== 15.1320.310.185.210.2y b k 1

a21 = a 12

=+++== 25.30)90.3(4)20.3(3)10.1(285.2yx b k k 2=

25.30

15.13

a

a

3010

105

2

1

a1 = 1.84, a 2 = 0.395, f(x) = 1.84 + 0.395x

8/2/2019 Cs1100Lec27 30 Numeric

47/67

Straight Line Fit (example)

0

0.5

1

1.5

2

2.5

3

3.54

4.5

0 1 2 3 4

Data

Linear Fit

Data points: (0, 2.10), (1, 2.85), (2, 1.10), (3, 3.20), (4, 3.90)

Linear fit: f( x) = 1.84 + 0.395 x

8/2/2019 Cs1100Lec27 30 Numeric

48/67

Two Kinds of Curve Fitting

8/2/2019 Cs1100Lec27 30 Numeric

49/67

Data Representation

Integers - Fixed Point Numbers

Decimal System - base 10 uses 0,1,2,,9

(396) 10 = (6 100) + (9 101) + (3 102) = (396) 10

Binary System - base 2 uses 0,1(11001) 2 = (1 20) + (0 21) + (0 22) + (1 23) + (1 24) = (25) 10

Decimal to Binary Conversion

8/2/2019 Cs1100Lec27 30 Numeric

50/67

Decimal to Binary ConversionConvert (39) 10 to binary form

base = 2392

2 19 + Remainder 12 9 + Remainder 1

2 4 + Remainder 12 2 + Remainder 0

2 1 + Remainder 0

0 + Remainder 1

Put the remainder in reverse order

(100111) 2 = (1 20 ) + (1 21) + (1 22) + (0 23) + (0 24)

+ (1 25

) = (39) 10

8/2/2019 Cs1100Lec27 30 Numeric

51/67

Largest number that can be stored in m-digits

base - 10 : (999999) = 10m

- 1 base - 2 : (111111) = 2 m - 1

m = 3 (999) = 10 3 - 1(111) = 2 3 - 1

Limitation: Memory cells consist of 8 bits (1 byte) multiples, each position containing 1 binary digit

8/2/2019 Cs1100Lec27 30 Numeric

52/67

Common cell lengths for integers : k = 16 bits

k = 32 bits

Sign - Magnitude NotationFirst bit is used for a sign

0 - non negative number 1 - negative number

The remaining bits are used to store the binary magnitudeof the number.

Limit of 16 bit cell : (32767) 10

Limit of 32 bit cell : (2 147 483 647) 10

8/2/2019 Cs1100Lec27 30 Numeric

53/67

Twos Complement notation

Definition : The twos complement of a negative integer I in a

k - bit cell :

Twos Complement of I = 2 k + I

(Eg) : Twos Complement of (-3) 10 in a 3 - bit cell

= (23

- 3) 10 = (5) 10 = (101) 2 (-3) 10 will be stored as 101

8/2/2019 Cs1100Lec27 30 Numeric

54/67

Storage Scheme for storing an integer I in a k - bit cell inTwos Complement notation

Stored Value C =

I , I 0 , first bit = 0

2k + I , I < 0

8/2/2019 Cs1100Lec27 30 Numeric

55/67

The Twos Complement notation

admits one more negative number than the sign - magnitude notation.

8/2/2019 Cs1100Lec27 30 Numeric

56/67

(Eg) Take a 2 bit cell (k = 2)

Range in Sign - magnitude notation : 2 1 - 1 = 1

-1 = 11

1 = 01

Range in Twos Compliment notation

Twos Compliment of -1 = 2 2-1 = (3) 10 = (11) 2

Twos Compliment of -2 = 2 2- 2 = (2) 10 = (10) 2

Twos Compliment of -3 = 2 2 - 2 = 0 - Not possible

8/2/2019 Cs1100Lec27 30 Numeric

57/67

Floating Point Numbers

Integer Part + Fractional PartDecimal System - base 10

235 . 7846

Binary System - base 210011 . 11101

410

6

310

4

210

8

10

7+++=Fractional Part (0.7846) 10

Fractional Part (0.11101) 2 52

1

42

0

32

1

22

1

2

1+++=

8/2/2019 Cs1100Lec27 30 Numeric

58/67

Binary Fraction Decimal Fraction (10.11)

Integer Part (10) 2 = 0.2 0 + 1.2 1 = 2

Fractional Part (11) 2 = 75.025.05.022

12

1=+=+

Decimal Fraction = ( 2.75 ) 10

8/2/2019 Cs1100Lec27 30 Numeric

59/67

Decimal Fraction Binary FractionConvert (0.9) 10 to binary fraction

0.9 20.8 + integer part 1

20.6 + integer part 1 20.2 + integer part 1 20.4 + integer part 0 20.8 + integer part 0 Repetition

(0.9)10

= (0.11100)2

8/2/2019 Cs1100Lec27 30 Numeric

60/67

Scientific Notation (Decimal)

0.0000747 = 7.47 10 -5

31.4159265 = 3.14159265 10

9,700,000,000 = 9.7 109

Binary

(10.01) 2 = (1.001) 2 21

(0.110) 2 = (1.10) 2 2-1

8/2/2019 Cs1100Lec27 30 Numeric

61/67

Computer stores a binary approximation to x

x q 2n

q - mantissa , n exponent

(-39.9) 10 = (-100111.1 1100) 2

= (-1.0001111 1100) 2 (25

)10

8/2/2019 Cs1100Lec27 30 Numeric

62/67

Decimal Value of stored number (-39.9) 10

= -1. 001111 1100 1100 1100 11001

23 bit

32 bits : First bit for sign

Next 8 bits for exponent

23 bits for mantissa

= -39. 900001525 878 906 25

8/2/2019 Cs1100Lec27 30 Numeric

63/67

Round off Errors can be reduced byEfficient Programming Practice

# The number of operations (multiplications andadditions ) must be kept minimum. (Complexity theory)

8/2/2019 Cs1100Lec27 30 Numeric

64/67

An Example of Efficient Programming

Problem : Evaluate the value of the Polynomial.

P(x) = a 4x4 + a 3x3 + a 2x2 + a 1x + a 0

at a given x.

Requires 13 mulitiplications and 4 additions.

Bad Programme!

8/2/2019 Cs1100Lec27 30 Numeric

65/67

An Efficient method (Horners method)

P(x) = a 4x4 + a 3x3 + a 2x2 + a 1x + a 0

= a 0 + x(a 1 + x(a 2 + x(a 3 + xa 4) ))

Requires 4 multiplications and 4 additions.

Pseudo-code for an n th degree polynomial

Input as, n, x

p = 0for i = n, n-1, , 0{

p = x p + a[i]}

8/2/2019 Cs1100Lec27 30 Numeric

66/67

Summary

Finding the root(s) of an equation Bisection, regula falsi, Newtons, secant Fitting curves to data:

Exact: Lagrange interpolation

Least squares fit: linear regression (also polynomial,exponential, etc)

OpenOffice functions: LINEST, LOGEST, TREND

Several techniques for a given problem Technique of choice depends on nature of the data Error analysis essential to determine reliability of

the computed results

8/2/2019 Cs1100Lec27 30 Numeric

67/67

References

M.C. Kohn, Practical Numerical Methods: Algorithms and Programs , Macmillan, 1987

R. Bhat & S. Chakraverty, Numerical Analysis in Engineering , Narosa, 2004M.K. Jain, S.R.K. Iyengar & R.K. Jain,

Numerical Methods for Scientific and Engineering Computation , 3 rd ed., WileyEastern, 1994

Cs1100Lec27 30 Numeric

Documents

Transcript of Cs1100Lec27 30 Numeric