Giansalvo EXIN CirrincioneGiansalvo EXIN Cirrincione

unité #5unité #5

Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))

021 n valeurs singulièresvaleurs singulières

aam1m1

aamnmn

aa1111 aa1n1n

UU

00

00

00 nn

11 00

VV

=

Full SVDFull SVDHA U V

Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))

aam1m1

aamnmn

aa1111 aa1n1n

UU

00

00

00 nn

11 00

VV

=

Reduced SVDReduced SVDˆ ˆ HA U V aam1m1

aamnmn

aa1111 aa1n1n

UU 00 nn

11 00

VV

= ^̂

Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))

Full SVDFull SVD

unitaire

unitaire

diagona le

m m

n n

m n

H

U

A U V

V

Reduced SVDReduced SVD

colonnes orthonormales

un

ˆ

ˆ

itaire

diagon

ˆ

al

ˆ

e

m n

n n

m

H

n

A U

V

V

U

ˆ ˆˆˆ ˆˆ ˆ ˆ ˆH

HH H 2H HH HA U V U U U V VVA V V

ˆ ˆˆ ˆ ˆ

ˆ ˆ ˆˆ ˆ ˆˆ

H11H H H 2 H H

H H H2 1 H

A A A A

V V

U V V V U

V

P

UU

V

UVU

ˆ ˆ HP UU

Approximation au sens des moindres carrées

x

2

Given , , ,

find such that is minimized.

m n m

n

A m n b

x b Ax

Example: polynomial data fittingExample: polynomial data fitting

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8-4

-3

-2

-1

0

1

2

f(x)

xi

yi

1,données : ,i i i m

x y

1

1

2

1

1

1

approximation

mi

( )

équations et inconnues ( ) approximation au s

interpolation

( )

ens

n (

) m

i ii

nj

nj

j

i i j i i

j

j

k

n m

f x y

f x x

n n

n m

f x y x y

des moindres carrés

Approximation au sens des moindres carrées

discrete square wavediscrete square wave

interpolationinterpolation

m m = = n n = = 1111

least squaresleast squares

m m = = 11 11 , , nn = = 88

Approximation au sens des moindres carrées

Posons le problème matriciellement

(1) ( 1) ( 1)1 2 1 1 1 1

(1) ( 1) ( 1)1 2 2 2 2 2

(1) ( 1) ( 1)1 2

1 2

... ...

... ...

... ...

j nj n

j nj n

j ni j i n i i

x x x f x

x x x f x

x x x f x

x

(1) ( 1) ( 1)... ...j nm j m n m mx x f x

1

1

1,

( )

pour ,

kj

jj

i i i m

f x x

x y

(1) ( 1) ( 1)11 1 1

(1) ( 1) ( 1)22 2 2

(1) ( 1) ( 1)

(1) ( 1) ( 1)

(1)

1

1) )

2

( ( 1

1 ... ...

1 ... ...

1 ... ...

1 ... ...

1

j n

j n

j nii i i

j nn n n

j nm m m

i

n

f xx x x

f xx x x

f xx x x

f xx x x

x x x

a

n

mf x

X f

Approximation au sens des moindres carrées

2

1

1

2

1

1 1

1

1 1

1 1 1

* argmin ( )

min ( ) avec ( )

principe : 1,...,

2 0

( *)

n

0

mi ( )m

i ii

mj

i ii

m nj

j i ii j

m

j

nj

i i iij

J J x yf x y

x xx y x

JnJ j

J

1

1 1

n mj

i ii

y x

système linéaire de n équations et n inconnues

11 1 1

1

1

0

1

1

1

( )

1

1

kj

i i i j i ij

n

ni i i

nn n n

nm m m

e x x

e x x

e x x

e x x

e f x y x y

e X y

1

1

2

2 2

( ) 2

( )

( ) 0

i

nn

T

m

T T

y

y

y

X X X

y

J e X y

J X X y yJ

Approximation au sens des moindres carrées

erreur erreur d’approximationd’approximation

Matrice de Vandermonde (1735-1796)

Approximation au sens des moindres carrées

2 2

1min

( )

2

min ( )

'( ) 2 '( ) 0

22

T

T T T T T

T

T T

J e X y

X y X y

X X X y y y

J

J X X y J

G h

G h

T TX X X y

Équations normalesÉquations normales

forme quadratiqueforme quadratique

x

2

Given , , ,

find such that is minimi

zed.

m n m

n

A m n

Ax b

b

x b Ax

find the closest point in range to so that

the norm of the residual (residu) is minimized

mAx A b

r b Ax

x

morthogonal projector that maps onto range

m m

A

Ax Pb P

range(range(AA)) range nA

x

morthogonal projector that maps onto range

m m

A

Ax Pb P

y = A xy = A x

r = b - A xb

b m

ranger A

1

1 1

H H

H H H H

P A A A A

Pb A A A A b A A A A Ax Ax

Pb Ax

0

normal equations

H

H H

A r

A Ax A b

The system is nonsingular iff A has full rank.

= Pb= Pb

x

morthogonal projector that maps onto range

m m

A

Ax Pb P

1

1 1

H H

H H H H

P A A A A

Pb A A A A b A A A A Ax Ax

Pb Ax

0

normal equations

H

H H

A r

A Ax A b

The system is nonsingular iff A has full rank.

1

x1

pseudoinverse of

H H

H H

n m

x A bA A A b

A A A A

A

Solution par les équations normales

factorisation de Cholesky

AHA est une matrice n x n hermitienne strictement définie positive

1. Form the matrix AAHHAA and the vector AAH H bb2. Compute the Cholesky factorization AAHHAA = = RRHHRR3. Solve the lower-triangular system RRHH w w = = AAH H bb for w4. Solve the upper-triangular system R xR x = = ww for x

3

2

3

2

computation of flops

computation

WORK

o

f

: fl

f

o

lop3

s

s

p3

H

H

A A mn

nR R

nmn

Solution par la factorisation QR (Householder)

2 32WORK : flops

32 nmn

x xˆ ˆˆ ˆ , , m n n nA QR Q R reduced reduced QR factorizationQR factorization

1

ˆ ˆ

ˆ ˆ

ˆˆˆ ˆ

ˆ ˆ

H

H

H H

P QQ

y Pb QQ b Ax Q

A

x

x Q

R

Q RR b

1. Compute the reduced QR factorization ˆ ˆA QR

2. Compute the vector ˆ HQ b

3. Solve the upper-triangular system for xˆˆ HRx Q b

Solution par la SVD

2 3WORK : 1 f o s2 1 l pnmn

1

ˆ ˆ

ˆ ˆ ˆ

ˆ ˆˆ

ˆ

ˆ

H

H H

H HH

P UU

y Pb UU b Ax U V x

V x U b A V U

1. Compute the reduced SVD ˆ ˆ HA U V

2. Compute the vector ˆ HU b

3. Solve the diagonal system for wˆˆ Hw U b

4. Set x Vw

For this cost is approximately the same as for QR factorization,

but for the SVD is more expensive

m n

m n

Comparison of algorithms

� speed : normal equations� standard : QR factorization� A close to singular : SVD

DrawbacksDrawbacks• normal equationsnormal equations : not always stable in the presence of rounding errorsnot always stable in the presence of rounding errors• QR factoriz.QR factoriz.: less-than-ideal stability properties if less-than-ideal stability properties if AA is close to singular is close to singular• SVDSVD : expensive for expensive for mm nn

Conditionnement et précision

rela

su

tive condition number

px

f x

xf x

p

su supx x

A x x Ax x

xA

A

A A

A x Ax

Ax x

x

x

x x

1

1

1

1

1

1

x b Ax

b x A b

A x A b

xA A A

b

bA A A

x

A A A

accuracy : relative erro

r

f x f x

f x

backward stability

accuracy of

a B

accurac

S algorithm

y

machine

f x f x

fxO

x

Conditionnement du problème des moindres carrées

range(range(AA))y = A xy = A x

r = b - A xb

= Pb= Pb

x A b y Pb ˆ ˆ ˆ ˆH HP QQ UU AA

Données : A , b Solutions : x , y

12 2 2

2

A A A

arccos 2

2

y

b

closeness of the fit

2 2 2 2

2 2

A x A x

y Ax 2 2 22 2 2 2

2 2 2

A AA AA xA

Ax Ax Ax

b Ax

0 11 A2A

Conditionnement du problème des moindres carrées

Données : A , b Solutions : x , y

12 2 2

2

A A A

arccos 2

2

y

b

2 2 2 2

2 2

A x A x

y Ax

0 11 A2A

2-norm relative condition numbers

exact for certain exact for certain b b

upper boundsupper bounds

Stabilité des méthodes des moindres carrées

exemple

Least squares fitting of the function exp(sin(4)) on the interval [0,1] by a polynomial of degree 14

x15 = 1

highly ill-conditioned basis

very close fit

16machine 10

Stabilité des méthodes des moindres carrées

exemplefactorisation QR (Householder)

16machine 10

7rel. err. 3 10

The rounding errors have been amplified by a factor of order 10 9. This inaccuracy is explained by ill-conditioning, not instability.

reducedreduced

Stabilité des méthodes des moindres carrées

exemplefactorisation QR (Householder)

16machine 10

7rel. err. 3 10

implicit calculation of the product QH b

explicit formingQ

same accuracy

Stabilité des méthodes des moindres carrées

exemplefactorisation QR (Householder)

16machine 10

7rel. err. 3 10

implicit calculation of the product QH b

explicit formingQ

same accuracy\ QR factorization

with column pivoting

ˆ ˆAP

b

R

A

Q

an order of magn. more acc.

Stabilité des méthodes des moindres carrées

exemplefactorisation QR (Householder)

16machine 10

backward stable backward stable

This is true whether is computed via

ˆformation of or . It also holds for

House

expli

holde

mi

cit

implicitly

column pivot

r triangulari

n

za

in

tion with arbitr

.g

a y

ˆ

r

machine

H

AA A x

Q b

A

Q

b O

Stabilité des méthodes des moindres carrées

exemple

SVD

16machine 10

backward stable backward stable

min machine

AA A x b O

A

It beats Householder triangularization with column It beats Householder triangularization with column pivoting ( pivoting ( MATLAB's \MATLAB's \ ) by a factor of about ) by a factor of about 33

Stabilité des méthodes des moindres carrées

exemple

équations normales

16machine 10

unstable unstable

not even a single digit of accuracynot even a single digit of accuracy

factorisation de Cholesky

Stabilité des méthodes des moindres carrées

BS least squares algorithmBS least squares algorithm

2 tanmachine

x xO

x

suppose ill-conditioned, i.e. 1, and is bounded away from 2A

and

ta 1

n

or

tan

0

2machineO machineO

The condition The condition number of the LS number of the LS problem may lie problem may lie anywhere in the anywhere in the range range to to 22 . .

Stabilité des méthodes des moindres carrées

BS least squares algorithmBS least squares algorithm Cholesky factorization (BS)Cholesky factorization (BS)

2 tanmachine

x xO

x

2machine

x xO

x

suppose ill-conditioned, i.e. 1, and is bounded away from 2A

and

ta 1

n

or

tan

0

2machineO machineO 2

machineO 2machineO

2cond HA A

The normal equations are typically unstable for The normal equations are typically unstable for ill-conditioned problems involving close fits.ill-conditioned problems involving close fits.

Stabilité des méthodes des moindres carrées

The normal equations are typically unstable for The normal equations are typically unstable for ill-conditioned problems involving close fits.ill-conditioned problems involving close fits.

The solution of the full-rank least squares problem via the normal equations is unstable. Stability can be achieved, however, by restriction to a class of problems in which (A) is uniformly bounded above or (tan)/ is uniformly bounded below.

FINEFINE