Binocular Stereo · Principle of binocular vision is . triangulation. Given a single image, the 3D...

E-mail: [email protected]://web.yonsei.ac.kr/hgjung

Binocular StereoBinocular Stereo


Stereo Vision [1]Stereo Vision [1]

scene pointscene point

optical centeroptical center

image planeimage plane


Stereo Vision [1]Stereo Vision [1]

• Basic Principle: Triangulation– Gives reconstruction as intersection of two rays– Requires

• calibration• point correspondence


Point Correspondence [1]Point Correspondence [1]

p p’ ?

Given p in left image, where can the corresponding point p’ in right image be?


The Simplest Case: The Simplest Case: RectiRecti--linear Configuration [1]linear Configuration [1]

• Image planes of cameras are parallel.• Focal points are at same height.• Focal lengths same.• Then, epipolar lines are horizontal scan lines.



Imageplane

),,( ZYXP

),,( ZYXP

f

Oy

x

z

ZZ

YY

XX

ZZYYXX

OPPO



Imageplane

),,( ZYXP

),,( ZYXP

f

Oy

x

z

)1,,()1,,(),,(ZYf

ZXfyxZYX

YyXxZYfY

ZXfXfZ



),,(1 ZYXP 1Oy

x

z

f

2Oy

x

z

B

BfxxZ ,,, offunction a as for expression Derive 21

1p

2p



),,(1 ZYXP 1Oy

x

z

f

2Oy

x

z

B

211

11

1

12

1

11 ,

xxBfZ

ZBfx

ZBXfx

ZXfx



OOll OOrr

PP

ppll pprr

TT

ZZ

xxll xxrr

ff

T T is the stereo baselineis the stereo baselined d measures the difference in retinal position between correspondinmeasures the difference in retinal position between corresponding pointsg points

Then given Z, we can compute X and Y.

(xl,yl)=(f X/Z, f Y/Z)(xr,yr)=(f (X-T)/Z, f Y/Z)

Disparity:d=xl-xr=f X/Z – f (X-T)/Z


Stereo Constraints [1]Stereo Constraints [1]

X1

Y1

Z1O1

Image plane

Focal plane

M

p p’Y2

X2

Z2O2

Epipolar Line

Epipole


Epipolar Geometry and Fundamental Matrix [1]Epipolar Geometry and Fundamental Matrix [1]

• The geometry of two different images of the same scene is called the epipolar geometry.

• The geometric information that relates two different viewpoints of the same scene is entirely contained in a mathematical construct known as fundamental matrix.


3D from two views [1]3D from two views [1]

Principle of binocular vision is triangulationtriangulation. Given a single image, the 3D location of any visible object point must lie on the straight line that passes through COP and image point. Intersection of two such lines from two views is triangulation.

There are two ways of extracting 3D from a pair of images. • Classical method, called calibrated route, we need to calibrate both cameras (or viewpoints) w.r.t some world coordinate system. i.e, calculate the so-called epipolar geometry by extracting the essential matrix of the system.

• Second method, called uncalibrated route, a quantity known as fundamental matrix is calculated from image correspondences, and this is then used to determine the 3D.


Point Mapping between Images [1]Point Mapping between Images [1]

courtesy of F. Dellaert

x1 x’1

x2x’2

x3 x’3

Is there a transformation relating the points xi to x’i ?


Point Mapping between Images [1]Point Mapping between Images [1]

• What is the relationship between the images x, x’ of the scene point X in two views?

• Intuitively, it depends on:

– The rigid transformation between cameras (derivable from the camera matrices P, P’)

– The scene structure (i.e., the depth of X)

• Parallax: Closer points appear to move more


Baseline and Epipolar Plane [1]Baseline and Epipolar Plane [1]

• Baseline: Line joining camera centers C, C’• Epipolar plane π

: Defined by baseline and scene point X

from Hartley& Zisserman

baseline


Epipoles and Epipolar Lines [1]Epipoles and Epipolar Lines [1]

• Epipolar lines l, l’: Intersection of epipolar plane π

with image planes

• Epipoles e, e’: Where baseline intersects image planes– Equivalently, the image in one view of the other camera center.

C C’from Hartley& Zisserman


Epipolar Pencil [1]Epipolar Pencil [1]

• As position of X varies, epipolar planes “rotate” about the baseline (like a book with pages)– This set of planes is called the epipolar pencil

• Epipolar lines “radiate” from epipole—this is the pencil of epipolar lines



Epipolar Constraints [1]Epipolar Constraints [1]

• Camera center C and image point x define ray in 3-D space that projects to epipolar line l’ in other view (since it’s on the epipolar plane)

• 3-D point X is on this ray, so image of X in other view x’ must be on l’

• In other words, the epipolar geometry defines a mapping x→l’, of points in one image to lines in the other


C C’

x’


Epipolar Lines of Converging CamerasEpipolar Lines of Converging Cameras

from Hartley & Zisserman

Which is left image? How can we determine it?


Epipolar Lines of Converging CamerasEpipolar Lines of Converging Cameras

from Hartley & ZissermanLeft view Right view

Intersection of epipolar lines = EpipoleIndicates direction of other camera


From Geometry to Algebra [1]From Geometry to Algebra [1]

O O’

P

pp’



Linear Constraint: Should be able to express as matrix multiplication.

Rotation arrow should be at the other end, to get p’ in p coordinates


Matrix Form of CrossMatrix Form of Cross--product [1]product [1]


The Essential Matrix [1]The Essential Matrix [1]

If un-calibrated, p gets multiplied by Intrisic matrix, K


The Fundamental Matrix [1]The Fundamental Matrix [1]

• Mapping of point in one image to epipolar line in other image x

l’ is expressed algebraically by the fundamental matrix F

• Write this as l’ = F x• Since x’ is on l’, by the point-on-line definition we know that

x’T l’ = 0• Substitute l’ = F x, we can thus relate corresponding points in

the camera pair (P, P’) to each other with the following:x’T F x = 0

linepoint



• F is 3 x 3, rank 2 (not invertible, in contrast to homographies)– 7 DOF (homogeneity and rank constraint take away 2 DOF)

• The fundamental matrix of (P’, P) is the transpose FT


In computer vision, the fundamental matrix F is a 3×3 matrix which relates corresponding points in stereo images. In epipolar geometry, with homogeneous image coordinates, x and x′, of corresponding points in a stereo image pair, Fx describes a line (an epipolar line) on which the corresponding point x′

on the other image must lie. That means, for all pairs of corresponding points holds

Being of rank two and determined only up to scale, the fundamental matrix can be estimated given at least seven point correspondences. Its seven parameters represent the only geometric information about cameras that can be obtained through point correspondences alone.

The above relation which defines the fundamental matrix was published in 1992 by both Faugeras and Hartley.



Longuet-Higgins' essential matrix satisfies a similar relationship, the essential matrix is a metric object pertaining to calibrated cameras,while the fundamental matrix describes the correspondence in more general and fundamental terms of projective geometry. This is captured mathematically by the relationship between a fundamental matrix F and its corresponding essential matrix E, which is

K and K’ being the intrinsic calibration matrices of the two images involved



Computing Fundamental Matrix [1]Computing Fundamental Matrix [1]

Fundamental Matrix is singular with rank 2

In principal F has 7 parameters up to scale and can be estimated from 7 point correspondences

Direct Simpler Method requires 8 correspondences


Pseudo InversePseudo Inverse--Based [1]Based [1]

0uFuT

Each point correspondence can be expressed as a linear equation

01

1

333231

232221

131211

vu

FFFFFFFFF

vu

01

33

32

31

23

22

21

13

12

11

FFFFFFFFF

vuvvvvuuvuuu


Pseudo InversePseudo Inverse--Based [1]Based [1]


• Input: n point correspondences ( n >= 8)– Construct homogeneous system Ax= 0 from

• x = (f11 ,f12 , ,f13 , f21 ,f22 ,f23 f31 ,f32 , f33 ) : entries in F• Each correspondence gives one equation• A is a nx9 matrix (in homogenous format)

– Obtain estimate F^ by SVD of A• x (up to a scale) is column of V corresponding to the

least singular value– Enforce singularity constraint: since Rank (F) = 2

• Compute SVD of F^• Set the smallest singular value to 0: D -> D’• Correct estimate of F :

• Output: the estimate of the fundamental matrix, F’• Similarly we can compute E given intrinsic parameters

The EightThe Eight--Point Algorithm [1]Point Algorithm [1]

0lT

r pFp

TUDVA

TUDVF ˆ

TVUDF' '


Locating the Epipoles from F [1]Locating the Epipoles from F [1]

• Input: Fundamental Matrix F– Find the SVD of F– The epipole el is the column of V corresponding to the null

singular value (as shown above)– The epipole er is the column of U corresponding to the null

singular value• Output: Epipole el and er

TUDVF

el lies on all the epipolar lines of the left image

0lT

r pFp

0lT

r eFp

F is not identically zero

True For every pr

0leF

pl pr

P

Ol Orel er

Pl Pr

Epipolar Plane

Epipolar Lines

Epipoles


Corner Detection [5]Corner Detection [5]Im1.jpg Im2.jpg

thresh = 500; % Harris corner threshold

% Find Harris corners in image1 and image2 [cim1, r1, c1] = harris(im1, 1, thresh, 3); [cim2, r2, c2] = harris(im2, 1, thresh, 3);

http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/Robust/example/im1.jpg

http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/Robust/example/im2.jpg


CorrelationCorrelation--Based Matching [5]Based Matching [5]

dmax = 50; % Maximum search distance for matchingw = 11; % Window size for correlation matching

% Use normalised correlation matching[m1,m2] = matchbycorrelation(im1, [r1';c1'], im2, [r2';c2'], w, dmax);

% Display putative matchesshow(im1,3), set(3,'name','Putative matches')

for n = 1:length(m1);line([m1(2,n) m2(2,n)], [m1(1,n) m2(1,n)])

end

Putative matches


RANSACRANSAC--Based Fundamental Matrix Estimation [5]Based Fundamental Matrix Estimation [5]

% Assemble homogeneous feature coordinates for fitting of the% fundamental matrix, note that [x,y] corresponds to [col, row]x1 = [m1(2,:); m1(1,:); ones(1,length(m1))];x2 = [m2(2,:); m2(1,:); ones(1,length(m1))];

t = .002; % Distance threshold for deciding outliers

[F, inliers] = ransacfitfundmatrix(x1, x2, t, 1);

Inlying

matches


EpipolarEpipolar

Lines [5]Lines [5]

% Step through each matched pair of points and display the% corresponding epipolar lines on the two images.

l2 = F*x1; % Epipolar lines in image2l1 = F'*x2; % Epipolar lines in image1

% Solve for epipoles[U,D,V] = svd(F);e1 = hnormalise(V(:,3));e2 = hnormalise(U(:,3));

for n = inliersfigure(1), clf, imshow(im1), hold on, plot(x1(1,n),x1(2,n),'r+');hline(l1(:,n)); plot(e1(1), e1(2), 'g*');

figure(2), clf, imshow(im2), hold on, plot(x2(1,n),x2(2,n),'r+');hline(l2(:,n)); plot(e2(1), e2(2), 'g*');

end


EpipolarEpipolar

Lines [5]Lines [5]


Estimation of Camera Matrix [6]Estimation of Camera Matrix [6]

• Once the essential matrix is known, the camera matrices may be retrieved from E. In contrast with the fundamental matrix case, where there is a projective ambiguity, the camera matrices may be retrieved from the essential matrix up to scale and a four-fold ambiguity.

• A 3x3 matrix is an essential matrix if and only if two of its singular values are equal, and the third is zero.

• We may assume that the first camera matrix is P=[I|0]. In order to compute the second camera matrix, P’, it is necessary to factor E into the product SR of a skew- symmetric matrix and a rotation matrix.

• Suppose that the SVD of E is Udiag(1,1,0)VT. Using the notation of W and Z, there are (ignoring signs) two possible factorizations E=SR as follows:

S=UZUT, R=UWVT or UWTVT

0 1 01 0 00 0 1

W0 1 01 0 0

0 0 0

Z

ZW= 1 0 00 1 00 0 0

E=SR=(UZUT)(UWVT)= UZ(UTU)WVT

= UZWVT

= U diag(1,1,0) VT


Estimation of Camera Matrix [7]Estimation of Camera Matrix [7]

S=UZUT, R=UWVT

or UWTVT

• For a given essential matrix E=Udiag(1,1,0)VT, and first camera matrix P=[I|0], there are four possible choices for the second camera matrix P’, namely

P’=[UWVT|u3 ] or [UWVT|-u3

] or [UWTVT|u3

] or [UWTVT|-u3

]

Where, u3

is the last column of U.

The four possible solutions for calibrated reconstruction from E.

Epipole


Linear triangulation methods [8]Linear triangulation methods [8]

•

In each image we have a measurement x=PX, x’=P’X, and these equations can

be combined into a form AX=0, which is an equation linear in X.

•

The homogeneous scale factor is eliminated by a cross product to give three

equations for

each image point, of which two are linearly independent.

• For the first image, x×(PX)=0 and writing this out gives

Where, piT

are the rows of P.•

1

2

3

0 11 0

0

T

T

T

yx

y x

p Xp X 0p X

3 1

3 2

2 1

( ) ( ) 0( ) ( ) 0( ) ( ) 0

T T

T T

T T

xyx y

p X p Xp X p Xp X p X

3 1

3 2

3 1

3 2

' ' '' ' '

T T

T T

T T

T T

xy

xy

p pp p

Ap pp p


Linear triangulation methods [8]Linear triangulation methods [8]

• An equation of the form AX=0

can then be composed, with

Where two equations have been included from each image, giving a

total of four

equations in four homogeneous unknowns. This is a redundant set of equations, since the solution is determined only up to scale.

X

can be calculated by SVD of A.

3 1

3 2

3 1

3 2

' ' '' ' '

T T

T T

T T

T T

xy

xy

p pp p

Ap pp p


Fundamental matrix estimation methodsFundamental matrix estimation methods

Y

X

Z

'X

'Y

'Z

camx X 'camx

cam x PX ' 'cam x P X

0cam x PX ' ' 0cam x P X3T 1T

3T 2T

3T 1T

3T 2T

p pp p

where A'p' p'' p' p'

xy

xy

AX = 03T 1T

3T 2T

2T 1T

p p 0p p 0p p 0

xyx y

X XX XX X

3T 1T

3T 2T

2T 1T

' p' p' 0' p' p' 0' p' ' p' 0

xyx y

X XX XX X


Stereo Rectification [1]Stereo Rectification [1]

• Image Reprojection– reproject image planes onto common

plane parallel to line between optical centers

•• Notice, only focal point of camera really Notice, only focal point of camera really mattersmatters

How can we make images as in recti- linear configuration?

Stereo Rectification



• Rectification – Given a stereo pair, the intrinsic and extrinsic parameters, find the

image transformation to achieve a stereo system of horizontal epipolar lines

– A simple algorithm: Assuming calibrated stereo cameras

p’ lp’r

P

Ol OrX’r

Pl Pr

Z’l

Y’l Y’r

TX’l

Z’r

Stereo System with Parallel Optical Axes

Epipoles are at infinity

Horizontal epipolar lines



• Algorithm– Rotate both left and

right camera so that they share the same X axis : Or -Ol = T

– Define a rotation matrix Rrect for the left camera

– Rotation Matrix for the right camera is RrectRT

– Rotation can be implemented by image transformation

pl pr

P

Ol Or

Xl

Xr

Pl Pr

Zl

Yl

Zr

Yr

R, T

TX’l

Xl ’ = T_axis, Yl ’ = Xl ’xZl , Z’l = Xl ’xYl ’


The Fundamental Matrix Song [4]The Fundamental Matrix Song [4]

The fundamental matrix

Used in stereo geometry

A matrix with nine entries

It's square with size 3 by 3

Has seven degrees of freedom

It has a rank deficiency

It's only of rank two

Call the matrix F and you'll see...

Two points that correspond

Column vectors called x and x-prime

x-prime transpose times F times x

Equals zero

every time

The epipolar constraint

Involves epipolar lines

Postmultiplying

F by x

Results in vector l-prime

It's the epipolar line

In the other view passing through x-prime

A three component vector

Of homogeneous design

The left and right nullspaces

of F

Are the epipoles

e-prime and e

All of the epipolar lines

Should pass through these

Here's a linear estimation example:

Take a set of 8 point samples

Construct a matrix, take the SVD

And the elements of F are in the last column of V

If you try to estimate

F with a coplanar set of points

Your sample set will be degenerate

And will not bring you joy

When doing the estimation

If you don't perform rank deprivation

Your epipolar lines

And the epipoles will not coincide

But if your scene has three views

The trifocal tensor

is what you'd use

Constraints from the third view act like glue

That can't be determined from just two views


ReferencesReferences

1. Chandra Kambhamettu, “Multipleview1,” Delaware University Lecture Material of Computer Vision (CISC 4/689), 2007.

2. Wikipedia, “Fundamental Matrix (Computer Vision),” http://en.wikipedia.org/wiki/Fundamental_matrix_(computer_vision)

3. Sebastian Thrun, Rick Szeliski, Hendrik Dahlkamp and Dan Morris, “Stereo1,” Stanford University Lecture Material of Computer Vision (CS223B), Winter 2005.

4. Daniel Wedge, “The Fundamental Matrix Song,” http://danielwedge.com/fmatrix/5. Peter Kovesi, “Example of finding the fundamental matrix using RANSAC,” available

at http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/Robust/example/index.html

6. Richard Hartley and Andrew Zisserman, “8.6 The essential matrix,” Multiple View Geometry in Computer Vision, Cambridge, pp. 238-241.

7. Richard Hartley and Andrew Zisserman, “11.2 Linear triangulation methods,” Multiple View Geometry in Computer Vision, Cambridge, pp. 238-241.

http://danielwedge.com/fmatrix/

http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/Robust/example/index.html


3D Simulation3D Simulation


1. Motion Field: Camera Model1. Motion Field: Camera Model

• Making a projection matrix P using related parameters.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%f=12*0.001; % 12mmImagerSize=1/3; % 1/3 inchHres=640; Vres=480;u0=320; v0=240;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%PixelSize=(ImagerSize*0.0254)*(Hres/sqrt(Hres^2+Vres^2))/Hres;ku=1/PixelSize; kv=1/PixelSize;

au=-f*ku; av=-f*kv;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%Mint=[au 0 u0 0; 0 av

v0 0; 0 0 1 0];Mext=[[RotationMatrix(0,0,0) [0 0 0]']; [0 0 0 1]];P=Mint*Mext;

• 1inch=25.4mm• Imager size is the

diagonal length of the imager


1. Motion Field: Initial Image1. Motion Field: Initial Image

• Making a test pattern and its image.

N=21;X=zeros(N,N,3);Z0=3;d=0.1;for i=1:N

for j=1:Nx=d*(i-round(N/2));y=d*(j-round(N/2));X(j,i,:)=[x,y,Z0];

endend

Xh=[reshape(X,[N*N,3]) ones(N*N,1)]';xh0=P*Xh;x0=zeros(2,size(xh0,2));x0(1,:)=xh0(1,:)./xh0(3,:);x0(2,:)=xh0(2,:)./xh0(3,:);

figure; axis([1 640 1 480]); hold on;plot(x0(1,:),x0(2,:),'b+');

100 200 300 400 500 600

50

100

150

200

250

300

350

400

450

-1-0.5

00.5

1

-1-0.5

0

0.512

2.5

3

3.5

4

plot3(X(:,:,1),X(:,:,2),X(:,:,3),'b.');


2. 3D Reconstruction: Two Projection Matrix2. 3D Reconstruction: Two Projection Matrix

• Defining each projection matrixe of two cameras.

Mint=[au 0 u0 0; 0 av

v0 0; 0 0 1 0];

Mext1=[[eye(3) [0 0 0]']; [0 0 0 1]];P1=Mint*Mext1;

R2=RotationMatrix(0,-0.2,0.3);T2=[0.3 0 -1]';Mext2=[[R2 T2]; [0 0 0 1]];P2=Mint*Mext2;


-2

0

2

-3 -2 -1 0 1 2 3

0

1

2

3

4

5

6

7

8

9

10

yx

z

2. 3D Reconstruction: 3D Objects2. 3D Reconstruction: 3D Objects

• Making two 3D objects, cubes.

Cube=[1 -1 -1

1 1 -1 -1

1; 1 1 -1 -1

1 1 -1 -1; -1 -1

-1

-1

1 1 1 1];R1=RotationMatrix(-10/180*pi, 20/180*pi, 45/180*pi);Cube1=0.3*R1*Cube+[0.5*ones(1,8); 0.8*ones(1,8); 5*ones(1,8)];R2=RotationMatrix(20/180*pi, -30/180*pi, -15/180*pi);Cube2=0.5*R2*Cube+[-0.1*ones(1,8); -0.5*ones(1,8); 7*ones(1,8)];

figure; hold on;DrawCube3D(Cube1);DrawCube3D(Cube2);axis equal;axis([-3 3 -3 3 0 10]);xlabel('x'); ylabel('y'); zlabel('z'); grid on;

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

x

y


2. 3D Reconstruction: DrawCube2D2. 3D Reconstruction: DrawCube2D

function DrawCube2D(cube)line([cube(1,1) cube(1,2)],[cube(2,1) cube(2,2)]);line([cube(1,2) cube(1,3)],[cube(2,2) cube(2,3)]);line([cube(1,3) cube(1,4)],[cube(2,3) cube(2,4)]);line([cube(1,4) cube(1,1)],[cube(2,4) cube(2,1)]);

line([cube(1,1) cube(1,5)],[cube(2,1) cube(2,5)]);line([cube(1,2) cube(1,6)],[cube(2,2) cube(2,6)]);line([cube(1,3) cube(1,7)],[cube(2,3) cube(2,7)]);line([cube(1,4) cube(1,8)],[cube(2,4) cube(2,8)]);

line([cube(1,5) cube(1,6)],[cube(2,5) cube(2,6)]);line([cube(1,6) cube(1,7)],[cube(2,6) cube(2,7)]);line([cube(1,7) cube(1,8)],[cube(2,7) cube(2,8)]);line([cube(1,8) cube(1,5)],[cube(2,8) cube(2,5)]);

plot(cube(1,1),cube(2,1),'r.');plot(cube(1,2),cube(2,2),'g.');plot(cube(1,3:8),cube(2,3:8),'b.');


2. 3D Reconstruction: DrawCube3D2. 3D Reconstruction: DrawCube3D

function DrawCube3D(Cube)line([Cube(1,1) Cube(1,2)],[Cube(2,1) Cube(2,2)], [Cube(3,1) Cube(3,2)]);line([Cube(1,2) Cube(1,3)],[Cube(2,2) Cube(2,3)], [Cube(3,2) Cube(3,3)]);line([Cube(1,3) Cube(1,4)],[Cube(2,3) Cube(2,4)], [Cube(3,3) Cube(3,4)]);line([Cube(1,4) Cube(1,1)],[Cube(2,4) Cube(2,1)], [Cube(3,4) Cube(3,1)]);

line([Cube(1,1) Cube(1,5)],[Cube(2,1) Cube(2,5)], [Cube(3,1) Cube(3,5)]);line([Cube(1,2) Cube(1,6)],[Cube(2,2) Cube(2,6)], [Cube(3,2) Cube(3,6)]);line([Cube(1,3) Cube(1,7)],[Cube(2,3) Cube(2,7)], [Cube(3,3) Cube(3,7)]);line([Cube(1,4) Cube(1,8)],[Cube(2,4) Cube(2,8)], [Cube(3,4) Cube(3,8)]);

line([Cube(1,5) Cube(1,6)],[Cube(2,5) Cube(2,6)], [Cube(3,5) Cube(3,6)]);line([Cube(1,6) Cube(1,7)],[Cube(2,6) Cube(2,7)], [Cube(3,6) Cube(3,7)]);line([Cube(1,7) Cube(1,8)],[Cube(2,7) Cube(2,8)], [Cube(3,7) Cube(3,8)]);line([Cube(1,8) Cube(1,5)],[Cube(2,8) Cube(2,5)], [Cube(3,8) Cube(3,5)]);

plot3(Cube(1,1),Cube(2,1),Cube(3,1),'r.');plot3(Cube(1,2),Cube(2,2),Cube(3,2),'g.');plot3(Cube(1,3:8),Cube(2,3:8),Cube(3,3:8),'b.');


2. 3D Reconstruction: Image onto Camera12. 3D Reconstruction: Image onto Camera1

• Making image onto camera1 using perspective transformation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Camera1에

대한

image 만들기%%%%%%%%%%%%%%%%%%%%%%%%%%%%Xh1=[Cube1; ones(1, 8)];th1=P1*Xh1;t1=zeros(2,size(th1,2));t1(1,:)=th1(1,:)./th1(3,:);t1(2,:)=th1(2,:)./th1(3,:);

Xh2=[Cube2; ones(1, 8)];th2=P1*Xh2;t2=zeros(2,size(th2,2));t2(1,:)=th2(1,:)./th2(3,:);t2(2,:)=th2(2,:)./th2(3,:);

x1=[t1 t2];

figure; axis([1 640 1 480]); hold on;DrawCube2D(t1);DrawCube2D(t2);


2. 3D Reconstruction: Image onto Camera22. 3D Reconstruction: Image onto Camera2

• Making image onto camera2 using perspective transformation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Camera2에

대한

image 만들기%%%%%%%%%%%%%%%%%%%%%%%%%%%%th1=P2*Xh1;t1=zeros(2,size(th1,2));t1(1,:)=th1(1,:)./th1(3,:);t1(2,:)=th1(2,:)./th1(3,:);

th2=P2*Xh2;t2=zeros(2,size(th2,2));t2(1,:)=th2(1,:)./th2(3,:);t2(2,:)=th2(2,:)./th2(3,:);

x2=[t1 t2];

figure; axis([1 640 1 480]); hold on;DrawCube2D(t1);DrawCube2D(t2);


2. 3D Reconstruction: Generated Stereo Images2. 3D Reconstruction: Generated Stereo Images

100 200 300 400 500 600

50

100

150

200

250

300

350

400

450

100 200 300 400 500 600

50

100

150

200

250

300

350

400

450

Camera1 Camera2


2. 3D Reconstruction: Camera Parameter Estimation2. 3D Reconstruction: Camera Parameter Estimation

%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Fundamental matrix와

camera matrix를

추정한다.%%%%%%%%%%%%%%%%%%%%%%%%%%%%tol

= .001; % Distance threshold for deciding outliers

[F, inliers] = ransacfitfundmatrix(x1, x2, tol, 1);

K=Mint(:,1:3);E=K'*F*K;[U,S,V]=svd(E);W=[0 -1 0; 1 0 0; 0 0 1];Z=[0 1 0; -1 0 0; 0 0 0];

S=U*Z*U';R_=U*W'*V';

P1_=K*[eye(3) [0 0 0]']P2_=K*[R_ U(:,3)];

K

is required if 3D reconstruction will be done with image coordinates.

Selecting the best from 4 candidates by hand.


2. 3D Reconstruction: Triangulation2. 3D Reconstruction: Triangulation

• In each image we have a measurement x=PX, x’=P’X, and these equations can

be combined into a form AX=0, which is an equation linear in X.• An equation of the form AX=0 can then be composed, with

Where two equations have been included from each image, giving a

total of four

equations in four homogeneous unknowns. This is a redundant set of equations, since the solution is determined only up to scale.

• X

can be calculated by SVD of A.

3 1

3 2

3 1

3 2

' ' '' ' '

T T

T T

T T

T T

xy

xy

p pp p

Ap pp p


2. 3D Reconstruction: Triangulation2. 3D Reconstruction: Triangulation

%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 3차원

정보

복원하기%%%%%%%%%%%%%%%%%%%%%%%%%%%%N=size(x1,2);

X_=zeros(3,N);

for n=1:NA=[x1(1,n)*P1_(3,:)-P1_(1,:);...

x1(2,n)*P1_(3,:)-P1_(2,:);...x2(1,n)*P2_(3,:)-P2_(1,:);...x2(2,n)*P2_(3,:)-P2_(2,:)];

[U,S,V]=svd(A);Xh_=V(:,4);X_(:,n)=Xh_(1:3)./Xh_(4);

end

Cube1_=X_(:,1:8);Cube2_=X_(:,9:16);

figure; hold on;DrawCube3D(Cube1_);DrawCube3D(Cube2_);axis equal;axis([-3 3 -3 3 0 10]);xlabel('x'); ylabel('y'); zlabel('z'); grid on;

• Notice that the 3D reconstruction is done 3D point by point.

3 1

3 2

3 1

3 2

' ' '' ' '

T T

T T

T T

T T

xy

xy

p pp p

Ap pp p


2. 3D Reconstruction: Reconstruction Result2. 3D Reconstruction: Reconstruction Result

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

x

y

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

xy

Original Objects Reconstructed Objects


2. 3D Reconstruction: Reconstruction Result2. 3D Reconstruction: Reconstruction Result

• Notice that the result is up to scale.

-2

0

2

-3 -2 -1 0 1 2 3

0

1

2

3

4

5

6

7

8

9

10

yx

z

Original Objects Reconstructed Objects

-2

0

2

-3 -2 -1 0 1 2 3

0

1

2

3

4

5

6

7

8

9

10

yx

z

Binocular Stereo · Principle of binocular vision is . triangulation. Given a single image, the 3D...

Documents

Transcript of Binocular Stereo · Principle of binocular vision is . triangulation. Given a single image, the 3D...