Computer Vision 2 Lecture 5...Compute an updated estimate of the state from prediction and...

Computer Vision 2 – Lecture 5 Tracking with Linear Dynamic Models(02.05.2016)

Prof. Dr. Bastian Leibe, Dr. Jörg Stückler

[email protected], [email protected]

RWTH Aachen University, Computer Vision Group

http://www.vision.rwth-aachen.de

mailto:[email protected]





http://www.vision.rwth-aachen.de/



2

Content of the Lecture

• Single-Object Tracking

• Bayesian Filtering Kalman Filters, EKF

Particle Filters

• Multi-Object Tracking

• Visual Odometry

• Visual SLAM & 3D Reconstruction

Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking


time

Measurements

States

3

Recap: Tracking-by-Detection

• Main ideas Apply a generic object detector to find objects of a certain class

Based on the detections, extract object appearance models

Link detections into trajectories



4

Recap: Elements of Tracking

• Detection Where are candidate objects?

• Data association Which detection corresponds to which object?

• Prediction Where will the tracked object be in the next time step?



Detection Data association Prediction

Last lecture

5

Recap: Sliding-Window Object Detection

• Fleshing out this

pipeline a bit more,

we need to: 1. Obtain training data

2. Define features

3. Define classifier



Car/non-car

Classifier

Feature

extraction

Training examples

Slide credit: Kristen Grauman

6

Recap: Object Detector Design

• In practice, the classifier often determines the design. Types of features

Speedup strategies

• We looked at 3 state-of-the-art detector designs Based on SVMs

Based on Boosting

Based on CNNs



7

Recap: Histograms of Oriented Gradients (HOG)

• Holistic object representation Localized gradient orientations



Image Window

Object/Non-object

Linear SVM

Collect HOGs over

detection window

Contrast normalize over

overlapping spatial cells

Weighted vote in spatial &

orientation cells

Compute gradients

Gamma compression

Slide adapted from Navneet Dalal

8

Recap: Deformable Part-based Model (DPM)

• Multiscale model captures features at two resolutions



Score of object hypothesis

is sum of filter scores

minus deformation costs

Score of filter:

dot product of filter with

HOG features underneath

it

Slide credit: Pedro Felzenszwalb

[Felzenszwalb, McAllister, Ramanan, CVPR’08]

9

Recap: DPM Hypothesis Score



Slide credit: Pedro Felzenszwalb

[Felzenszwalb, McAllister, Ramanan, CVPR’08]

10

Recap: Integral Channel Features

• Generalization of Haar Wavelet idea from Viola-Jones Instead of only considering intensities, also take into account other

feature channels (gradient orientations, color, texture).

Still efficiently represented as integral images.



P. Dollar, Z. Tu, P. Perona, S. Belongie. Integral Channel Features, BMVC’09.

http://vision.ucsd.edu/~pdollar/files/papers/DollarBMVC09ChnFtrs.pdf

11

Recap: Integral Channel Features

• Generalize also block computation 1st order features:

Sum of pixels in rectangular region.

2nd-order features:

Haar-like difference of sum-over-blocks

Generalized Haar:

More complex combinations of weighted rectangles

Histograms

Computed by evaluating local sums on quantized images.



12

Recap: VeryFast Detector

• Idea 1: Invert the template scale vs. image scale relation



1 model,

50 image scales

50 models,

1 image scale

R. Benenson, M. Mathias, R. Timofte, L. Van Gool. Pedestrian Detection at

100 Frames per Second, CVPR’12.

Slide credit: Rodrigo Benenson

http://rodrigob.github.io/documents/2012_cvpr_pedestrian_detection_at_100_frames_per_second.pdf

http://rodrigob.github.io/documents/2012_cvpr_pedestrian_detection_at_100_frames_per_second.pdf

13

Recap: VeryFast Detector

• Idea 2: Reduce training time by feature interpolation

• Shown to be possible for Integral Channel features P. Dollár, S. Belongie, Perona. The Fastest Pedestrian Detector in the

West, BMVC 2010.




5 models,

1 image scale

50 models,

1 image scale

≈

http://vision.ucsd.edu/~pdollar/files/papers/DollarBMVC10FPDW.pdf

http://vision.ucsd.edu/~pdollar/files/papers/DollarBMVC10FPDW.pdf

14

Recap: VeryFast Classifier Construction

• Ensemble of short trees, learned by AdaBoost



𝑠𝑐𝑜𝑟𝑒 = 𝑤1 ⋅ ℎ1 +

+1 -1 +1 -1 +1 -1 +1 -1

𝑤2 ⋅ ℎ2 +

+1 -1 +1 -1

⋯

⋯ +𝑤𝑁 ⋅ ℎ𝑁


15

Recap: Elements of Tracking

• Detection Where are candidate objects?

• Data association Which detection corresponds to which object?

• Prediction Where will the tracked object be in the next time step?



Detection Data association Prediction

Last lecture

Today’s topic

16

Today: Tracking with Linear Dynamic Models



Figure from Forsyth & Ponce

17

Topics of This Lecture

• Tracking with Dynamics Detection vs. Tracking

Tracking as probabilistic inference

Prediction and Correction

• Linear Dynamic Models Zero velocity model

Constant velocity model

Constant acceleration model

• The Kalman Filter Kalman filter for 1D state

General Kalman filter

Limitations



18

Tracking with Dynamics

• Key idea Given a model of expected motion, predict

where objects will occur in next frame,

even before seeing the image.

• Goals Restrict search for the object

Improved estimates since measurement noise is reduced by trajectory

smoothness.

• Assumption: continuous motion patterns Camera is not moving instantly to new viewpoint.

Objects do not disappear and reappear in different places.

Gradual change in pose between camera and scene.



Slide adapted from S. Lazebnik, K. Grauman

19

General Model for Tracking

• Representation

The moving object of interest is characterized by an underlying state X.

State X gives rise to measurements or observations Y.

At each time t, the state changes to Xt and we get a new observation

Yt.



X1 X2

Y1 Y2

Xt

Yt

…

Slide credit: Svetlana Lazebnik

20

State vs. Observation

• Hidden state : parameters of interest

• Measurement: what we get to directly observe




21

Tracking as Inference

• Inference problem The hidden state consists of the true parameters we care about,

denoted X.

The measurement is our noisy observation that results from the underlying state, denoted Y.

At each time step, state changes (from Xt-1 to Xt) and we get a new

observation Yt.

• Our goal: recover most likely state Xt given

All observations seen so far.

Knowledge about dynamics of state transitions.




22

Steps of Tracking

• Prediction: What is the next state of the object given past measurements?

• Correction: Compute an updated estimate of the state from prediction and

measurements.

• Tracking Can be seen as the process of propagating the posterior distribution of

state given measurements across time.



1100 ,, ttt yYyYXP

ttttt yYyYyYXP ,,, 1100

23

Simplifying Assumptions

• Only the immediate past matters

• Measurements depend only on the current state



110 ,, tttt XXPXXXP

Dynamics model

tttttt XYPXYXYXYP ,,,, 1100

Observation model

X1 X2

Y1 Y2

Xt

Yt

…


24

Tracking as Induction

• Base case: Assume we have initial prior that predicts state in absence of any

evidence: P(X0)

At the first frame, correct this given the value of Y0=y0



)()|()(

)()|()|( 000

0

000000 XPXyP

yP

XPXyPyYXP

Posterior prob.

of state given

measurement

Likelihood of

measurement

Prior of

the state


25

Tracking as Induction

• Base case: Assume we have initial prior that predicts state in absence of any

evidence: P(X0)

At the first frame, correct this given the value of Y0=y0

• Given corrected estimate for frame t:

Predict for frame t+1

Correct for frame t+1




predict correct

26

Induction Step: Prediction

• Prediction involves representing

given



10 ,, tt yyXP 101 ,, tt yyXP

11011

1101101

1101

,,||

,,|,,,|

,,,

ttttt

tttttt

tttt

dXyyXPXXP

dXyyXPyyXXP

dXyyXXP

10 ,, tt yyXP

Law of total probability

,P A P A B dB


27



given





11011

1101101

1101

,,||

,,|,,,|

,,,

ttttt

tttttt

tttt

dXyyXPXXP

dXyyXPyyXXP

dXyyXXP

10 ,, tt yyXP

Conditioning on Xt–1

, |P A B P A B P B

28



given





11011

1101101

1101

,,||

,,|,,,|

,,,

ttttt

tttttt

tttt

dXyyXPXXP

dXyyXPyyXXP

dXyyXXP

10 ,, tt yyXP

Independence assumption

29

Induction Step: Correction

• Correction involves computing

given predicted value



tt yyXP ,,0 10 ,, tt yyXP

ttttt

tttt

tt

tttt

tt

ttttt

dXyyXPXyP

yyXPXyP

yyyP

yyXPXyP

yyyP

yyXPyyXyP

10

10

10

10

10

1010

,,||

,,||

,,|

,,||

,,|

,,|,,,|

tt yyXP ,,0

Bayes rule

||

P B A P AP A B

P B


30








ttttt

tttt

tt

tttt

tt

ttttt

dXyyXPXyP

yyXPXyP

yyyP

yyXPXyP

yyyP

yyXPyyXyP

10

10

10

10

10

1010

,,||

,,||

,,|

,,||

,,|

,,|,,,|

tt yyXP ,,0

Independence assumption

(observation yt depends only on state Xt)

31








ttttt

tttt

tt

tttt

tt

ttttt

dXyyXPXyP

yyXPXyP

yyyP

yyXPXyP

yyyP

yyXPyyXyP

10

10

10

10

10

1010

,,||

,,||

,,|

,,||

,,|

,,|,,,|

tt yyXP ,,0

Conditioning on Xt

32

Summary: Prediction and Correction

• Prediction:



1101110 ,,||,,| ttttttt dXyyXPXXPyyXP

Dynamics

model

Corrected estimate

from previous step


33

Summary: Prediction and Correction

• Prediction:

• Correction:



1101110 ,,||,,| ttttttt dXyyXPXXPyyXP

Dynamics

model

Corrected estimate

from previous step


ttttt

tttttt

dXyyXPXyP

yyXPXyPyyXP

10

100

,,||

,,||,,|

Observation

model

Predicted

estimate

34










Limitations



35

Notation Reminder

• Random variable with Gaussian probability distribution that has the mean vector ¹ and covariance matrix .

• x and ¹ are d-dimensional, is d£d.



),(~ Σμx N

d =2 d =1

If x is 1D, we

just have one

parameter:

the variance ¾2


36

Linear Dynamic Models

• Dynamics model State undergoes linear tranformation Dt plus Gaussian noise

• Observation model Measurement is linearly transformed state plus Gaussian noise



1~ ,tt t t dN x D x

~ ,tt t t mN y M x

nn n1 n£1

m1 mn n1

Slide credit: Svetlana Lazebnik, Kristen Grauman

37

Example: Randomly Drifting Points

• Consider a stationary object, with state as position. Position is constant, only motion due to random noise term.

State evolution is described by identity matrix D=I



1t tp p t tx p

1 1t t t tx D x noise Ip noise


38

Example: Constant Velocity (1D Points)



time

Measurements

States

Slide credit: Kristen Grauman Figure from Forsyth & Ponce

39


• State vector: position p and velocity v

• Measurement is position only



1

11 )(

tt

ttt

vv

vtpp

t

t

tv

px

noisev

ptnoisexDx

t

t

ttt

1

1

110

1

(greek letters

denote noise

terms)

noisev

pnoiseMxy

t

t

tt

01


40


• State vector: position p and velocity v




1

11 )(

tt

ttt

vv

vtpp

t

t

tv

px

noisev

ptnoisexDx

t

t

ttt

1

1

110

1

(greek letters

denote noise

terms)

noisev

pnoiseMxy

t

t

tt

01


41

Example: Constant Acceleration (1D Points)




42


• State vector: position p, velocity v, and acceleration a.




1

11

1

2

21

11

)(

)()(

tt

ttt

tttt

aa

atvv

atvtpp

t

t

t

t

a

v

p

x

noise

a

v

p

t

tt

noisexDx

t

t

t

ttt

1

1

1

2

1

100

10

1

(greek letters

denote noise

terms)

noise

a

v

p

noiseMxy

t

t

t

tt

001


43


• State vector: position p, velocity v, and acceleration a.




1

11

1

2

21

11

)(

)()(

tt

ttt

tttt

aa

atvv

atvtpp

t

t

t

t

a

v

p

x

noise

a

v

p

t

tt

noisexDx

t

t

t

ttt

1

1

1

2

21

1

100

10

1

(greek letters

denote noise

terms)

noise

a

v

p

noiseMxy

t

t

t

tt

001


44

Recap: General Motion Models

• Assuming we have differential equations for the motion E.g. for (undampened) periodic motion of a linear spring

• Substitute variables to transform this into linear system

• Then we have



2

2

d pp

dt

1p p2

dpp

dt

2

3 2

d pp

dt

1,

2,

3,

t

t t

t

p

x p

p

001

10

12

21

t

tt

Dt

1,1,3

1,31,2,2

1,3

2

21

1,21,1,1

)(

)()(

tt

ttt

tttt

pp

ptpp

ptptpp

45










Limitations



46

The Kalman Filter

• Kalman filter Method for tracking linear dynamical models in Gaussian noise

• The predicted/corrected state distributions are Gaussian You only need to maintain the mean and covariance.

The calculations are easy (all the integrals can be done in closed form).




47

The Kalman Filter



Know corrected state from

previous time step, and all

measurements up to the

current one

Predict distribution over

next state.

Time advances: t++

Time update

(“Predict”)

Measurement update

(“Correct”)

Receive measurement

10 ,, tt yyXP

tt ,

Mean and std. dev.

of predicted state:

tt yyXP ,,0

tt ,

Mean and std. dev.

of corrected state:

Know prediction of state,

and next measurement

Update distribution over

current state.


48

Kalman Filter for 1D State

• Want to represent and update



2

10 )(,,,

tttt NyyxP

2

0 )(,,, tttt NyyxP

49

Propagation of Gaussian densities



Shifting the mean

Increasing the variance Bayesian update

50

1D Kalman Filter: Prediction

• Have linear dynamic model defining predicted state

evolution, with noise

• Want to estimate predicted distribution for next state

• Update the mean:

• Update the variance:



1tt d

2

10 )(,,,

tttt NyyXP

2

1,~ dtt dxNX

2

1

22 )()(

tdt d

for derivations,

see F&P Chapter 17.3


51

1D Kalman Filter: Correction

• Have linear model defining the mapping of state to

measurements:

• Want to estimate corrected distribution given latest

measurement:

• Update the mean:

• Update the variance:



2,~ mtt mxNY

2

0 )(,,, tttt NyyXP

222

22

)(

)(

tm

ttmtt

m

my

222

222

)(

)()(

tm

tmt

m

Slide credit: Kristen Grauman Derivations: F&P Chapter 17.3

52

Prediction vs. Correction

• What if there is no prediction uncertainty

• What if there is no measurement uncertainty



tt 0)( 2

t

222

22

)(

)(

tm

ttmtt

m

my

222

222

)(

)()(

tm

tmt

m

?)0( m

?)0(

t

m

ytt 0)( 2

t

The measurement is ignored!

The prediction is ignored!


53

Recall: Constant Velocity Example



State is 2D: position + velocity

Measurement is 1D: position

measurements

state

time

positio

n


54

Constant Velocity Model



o state

x measurement

* predicted mean estimate

+ corrected mean estimate

bars: variance estimates

before and after

measurements


55




o state

x measurement




before and after

measurements


56




o state

x measurement




before and after

measurements


57




o state

x measurement




before and after

measurements


58

Kalman Filter: General Case (>1dim)



PREDICT CORRECT

1ttt xDx

td

T

tttt DD

1 tttttt xMyKxx

tttt MKI

1 tm

T

ttt

T

ttt MMMK

More weight on residual

when measurement error

covariance approaches 0.

Less weight on residual as a

priori estimate error

covariance approaches 0.

“residual”

for derivations,

see F&P Chapter 17.3


“Kalman gain”

59

Summary: Kalman Filter

• Pros: Gaussian densities everywhere

Simple updates, compact and efficient

Very established method, very well understood

• Cons: Unimodal distribution, only single hypothesis

Restricted class of motions defined by linear model



Slide adapted from Svetlana Lazebnik

60

Why Is This A Restriction?

• Many interesting cases don’t have linear dynamics E.g. pedestrians walking

E.g. a ball bouncing



61

Ball Example: What Goes Wrong Here?

• Assuming constant acceleration model

• Prediction is too far

from true position to

compensate…

• Possible solution: Keep multiple different

motion models in parallel

I.e. would check for

bouncing at each time step



Prediction t1

Prediction t3

Prediction t2

Prediction t4

Prediction t5

Correct

prediction

62

References and Further Reading

• A very good introduction to tracking with linear dynamic

models and Kalman filters can be found in Chapter 17 of

D. Forsyth, J. Ponce,

Computer Vision – A Modern Approach.

Prentice Hall, 2003



Computer Vision 2 Lecture 5...Compute an updated estimate of the state from prediction and...

Documents

Transcript of Computer Vision 2 Lecture 5...Compute an updated estimate of the state from prediction and...