Gaze interaction (2): models and boccignone/GiuseppeBoccignone...آ  •2. may require high...

download Gaze interaction (2): models and boccignone/GiuseppeBoccignone...آ  •2. may require high contrast

of 37

  • date post

    21-May-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of Gaze interaction (2): models and boccignone/GiuseppeBoccignone...آ  •2. may require high...

  • Gaze interaction (2):

    models and technologies

    Corso di Interazione uomo-macchina II

    Prof. Giuseppe Boccignone

    Dipartimento di Scienze dell’Informazione

    Università di Milano

    boccignone@dsi.unimi.it http://homes.dsi.unimi.it/~boccignone/l

    A. Vinciarelli, M. Pantic, H. Bourlard, Social Signal Processing: Survey of an Emerging Domain, Image and Vision Computing (2008)

    Gaze interaction

  • Gaze estimation without eye trackers

    • Problem!

    • Eye detection

    • detect the existence of eyes

    • accurately interpret eye positions in the images

    • using the pupil or iris center.

    • for video images, the detected eyes are tracked from frame to frame.

    • Gaze estimation : detected eyes in the images used to estimate and track

    where a person is looking in 3D, or alternatively, determining the 3D line of

    sight.

    Gaze estimation without eye trackers

  • Eye detection

    //eye models

    • Identify a model of the eye which is sufficiently expressive to take account of

    large variability in the appearance and dynamics, while also sufficiently

    constrained to be computationally efficient

    • Even for the same subject, a relatively small variation in viewing angles can

    cause significant changes in appearance

    Eyelids may appear straight from one

    view but highly curved from another.

    The iris contour also changes with

    viewing angle.

    The dashed lines indicate when the

    eyelids appear straight

    the solid yellow lines represent the

    major axis of the iris ellipse

    Eye detection

    //eye models

    • The eye image may be characterized by

    • the intensity distribution of the pupil(s), iris, and cornea,

    • their shapes.

    • Ethnicity, viewing angle, head pose, color, texture, light conditions, the

    position of the iris within the eye socket, and the state of the eye (i.e., open/

    close) are issues that heavily influence the appearance of the eye.

    • The intended application and available image data lead to different prior eye

    models.

    • The prior model representation is often applied at different positions,

    orientations, and scales to reject false candidates

  • Eye detection

    //eye models

    • Shape-based methods: use a prior model of eye shape and surrounding

    structures

    • fixed shape

    • deformable shape

    • Appearance-based methods: rely on models built directly on the appearance

    of the eye region: template matching by constructing an image patch model

    and performing eye detection through model matching using a similarity

    measure

    • intensity-based methods

    • subspace-based methods

    • Hybrid methods: combine feature, shape, and appearance approaches to

    exploit their respective benefits

    Eye detection

    //eye models: Shape-Based Approaches

    • Shape-based methods: use a prior model of eye shape and and a similarity

    measure

    • Prior model of eye shape and surrounding structures

    • iris and pupil contours and the exterior shape of the eye (eyelids)

    • simple elliptical or of a more complex nature

    • parameters of the geometric model define the allowable template deformations

    and contain parameters for rigid (similarity) transformations and parameters for

    nonrigid template deformations

    • ability to handle shape, scale, and rotation changes

  • Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

    • example: Valenti and Gevers

    • uses isophote (i.e., curves connecting points of equal intensity) properties to infer the

    center of (semi)circular patterns which represent the eyes

    Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

  • Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

    Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

    • example: Webcam-based Visual Gaze Estimation (Valenti et al)

    • uses isophote (i.e., curves connecting points of equal intensity) no head pose

    voting

    Direction to the

    center

  • Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

    • example: Webcam-based Visual Gaze Estimation (Valenti et al)

    • uses isophote (i.e., curves connecting points of equal intensity) no head pose

    Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

    • example: Webcam-based Visual Gaze Estimation (Valenti et al)

    • uses isophote (i.e., curves connecting points of equal intensity) no head pose

  • Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

    • example: Webcam-based Visual Gaze Estimation (Valenti et al)

    • uses isophote (i.e., curves connecting points of equal intensity) no head pose

    Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

    • example: Webcam-based Visual Gaze Estimation (Valenti et al)

    • uses scale space framework for multiresolution

  • Eye detection

    //eye models: Shape-Based Approaches

    • Simple Elliptical Shape Models:

    • example: Webcam-based Visual Gaze Estimation (Valenti et al)

    • simple interpolants for easy calibration

    Eye detection

    //eye models: Shape-Based Approaches

    • Complex Shape Models:

    • example: Yuille deformable templates

  • Eye detection

    //eye models: Shape-Based Approaches

    • Complex Shape Models:

    • example: Yuille deformable templates

    Eye detection

    //eye models: Shape-Based Approaches

    • Complex Shape Models:

    • example: Yuille deformable templates

  • Eye detection

    //eye models: Shape-Based Approaches

    • Complex Shape Models:

    • example: Yuille deformable templates

    Eye detection

    //eye models: Shape-Based Approaches

    • Complex Shape Models:

    • 1. computationally demanding,

    • 2. may require high contrast images, and

    • 3. usually need to be initialized close to the eye for successful localization. For

    large head movements, they consequently need other methods to provide agood

    initialization

  • Eye detection

    //eye models: Feature-Based Shape Methods

    • Explore the characteristics of the human eye to identify a set of distinctive

    features around the eyes.

    • The limbus, pupil (dark/bright pupil images), and cornea reflections are

    common features used for eye localization

    • Local Features by Intensity

    • The eye region contains several boundaries that may bedetected by gray-level

    differences

    • Local Feature by Filter Responses

    • Filter responses enhance particular characteristics in the image while suppressing

    others. A filter bank may therefore enhance desired features of the image and, if

    appropriately defined, deemphasize irrelevant features

    Eye detection

    //eye models: Feature-Based Shape Methods

    • Local Features by Intensity

    • The eye region contains several boundaries that may be detected by gray-level

    differences

  • Eye detection

    //eye models: Feature-Based Shape Methods

    • Local Features by Intensity

    • The eye region contains several boundaries that may be detected by gray-level

    differences (Harper et al.)

    Eye detection

    //eye models: Feature-Based Shape Methods

    • Local Features by Intensity

    • The eye region contains several boundaries that may be detected by gray-level

    differences

    Sequential search strategy

  • Eye detection

    //eye models: Feature-Based Shape Methods

    • Local Features by Intensity

    • The eye region contains several boundaries that may be detected by gray-level

    differences

    Eye detection

    //eye models: Feature-Based Shape Methods

    • Local Features by Intensity

    • The eye region contains several boundaries that may be detected by gray-level

    differences

  • Eye detection

    //eye models: Feature-Based Shape Methods

    • Local Feature by Filter Responses

    • Filter responses enhance particular characteristics in the image while suppressing

    others

    • Example Sirohey and Rosenfeld:

    • Edges of the eye’s sclera are detected with four Gabor wavelets. A nonlinear filter is

    constructed to detect the left and right eye corner candidates.

    • The eye corners are used to determine eye regions for further analysis. Postprocessing

    steps are employed to eliminate the spurious eye corner candidates.

    • A voting method is used to locate the edge of the iris. Since the upper part of the iris may

    not be visible, the votes are accumulated by summing edge pixels in a U-shaped annular

    region. The annul