1

La photographienumérique

Frank NIELSENLundi 7 Juin 2010

Le Monde digitalLe Monde digitalKey benefits of the analog2digital paradigm shift?

• Dissociate contents from support : “binarize”• Universal playerUniversal player (CPU, Turing machine)• Generic algorithms

Le Monde numériqueMonde numérique omnipresent, informatique ubiquitaire

Nouveautés

Numérique = Digital + Calcul

Exemple:Image numérique

(=calculée)

Photographie numérique : Une profonde Photographie numérique : Une profonde révolution a venir

??Photography Computational Photography

Pas seulement dans le domaine grand public… … mais aussi dans beaucoup d’autres domaines des sciences

Computational anatomy(differential geometry)

Example 1Non Photo Realistic Camera

Introduction: What’s computational photography?Introduction: What’s computational photography?

??

Warm up: Warm up: Nonphotorealistic camera (NPR camera) Nonphotorealistic camera (NPR camera) Multiple flashes to easily get depth discontinuities

Baseline 50mm (depth 5mm at 2 meters)

Canny edge detector

(for comparison only)

Multiflash depth edgeStylized rendering

Source image

Shadow to the leftShadow to the right

Remove shadows, show lesionfrom depth discontinuity analysis

Warm up: Nonphotorealistic (NPR) cameraWarm up: Nonphotorealistic (NPR) cameraLaparoscope camera with two fiber optics lightingtwo fiber optics lighting

Difficult to analyze using traditional image processing techniques

A Non-Photorealistic Camera: Depth Edge Detection and Stylized Rendering with Multi-Flash Imaging.SIGGRAPH 2004 http://www.merl.com/people/raskar/NprCamera/

Example 2Synthetic Aperture Focusing Camera

Introduction: What’s computational Introduction: What’s computational photography?photography?

??

Camera array provides many individual apertures → synthetic aperture focusing

Aperture means beyond pinhole camera algorithms

Warm up: Synthetic aperture focusing camera Warm up: Synthetic aperture focusing camera (SAF)(SAF)

High Performance Imaging Using Large Camera Arrays. SIGGRAPH 2005http://graphics.stanford.edu/projects/array/

focal length

sensorsize ap

ertu

reap

ertu

re

lens( SAF camera: 1-shot many images!)

Warm up: Synthetic aperture focusing Warm up: Synthetic aperture focusing cameracamera(SAF)(SAF)

Synthetic aperture focusing

+Averaging multiple images also improve Signal-to-Noise ratio (SNR)

SAF is good enough for image recognition

Σ

Single camera aperture

Synthetic (=calcul) aperture

3D World Sensor planeCamera array

Example 3Shape-Time Camera

(Depict the world)

Introduction: What’s computational photography?Introduction: What’s computational photography?

??

Depict world in new ways: Shape-time photography

(burst-mode on stereo adaptor)

Warm up: Depicting the worldWarm up: Depicting the world

Shape-time photography. CVPR 2003people.csail.mit.edu/billf/

Picasso Hockney

Stereo mount Depiction

Matte extraction: strobing application

Old film of Etienne-Jules Marey

Visualizing motion is important for video-based applications (PVR,etc.)

Mosaicing+matting provides a kinetic experience

Comp. Photography: Novel World DepictionsComp. Photography: Novel World Depictions

Computer generated motion lines

Comp. Photography: Novel World DepictionsComp. Photography: Novel World Depictions

Computational Photography: Motion amplificationComputational Photography: Motion amplification

Motion magnification, SIGGRAPH 2005http://people.csail.mit.edu/celiu/motionmag/motionmag.html

A video example best described the result(Applications to telesurveillance, etc.)

Computational Photography: Motion amplificationComputational Photography: Motion amplification

Motion magnification, SIGGRAPH 2005http://people.csail.mit.edu/celiu/motionmag/motionmag.html

Inpainting… Texture Synthesis…

Hallucination….

Computational PhotographyComputational Photography

Region filling and object removal by exemplar-based inpainting. IEEE Trans. Image Process. 2004http://research.microsoft.com/~antcrim/papers.htm

Computational Photography: Computational Photography: ClickRemovalClickRemoval applet applet

Frank Nielsen, Richard Nock: ClickRemoval: interactive pinpoint image object removal. ACM Multimedia 2005:

Demo

Image retargettingImage retargettingAdjust contents to screen size(TV, PDA, Phone, etc.)

SIGGRAPH 2007

Demo

Human Perceptionversus

Digital Image Processing

Computational Photography: Human PerceptionComputational Photography: Human Perception

S/W Computational Photo.: Hybrid imagesS/W Computational Photo.: Hybrid images

Hybrid images, SIGGRAPH 2006.

Low frequencyat far distance

High frequencyat close distance

Overriding Dynamic range… Tone mappingOverriding Dynamic range… Tone mappingScientific (measurement) images ≠ Human perceptual images

Image segmentation and lightness perception, Nature 434, 79-83, 2005

‘light disks’ visible through dark haze

‘dark disks’ visible through light haze

Disks are exactly identicalbut are perceived differently

Computational PhotographyNovel hardware & processing techniques

Computational Photography: H/WComputational Photography: H/W

Coaxial Infrared camera + ProjectorTranscoding (pseudo-coloring)Coaxial Infrared camera + ProjectorTranscoding (pseudo-coloring)

VeinViewer (Luminetx) | http://www.luminetx.com/main/default.aspx

Computational Photography: Vein ViewerComputational Photography: Vein Viewer

Computational Photography:Computational Photography:

Computing inOptical Domain

H/W Comp. Photo.: Computing in Optical domainH/W Comp. Photo.: Computing in Optical domain

Control the rays in space-time:Exposure allows optical computations→Light integration on the sensor

Programmable imaging using a digital micromirror array (CVPR’04)Programmable Imaging: Towards a Flexible Camera, Int. Journal of Computer Vision. 2006

H/W Comp. Photo.: Computing in Optical domainH/W Comp. Photo.: Computing in Optical domain

Programmable imaging using a digital micromirror array (CVPR’04)Programmable Imaging: Towards a Flexible Camera, Int. Journal of Computer Vision. 2006

Require to calibrate the DMDwith the camera coarsely

Convolution in optical domain Convolution in optical domainfor face recognition

Computational Photography:Computational Photography:

Computing withexotic “lenses”

Control the light rays on each layer:

Multiple-layer aperture

Computational Photo.: Lensless CameraComputational Photo.: Lensless Camera

Lensless Imaging with a Controllable Aperture, CVPR 2006

Traditional New

Computational Photo.: Lensless CameraComputational Photo.: Lensless Camera

Lensless Imaging with a Controllable Aperture, CVPR 2006

Pan/tilt field of view (fov) without physical moving parts

Split field of view, spatially varying zoom

Computations in optical domain

Computational Photo.: Lensless CameraComputational Photo.: Lensless Camera

Lensless Imaging with a Controllable Aperture, CVPR 2006

Appearances of eyes captures both the environment and gazing direction

Spherical panorama(latitude-longitude)

Corneal Imaging System Environment from Eyes, Int. Journal on Computer Vision (IJCV) 2006.Eyes for relighting, SIGGRAPH 2004.

Computational Photography: Eye OpticsComputational Photography: Eye Optics

Comp. Photography: Radial Catadioptric CameraComp. Photography: Radial Catadioptric Camera

Multiview Radial Catadioptric Imaging for Scene Capture SIGGRAPH 2006

• Capture a radial space of rays• Both mirrored and object parts• 3D reconstruction with BRDF(using a single shot!)

Computational Photography:Computational Photography:

Beyond 2D pixels:4D+ Light fields

The Plenoptic Function and the Elements of Early Vision 1991Plenoptic Modeling: An Image-Based Rendering System, SIGGRAPH 1995

Computational Photography: Plenoptic cameraComputational Photography: Plenoptic camera

Plenoptic (latin plenus+optics) is a 7D function (X,Y,Z,θ,φ,λ,t)

16 MP: 300x300 lens images

Moving the viewpoint

Digital refocusing

Acquire first, postprocess later.

Computational Photography: Light field cameraComputational Photography: Light field camera

Fourier Slice Photography, SIGGRAPH 2006

H/WComp. Photography: Light field cameraH/WComp. Photography: Light field camera

Fourier Slice Photography, SIGGRAPH 2006

Fourier Slice Photography

Lens Sensor Image

Generalizedoptics

Computationalsensor

Computational imaging

Display

Novel displays

Computational Photography: Computational Photography: Images in the 21Images in the 21stst Century Century

Image numérique

= calcul