Intel Confidential1

Perceptual Computing

– Pour l’instant destiné aux développeurs– Camera Creative Senz3D disponible au prochain

trimestre– Intégration dans des machines Intel prévue pour

le second semestre 2014 !

• Perceptual Computing SDK– C++– Passerelles C#, Unity, Processing, open

frameworks…

Intel Confidential2

Speech Recognition:  Voice command and control, short sentence dictation, and text to speech synthesis

SDK Usage Modes Today1

1 New usage modes may be added in the future

Close-range Depth Tracking (6 in. to 3 ft.):  Recognize the positions of each of the user’s hands, fingers, static hand poses and moving hand gestures.Facial Analysis:  

Face detection and recognition (six and seven point landmark and attribution detection, including smiles, blinks, and age groups)

Augmented Reality:  Combine real-time images from the camera and close-range tracking from the depth sensor with 2D or 3D graphical images.

Intel Confidential3

Close-Range Finger Tracking• Geometric Node Tracking

– 7-point tracking: fingertips, palm center and elbow.– Special usability points: fingertip and grabbing points.– Estimate positions, volumes, openness and

handedness.

• Standardized Poses/Gestures– Poses: thumb up/down and peace– Gestures: swipe left/right/up/down, circle, and wave

• Label map– Hand image and its parameters.

Intel Confidential4

Face Tracking and Analysis• Multiple face detection/tracking• Landmark detection

– 6/7-point detection including eyes, nose, and mouth

• Facial attribute detection– Age-group including baby/youth/adult/senior– Gender detection– Smile/blink detection

• Face recognition– Similarity among a set of faces

Intel Confidential5

• Nuance* Voice Command and Control– Recognize within a list of predefined

commands.

• Nuance Voice Dictation– Recognize short sentences (<30 seconds).

• Nuance Voice Synthesis– Text to speech for short sentences

Voice Recognition and Synthesis

Intel Confidential6

• Track any 2D planer surfaces

– Report position, orientation andother parameters

• Track limited 3D objects

– Based on 3D models

• Track faces orientation

Augmented Reality

Intel Confidential7

SDK Usage H/W RequirementsSDK Usage Mode

Speech Certified Dual-

Array Microphones

RGB Webcam

Creative* Camera

Close-range Depth tracking

X

Speech Recognition

X X

Facial Tracking

X X

Augmented Reality

X X

Intel Confidential8

PXCSession, PXCImage, PXCAudio, PXCCapture, PXCGesture, PXCFaceAnalysis, PXCVoice

UtilCapture, UtilPipeline

C#PXCMSessionPXCMImagePXCMAudioPXCMCapturePXCMGesturePXCMFaceAnalysisPXCMVoiceUtilMCaptureUtilMPipeline

pxcupipeline

Unity* Pro Processing openFrame

works*

Applications

Core Functionalities

Module Interaction

Additional Language and Framework Support

SDK API Hierarchy

*Other brands and trademarks may be claimed as the property of their respective owners

Intel Confidential9

• Color streams RGB24 640x480 25fps, 30fps RGB24 640x360 25fps, 30fps RGB24 1280x720 25fps, 30fps

• Depth streams (16-bit integer, 0-32000) 320x240 25fps, 30fps, 50fps, 60fps UVMAP (Depth Color) Confidence Map (16-bit integer)

• Vertices streams (real world coordinates in 3D fixed-point integers)

• Audio streams (At least 2-array MIC) 44.1KHz mono/stereo 48KHz mono/stereo

Camera Streams

Intel Confidential10

PXCImage: Depth & Color

IMAGE_TYPE_COLOR

(of any color format)• planes[0]: color

data• Planes[1]: empty• planes[2]: empty• planes[3]: empty

IMAGE_FORMAT_DEPTH

• planes[0]: depthmap• Planes[1]: confidencemap• planes[2]: uvmap• planes[3]: empty

2-byte per pixel

8-byte per pixel

IMAGE_FORMAT_VERTICES

• planes[0]: vertex (x,y,z)• Planes[1]: confidencemap• planes[2]: uvmap• planes[3]: empty

12-byte per pixel

One real depth stream, two virtual

Intel Confidential11

Programming Language and Framework Support• C++, C#, Java• Supported Frameworks

– processing– openFrameworks– Unity– Havok– Total Immersion AR

Intel Confidential12

Declare The SDK Object

• Unityprivate PXCUPipeline pipeline;

pipeline = new PXCUPipeline();

Intel Confidential13

Select Features and Initialize

• Select Features using PXCUPipeline.Mode enum• Use Bitwise OR (|) for Multiple Features• Use Init() To Set Features and Enable SDK

Access

pipeline.Init(PXCUPipeline.Mode.COLOR_VGA|

PXCUPipeline.Mode.DEPTH_QVGA|

PXCUPipeline.Mode.GESTURE);

Intel Confidential14

Capture A Frame

• Poll For A Frame Using AcquireFrame(bool);– Can be blocking or non-blocking– AcquireFrame(true) is blocking,

AcquireFrame(false) is non-blocking

• Returns true If A Frame Is Availableif(pipeline.AcquireFrame(false))

{

}

Intel Confidential15

Retrieve Image Data

• Data Is Retrieved via Query*(<T>)– QueryRGB(), QueryLabelMapAsImage(), etc…

• UnityTexture2D rgbTexture = new Texture2D(640,480,TextureFormat.ARGB32, false);

pipeline.QueryRGB(rgbTexture);

Intel Confidential16

Retrieve Geonodes Data

• Data Is Retrieved via QueryGeoNode()– Use PXCGesture::GeoNode::Label and

PXCGesture::GeoNode to retrieve data

• UnityPXCMGesture.GeoNode node;

pipeline.QueryGeoNode(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, out node);

Intel Confidential17

Retrieve Gesture Data

• Data Is Retrieved via QueryGesture()– Use PXCGesture::GeoNode::Label,

PXCGesture::Gesture::Label and PXCGesture::Gesture to retrieve data

• UnityPXCMGesture.Gesture gest;

pipeline.QueryGesture(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, out gest);

if(gest.label==PXCMGesture.Gesture.Label.LABEL_HAND_WAVE)…

Intel Confidential18

Clean Up

• Use ReleaseFrame() To “Free Up The Pipeline”pipeline.ReleaseFrame();

Intel Confidential19

What next ?

http://perchackparis.eventbrite.fr/https://perceptualchallenge.intel.com/