Reality Cues-Based Interaction UsingWhole-Body Awareness

Anderson MacielUniversidade de Caxias do Sul

Rod RS 122, km 69 sn91501-970 Caxias do Sul,

Brazil+55 54 3289.9009

amaciel5@ucs.br

Luciana P. NedelInstituto de Informática,

UFRGSCaixa Postal 15064

91501-970 Porto Alegre, Brazil+55 51 3308.7037

nedel@inf.ufrgs.br

Vitor A.M. JorgeInstituto de Informática,

UFRGSCaixa Postal 15064

91501-970 Porto Alegre, Brazil+55 51 3308.6168

vamjorge@inf.ufrgs.brJuan M.T. Ibiapina

Instituto de Informática,UFRGS

Caixa Postal 1506491501-970 Porto Alegre, Brazil

+55 51 3308.6168jmtibiapina@inf.ufrgs.br

Luis F.M.S. SilvaInstituto de Informática,

UFRGSCaixa Postal 15064

91501-970 Porto Alegre, Brazil+55 51 3308.6168

luisfmssilva@inf.ufrgs.br

ABSTRACTThe exploration of 3D environments using 6 degrees-of-free-dom interaction is still a challenge since users easily becomedisoriented. In this paper we discuss the benefits of thewhole-body awareness in 3D interactive applications. Wepropose a technique for navigation and selection in 3D en-vironments which explores the peephole metaphor with atablet PC.In practice, the tablet is held by the participantwho moves it around and points it in any direction for vi-sualization and interaction. The method was tested witha set of users who were asked to perform selection tasks.The technique presented competitive results when comparedwith conventional interaction methods and also showed thatreal world body orientation memory helps users to performbetter in the virtual world.

Categories and Subject DescriptorsH.5.2 [Information Interfaces and Presentation]: UserInterfaces; H.1.2 [Models and Principles]: User/MachineSystems—human factors

1. INTRODUCTIONCommon interfaces for traditional 2D environments make

extensive use of visual cues to help with user orientation.Although visual cues seem to be best suited for 2D inter-action, the new 3D environments used in computer games,virtual world experiences, scientific visualization, and so onmight require a less conventional choice. Even though it is

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SAC’10 March 22-26, 2010, Sierre, Switzerland.Copyright 2010 ACM 978-1-60558-638-0/10/03 ...$10.00.

relatively easy to construct a new interface, the fact is thatmouse and keyboard are still the first option for computerinteraction.

Computers are changing fast. They are becoming smallerand getting increased mobility. Powerful graphics cards areavailable even in mobile phones, and some very interest-ing new applications using virtual reality and its variantsarise. However, user interfaces are not pursuing this evolu-tion of computers, especially when 3D information is avail-able. When exploring a 3D virtual environment with amouse and keyboard, users easily become disoriented. Jacobet al. [2] remind us that we are not guided purely by visualcues when moving in the real world. We are also guided bysome understanding of the surrounding environment, of ourbody and the presence of other people, as well as by somesense about physics. The labyrinth provides the informationabout up, down and balance. Our sense of touch makes usaware of obstacles when we are in direct contact, even whenwe cannot see them. To finish, we dispose of a sense of po-sition and orientation which makes us know, all the time,where our limbs and other body parts are, which is calledproprioception.

However, when the focus passes from the real world intoa virtual world, and one starts to interact with a virtualenvironment using conventional interfaces, all corporal cuesvanish. This often reduces body sensation and contributesfor disorientation.

In the present work we explore human orientation capabil-ities without relying only on the sense of vision. We proposeto do so using the memory of history one has about one’sbody postures while moving in the real world. We do thatby implementing the peephole metaphor using a tablet PCas a window to the virtual world, artificially enlarging thedisplay and interaction area. The tablet is held by the par-ticipant, who moves it around and point it in any directionfor visualization and interaction. Depending on user ori-entation and position, the display shows different portionsof the environment and the system allows interaction withobjects in that specific portion.

1224

2. RELATED WORKWhen researchers first introduced techniques for naviga-

tion in virtual reality (VR) they naturally proposed the cre-ation of virtual navigation tools analog to real ones. Thisincludes the use of maps, compasses, etc. Darken and Sib-ert [1] created a toolset based on this principle and studiedhow their tools influence the navigation behavior of somesubjects. One of their conclusions is that people need cues,e.g. visual and audible information, which can be combinedto make targets easier to find.

Experimental results reported by Wartenberg et al. [7]indicate that whole-body movement information - vestibularand proprioceptive signals, motor efferent commands - aredecisive in grabbing correct directional information whilenavigating through the real space. They showed that allsenses contribute to spatial orientation.

The implications of body awareness are being more andmore explored by researchers in VR, user interfaces and hu-man factors. Several authors point out the importance ofthis feature for the design of new interfaces. The notionof Reality-Based Interaction (RBI), which is a frameworkcombining intuitive physics, body awareness, environmentawareness, social awareness and all related skills, has beenintroduced by Jacob et al. [2]. Such new concept ties to-gether a large group of new interaction styles. A number ofexisting applications, as tangible user interfaces, electronictourist guides and the Apple iPhone itself, partially exploitthis idea.

While previous research works explore body awareness inspecific situations, either limited to two dimensions or to areduced number of DOFs, in this work we propose a gen-eral interaction case in 3D with 6 DOFs. Similar to ourapproach, the Boom Chameleon [5] system provides a one-to-one mapping between real and virtual space. However,while the Boom Chameleon is mounted on a cumbersomestructure which limits motion, the tablet PC is fully mobileand wireless.

More recently, mobile phones enabled with spatial aware-ness, using markers and also by natural pose tracking weredeveloped by Wagner et al. [6]. The same is happeningwith handhelds, now capable of providing Augmented Re-ality (AR), like the LightSense [3] which is enabled withspatial awareness or more recent works of Olwal and Anders[4]. These light weight devices are enabled with novel inter-action techniques that take advantage of aditional humansenses.

Although most researchers agree with the statement thatbetter results are obtained when the user makes use of allsenses in virtual environments, the implication of real worldperception in the orientation inside a virtual environment isstill not fully evaluated. Further, to our knowledge, spatialmemory has not been explored as a tool for orientation yet.

3. DESIGN AND IMPLEMENTATIONIn this work we propose a peephole implementation where

a tablet PC is used as a window to a virtual world. Throughthe tablet display, it is possible to inspect and interact withina virtual world. When the user walks or turns around, theposition and orientation of his/her head are tracked usingan optical system. As a consequence, images exhibited inthe tablet display change accordingly.

In order to obtain spatial awareness, we are using Bra-

Track, a precise and low-cost marker-based commercial op-tical tracking system. The system is composed by a set ofoff-the-shelf USB cameras (two in this setup) customizedwith electronic boards that provide flash strobes using ahuge number of infrared LEDs. Figure 2 shows the Bra-Track system in use. Trackable artifacts are built using setsof reflective spheres. Synchronized images are acquired bythe camera modules at 60 Hz. A 2D pre-processing identifiesinfrared artifacts in each camera, and a 3D reconstructionmodule finds the 3D position and orientation of each arti-fact that is visible in more than one camera. In case thereis occlusion, the system cannot provide accurate tracking.

Figure 1: Overview of the system setup.

Figure 2: The BraTrack system tracks the reflectiveartifact on the hat the user is wearing by camerason the upper right of the picture.

Such occlusion limitation of the device is the main reasonwhy we decided to attach the artifact to the head insteadof the tablet PC, as many projects in handheld augmentedreality do.

To track the user’s head and defining a pose for the vir-tual camera, an artifact was attached to a hat, as shown inFigure 2. In such a way, the head movement controls thevirtual camera position and orientation in space. The mo-tion tracked by BraTrack is sent to the tablet PC through awireless connection (Wifi network) between BraTrack serverand the tablet, as shown in Figure 1.

1225

Regarding the display, we have chosen to use a tablet PC,because changing it into a peephole device is straightfor-ward. Additionally, the display size is large enough to beused as a window to a virtual world. To do that, we adoptedthe fixed-cursor design. With this design, the person sees thevirtual environment through the tablet display with a fixedcrosshair positioned at the center of the screen. Thus, mov-ing the device means aiming, and objects placed under thecrosshair can be selected with a click. Since both hands wereused to hold the device, we needed some strategy to allow theuser to point and click objects without the stylus, keyboardor mouse. Due to a limitation of the tablet PC used (anHP Compaq 2710p Notebook PC), we attached two mousebuttons on the left of the tablet and the mouse wheel on theright to chose among menu options (see Figure 2b).

4. EXPERIMENT SETUPAn experiment was designed with the purpose of under-

standing the possible advantages of an interaction techniquecoherent with the real space. More specifically, with the pur-pose of testing to which extent the use of body mnemonicsand intuitiveness affects the user performance in interac-tive tasks. The experiment was performed by a group of10 subjects, 9 male and 1 female. Participants were under-graduate and graduate students in Computer Science. Also,they were all experienced computer and portable devicesusers, but only few of them had previous experience withvirtual reality. Before the experiment, users were instructedon how to operate the device and encouraged to perform asmany practice trials as needed so that they could feel confi-dent when operating the device during the actual experimentconditions.

Each trial consisted of selecting objects in a 3D environ-ment using either of the interaction techniques provided:mouse and keyboard; peephole interface. To avoid thattest order influences the results, users were divided into twogroups, each performing the test with one of the interfacesfirstly and with the other interface secondly. Visual cuessuch as virtual floor and walls were removed from the vir-tual environment of both the tests to force users to rely onlyon the camera movement to execute the tasks in the 3Dworld. This is expected to remove any bias and make theresults more reliable. Moreover, each user repeated the testthree times with each interaction method.

The interaction tasks were proposed in the form of a game.Using a game to evaluate interaction techniques has the ad-vantages of proposing a clear goal to the user and a score forus to compare and analyse. The game and rules are detailedin Section 4.1 below. Later in Section 4.2, we establish thehypotheses of the experiment. In Section 4.3 we explainedhow qualitative data was collected.

4.1 Pair-Matching Memory GameThe game environment is a 3D room with the same dimen-

sions of the physical experiment room. Eight question marks”?” are randomly distributed in the space in such a way thatthey are arranged all around the user. Each mark repre-sents the position of one hidden geometrical object: cubes,spheres and tori of different colors. To accomplish the exper-iment, the participant has to navigate through the virtualenvironment and correctly match four pairs of objects, justas in a classic memory game. See Figure 3 for an explicativescreenshots sequence.

Some dependent variables we logged for posterior analysisare: the time and the number of clicks to complete the ex-periment; the complete path followed to complete the task;the subjective impressions of the volunteers about the ex-perience of using the peephole display compared with themouse and keyboard.

Figure 3: A sequence of screen shots from the gameapplication. Question marks are shown in (a), theuser selects a question mark that turns into a cube in(b) and then they select a torus in (c), which is nota match; notice the red ”X”. In (d), when the userfinds the two cubes they score a match, highlightedby the green ”V” sign.

4.2 HypothesesThe purpose of the game described in Section 4.1 above is

to analyse the role played by body mnemonics on the userperformance while performing interactive tasks. In this con-text, we formulated one main hypothesis and one secondaryhypothesis.

The main hypothesis is that it will be easier for the usersto locate themselves inside a virtual 3D world using thepeephole interface than using the mouse and keyboard. Thereason to believe in the confirmation of this hypothesis re-lies on the application of the proprioception concept. Thesecondary hypothesis is that the weight of the tablet PCwill negatively affects the user performance when using thepeephole interface. This hypothesis, in fact, relates to a lim-itation of our implementation that we suppose will interferein the user performance. The tablet PC is not fully readyto be a lightweight peephole device. Even if they cannot beconsidered heavy, the use for long periods should negativelyaffect the user performance.

4.3 QuestionnaireAt the end of the tests, the subjects answered a question-

naire involving subjective interaction experience and impres-sions about speed, intuitiveness, fun and overall satisfactionwhile comparing the two interaction methods. Based onparticipants’ responses, qualitative data was collected. Eachquestion consisted of selecting the preferred technique usedto perform the task. The options available were: mouse andkeyboard; peephole; none of them. In case one of them was

1226

Table 1: Dependent variables of the analysis

Mouse & Keyboard PeepholeVariable µ Wins Best µ Wins Best

(%) (%)Total time 60.53s 60 32.7s 61.63s 40 31.7sTime 2nd half 32.03s 52 15.8s 34.73s 48 14.3sTime to match 15.01s 51 1.5s 23.02s 49 1.5sTotal clicks 12.4 50 10.0 12.25 30 10.0Clicks 2nd half 8.07 44 5 8.07 40 6Clicks to match 3.97 39 1 4.18 39 1

preferred, participants were asked to specify how much per-formance enhancement they think they obtained using theselected technique in comparison to the other one. Theycould select five different possibilities going from very smallto very high improvement. If none of the techniques waschosen as preferred, the performance enhancement degreewas taken as not applicable.

5. RESULTSWe analyzed the data in the logs and computed Table 1,

which summarizes the variables of interest and their respec-tive values. We are mainly interested in the time to completetasks and the number of clicks required to accomplish them.We compared the data acquired using the mouse-keyboardinterface with the data acquired using the peephole inter-face, and evaluated mean values, number of wins and thebest try for each interface.

Total time is the total time to match 4 pairs and con-clude the game (best try in three). Best try was used in-stead of mean for fairness of comparison as non conventionaldevices present a steeper learning curve then conventionalones. Time of the second half of the clicks corresponds tothe time elapsed when applying only the second half of theclicks, when the player clicks more consciously (consideringthat in the first half of the clicks the player is just browsingrandomly and clicks are mostly blind). Time to match sincea pair is first seen counts the time elapsed since the userfirst see the two members of a pair until it matches the pair.Total clicks is the total number of clicks to match 4 pairsand conclude the game. Clicks of the second half of the timecorresponds to the clicks made only during the second halfof the game time, when the player clicks more consciously.Clicks to match since a pair is first seen counts the clicksspent since the user first see the two members of a pair untilit matches the pair.

In Table 1, µ is the mean. A win is added to the technique(mouse and keyboard or peephole) which took the lesser timeor lesser number of clicks to perform. Then, the percentageof wins does not add up to 100% due to the cases in whichthere is a tie.

Total time and total number of clicks are the values wedecided to analyze first. We observed that the values com-puted for mouse-keyboard came out very similar to the onescomputed for the peephole in both analyzes. That is thereason why we decided to dig deeper in the data and explorthe clicks distribution in time. We analyzed the second halfof the game activity to see if one of the techniques wouldshow better performance there, and they did not. We alsomeasured the period of time and number of clicks it takesfor a player to remember an object they have already seenwhen they first see its match. Again, the results obtained

have proven to be very similar.As the two interaction modes gave so similar results, we

provide in Table 2 the results for a Student’s t-Test to ver-ify the hypothesis of equality of the population means thatunderlies each sample. We used α = 0.05 in the test. Bothvalues for tStat., when compared with tCritical, fall in theinterval, showing that we cannot refute our main hypothesisneither for time nor for number of clicks.

Table 2: Statistical analysis: Student’s t-TestTime Clicks

Mouse Peephole Mouse PeepholeMean 60.53 61.63 12.4 12.25Variance 2254.57 826.27 4.266 3.958t Stat. -0.0624 0.165t Critical one tail 1.734 1.734

Our evaluation also included user’s subjective data col-lected through a questionnaire. Figure 4 brings the meanresults. Most of the users said that the tablet PC interac-tion is clearly more fun than conventional interaction. Mostof them also found it is slightly easier to use as well as toremember objects positions with the peephole interface. Onthe other hand, most of the users reported that handlingthe tablet PC for more than a few minutes is tiresome. De-spite of that, most of them chose the peephole interactionas favorite.

NoPreference

Tablet

More Fun

Mouse

Easier to Remember

Easier to Use

Overall Preference

Ergonomics

BestBest

Figure 4: Subjective analysis. Results extractedfrom a questionnaire show the users’ preferences.

Reviewing the hypotheses proposed in Section 4.2, we con-clude that the first hypothesis - it will be easier for the usersto locate themselves inside a virtual world using the peepholeinterface than using the mouse and keyboard - cannot be con-firmed. The analysis of quantitative data cannot show thatthe peephole interface is more efficient than the mouse andkeyboard, since times and number of clicks measured do notpresent a statistically significant difference. The analysiscannot show that mouse and keyboard interaction is moreefficient neither. Student’s t-Test finally showed they havethe same efficiency for the sampled data. However, subjec-tive data indicates a preference of the users for the peepholeinteraction.

The second hypothesis - the weight of the tablet PC willaffect the performance of the users when using the peepholeinterface - was confirmed by the qualitative data collected.In Figure 4, the label Ergonomics indicates a large preferenceof the users by the mouse and keyboard interaction.

1227

6. DISCUSSION AND CONCLUSIONSThe results obtained are very significative. However, the

number of tests was insufficient to statistically confirm thefirst hypothesis. In face of that, we analyzed the individ-ual performance of each subject in both tasks. We no-ticed that people with game oriented behavior are extremelyskilled with mouse and keyboard, reaching a much higherperformance in this modality comparing to their non-game-oriented fellows. At the same time, we noticed that non-game-oriented people always had their best performance withthe peephole. In addition, their delta between mouse andpeephole is often higher with the peephole than the deltaof the gamers is with the mouse and keyboard. This sug-gests that the subjects profile plays an important role whenit comes to choosing the best interaction technique. Futurework should consider extending the test to a greater numberof subjects to explore this issue.

Another source of data that can help us to better un-derstand the results are the viewer trajectories while inter-acting. We drew 2D graphs representing the trajectoriesprojected on the floor plane, and visually compared them inFigure 5. Observing them, one can see that the mouse tra-jectories were closer to a bunch of straight lines, while theones produced with the tablet PC are smoother and morecomplex. The reason for this is that the movement producedwith the peephole display is fully made in 6 DOFs. With themouse (a 2D device) and keyboard, in order to produce ro-tation and translation in 3D, one needs to compose 1D and2D movements. It is known in computer human interactionthat limiting the number of degrees of freedom makes theinteraction task easier. Surprisingly, times obtained whileperforming in 6 DOF are the same as times obtained forsimple 1D and 2D motions. By observing the trajectorieseven more carefully we can also see that the ones respec-tive to the peephole are longer and more complex than themouse ones. As they are taking basically the same time,we can say that navigation with the tablet PC occurs at ahigher velocity than with the mouse.

Figure 5: Examples of paths followed by two dif-ferent subjects while accomplishing the experimentswith the mouse and keyboard (left column) and withthe peephole display (right). The dark points arethe positions of the objects to be found.

As a conclusion, the interactive technique proposed is anefficient alternative for mixed reality interaction. Experi-mental results have shown that it presents competitive timesif compared with the use of the mouse and keyboard. We

believe that the kind of interface presented here should beconsidered in next-generation interfaces due to the intuitive-ness and whole-body memory clues it provides to users whennavigating in 3D virtual environments.

While tablet PCs are still limited, new models to be re-leased would benefit a lot if they also provided mouse but-tons on the top or the bottom to be more ergonomic. Theycould also have acceleration sensors, like the Wiimote andmany smartphones already have today. This combinationwould allow participants to use general purpose computersin a very different way they do today. More than ubiqui-tous, computers tend to become invisible. Next-generationapplications would also benefit from tablet PCs with built-in tracking systems.We believe that a standard and accu-rate method for tracking devices locally is needed in orderto turn 3D tracking into a standard feature of new portablecomputers. The incorporation of these new features couldreally change the way people use mobile computers and vir-tual reality would possibly become more and more relevantout of the academic and entertainment fields.

7. ACKNOWLEDGEMENTSThanks to the volunteers who tested the prototype, Hewlett-

Packard for the tablet PC and MAMS Tecnologia for theBraTrack system. This work was supported by CNPq-Brazil.

8. REFERENCES[1] R. P. Darken and J. L. Sibert. A toolset for navigation

in virtual environments. In UIST ’93: Proceedings ofthe 6th annual ACM symposium on User interfacesoftware and technology, pages 157–165, New York, NY,USA, 1993. ACM.

[2] R. J. Jacob, A. Girouard, L. M. Hirshfield, M. S. Horn,O. Shaer, E. T. Solovey, and J. Zigelbaum.Reality-based interaction: a framework for post-wimpinterfaces. In CHI ’08: Proceeding of the twenty-sixthannual SIGCHI conference on Human factors incomputing systems, pages 201–210, New York, NY,USA, 2008. ACM, ACM.

[3] A. Olwal. Lightsense: enabling spatially awarehandheld interaction devices. Mixed and AugmentedReality, IEEE / ACM International Symposium on,0:119–122, 2006.

[4] A. Olwal and A. Henrysson. Lumar: A hybrid spatialdisplay system for 2d and 3d handheld augmentedreality. International Conference on Artificial Realityand Telexistence, 0:63–70, 2007.

[5] M. Tsang, G. W. Fitzmzurice, G. Kurtenbach,A. Khan, and B. Buxton. Boom chameleon:simultaneous capture of 3d viewpoint, voice and gestureannotations on a spatially-aware display. InSIGGRAPH ’03: ACM SIGGRAPH 2003 Papers,pages 698–698, New York, NY, USA, 2003. ACM.

[6] D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond,and D. Schmalstieg. Pose tracking from naturalfeatures on mobile phones. In Mixed and AugmentedReality, 2008. ISMAR 2008. 7th IEEE/ACMInternational Symposium on, pages 125–134, 2008.

[7] F. Wartenberg, M. May, and P. Peruch. SpatialCognition, volume 1404/1998 of Lecture Notes inComputer Science. Springer, Berlin / Heidelberg,January 1998.

1228