How applications for 3D user interfaces are like the ocean

Pristis pectinata - Georgia Aquarium Jan 2006

So what’s 3D navigation really good for?

After learning how to fly around the world, I stepped back to think about where the innovations might lie – what the ‘low hanging fruit’ and the ‘holy grail’ might be for making the most of Leap Motion and similar 3D interfaces.


In some environments, not touching a control device is a Good Thing. A few scenarios that quickly came to mind:

  1. wearing gloves (e.g. for insulation or protection) which could make it hard to use conventional devices (like mice and keyboards)
  2. not wanting dirty hands (or gloves) to crud up control devices, and
  3. not wanting clean hands (or gloves) to be contaminated by potentially dirty control devices.

To me, these exploits seem like the low-hanging fruit – quick wins. (I’m not saying they’ll be easy to accomplish – for instance, how much will a normal pair of winter gloves interfere with recognition of hand gestures by people working outdoors? But they’re somewhat obvious applications to target.)

Beyond touch-free

Thinking about ways to exploit touch-free user interfaces with existing applications is fun and cool, just like Tom Cruise’s gloved computer navigation in Minority Report. It’s like having a touchscreen in the air, and some useful things can be done with just that.

Yet it’s a far bigger, but more interesting, challenge to think of completely new ways to use 3D touch-free controllers to do things that maybe we don’t yet dream of being able to do. That’s where the real innovations to rock our world will come from. But how can we discover them?

Trying to ‘see the water’

We humans are capable of being “perfect filters”: our assumptions can be so deeply ingrained that often we don’t even realize we’re making them. As Alan Willett so nicely analogized at SEPGNA 2012:

“Most of us are like fish in the water – we don’t see the water.”

Just as ocean water is hard for the fish swimming in it to see, applications where 3D control (not just a 2D virtual touchscreen) is useful can be hard to see. Finding the killer apps for 3D interfaces will require questioning ourselves about assumptions we probably don’t even realize we’re making about how we interact with our computers.

3D spreadsheets FTW?

As an example, look at how we use spreadsheets to store and manipulate data. We are so accustomed to a 2D paradigm that thinking about navigating our row,column oriented spreadsheet tools with a 3D interface like Leap Motion seems silly. (At least, that’s what many of my colleagues said when I asked if or how a Leap controller could help them do their daily work on computers).

But what if 3D control became commonplace? What kinds of workbook tools could we have, and what new insights might we gain, if we could explore data cubes instead of sheets?

Look Ma, two hands

What sophisticated controls might be possible with two coordinated hands instead of one? In my experience this is very hard to do with devices, yet humans can learn to do things with two independent hands all the time, like playing the guitar. (Seriously, have you ever tried to use two joysticks or mice at once, one per hand? If you’ve ever gotten it to work, let me know! I’m not a hard-core gamer, but to me, it’s like patting your head and rubbing your tummy at the same time.)

The holy grail for gestural control

A perfect storm for gestural controls would seem to be be combining all three benefits: to have touch-free, two-handed, 3D navigation. Now that I want to see – and do – someday soon.

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.

Photo by DAVID ILIFF. License: CC-BY-SA 3.0


Leaping around the world

My next adventure after Touchless app for Windows was flying through Google Earth with Leap Motion.

I quickly got GE configured for 3D control and, like everyone else, promptly had the earth spinning wildly! I also kept seeing the ‘dots’ on the screen from Touchless, along with GE’s unexplained fat and thin up/down and left/right arrows. After disabling Touchless from the system tray, the dots went away, but my control didn’t improve.

The support forum FAQ article showed two distinct user populations: those that loved flying around the world via Leaping, and those who found it frustrating. Videos seemed to help many people, so I visited the company’s channel (it had 13,741 subscribers as of my visit).

I found lots of videos, mostly 2 weeks to 2 months old. The initial promo video provided by the company for leaping with Google Earth was cool-looking, but only 29 seconds long, and pretty useless for helping someone learn how to fly like that: it was impossible to judge correct hand position relative to the controller because the controller wasn’t visible in the video. More promising was a playlist of 40 videos on how to #LeapInto Google Earth, plus some direct video links from the support forum.

Fortunately, it became clear that some early adopters who tried it out had started documenting. Julia’s #LeapInto Sweden was short but helpful. By far the most useful link I found for guidance on flying without spinning was Leap Motion – Google Earth instructions. This video showed correct hand orientation and control, and explained what the fat and skinny arrows meant and how to navigate.

With that start, within an hour I was able to navigate comfortably around the buildings of NC State Centennial Campus and around Raleigh. Relaunching it an hour later required a bit of re-learning, but I was soon able to fly around San Francisco and under the Golden Gate Bridge, just like another demo video showed. I spent another 30 min or so flying to Vatican City, Sweden, the Statue of Liberty, and some places I used to live in the US.

Now I actually feel comfortable enough to consider showing it at an internal technology demo tomorrow. I can’t wait to hear what people think!

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.

All other trademarks and registered marks are the property of their respective owners.

Leaping into Windows 7

Next adventure after Painting by Leaping: the Touchless app for Windows – can I wave at my laptop to scroll and click instead of using a mouse?

I found a developer video (based on v0.7.3) illustrating Visualizer and other functions on Windows 8, and another video which seemed to show using Touchless instead of keyboard and mouse on Windows 8. Great, but I have Windows 7.

One thing that perplexed me after installing my Leap Motion software was the appearance of a new “Flicks” system tray item. Microsoft’s “What are flicks?” article says flicks are for a pen or finger only. It seemed logical though that the ‘flicks’ conventions for navigation (e.g. upward flick to scroll down) would apply to the Leap Motion controller.

Today (August 8) I launched the Control Panel to try out the Visualizer and was prompted to download another update. I did so, then tried launching Touchless from Airspace. It threw up the User Account Control dialog, I granted permissions, then – nothing. It just stopped. I saw one brief popup from the system tray about a smudge and needing to clean the window of the controller. I did that, but still no dice.

Restarted just for fun and relaunched Airspace and the Touchless app. No apparent change – nothing happened when I moved my hands over the sensor. This time I looked for hidden system tray icons, and found a new one. It didn’t actually say “Touchless” – it just said “Interaction”. “Open Help Guide” launched IE8 (my default browser for corporate reasons) with the tutorial. It doesn’t run well in IE though. Switching to Chrome let me see the tutorial. Back in the system tray icon, I looked under the Interaction submenu and saw that it was “Disabled”. I changed it to “Basic” and now I was able to see fingers on screen. Yay!

One immediate observation: the mapping of gestures to monitor only works on the primary monitor – no obvious way to navigate or control apps on the second monitor that I use when my laptop is in its dock.

Next I tried practicing scrolls and clicks and zooms on the tutorial page in Chrome. Clicking in the vertical scroll bar worked. The scroll and zoom gestures didn’t. Tried a Word document and had trouble scrolling in that, although a light overlay of a mouse kept showing on screen. I had some luck “clicking” the mouse buttons, but that’s not really what the sensor is good for! The keyboard icon popped up sporadically too – it wasn’t clear what I was doing with my gestures to trigger it. I opened a PDF (saved earlier from the PC Magazine article “10 Best Apps for the Leap Motion Controller“) and had similar results trying to navigate in Reader.

I revisited the Support Forum at this point to look for clues. I found one topic on what to do if the error “Touch Emulation Driver Corrupt or missing” appeared. Well, I wasn’t seeing that error, but touch emulation certainly wasn’t right. So I figured it wouldn’t hurt to try the advice – manually installing MultiTouch from the Leap Motion directory. After a User Account Control dialog, I got a dialog for about 15 seconds saying “Installing, this could take a few minutes…” When it finished, I switched back to Reader, and now two-finger zooming worked. Yay!

All in all, some progress and some frustrations. I tend to work with multiple apps and windows open at once, and I frequently made accidental clicks on other windows on the screen. I clearly need practice – and will look for more crowd-sourced tips before I try it again.

For fun, I tried turning ‘flicks’ off from the system tray. It didn’t seem to affect behavior at all.

Next stop: world travel with Leap Motion and Google Earth (the 11th item listed in the PC Magazine article).

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.

Leaping forward?

Got a Xbox Kinect, Playstation Move, or similar 3D gaming interface at home? Isn’t it cool to not have to hunt for a remote or wand or controller to be able to participate in a game? And it’s nice to not worry whether our devices have power, or whether kids (or adults) will spill food, drinks, crumbs, or dirt on them and make them unusable.

Wouldn’t it be even cooler if we had touch-free control of our desktops, laptops, and enterprise computers, the way Tom Cruise did in “Minority Report”? (see WIMP video here)

Technology has been getting closer to making this vision a reality. For instance, this demo at TED 2010 showed a spatial user interface, but it required the user to wear special gloves.
TED 2010 demo of a spatial user interface by John Underkoffler

A new device by LeapMotion proposes to make the vision of accessory-free, touch-free navigation a reality. I pre-ordered a Leap Motion controller last year, and watched as beta stories like NASA controlling a space robot with a Leap Motion sensor began to surface.

Today I received my new toy technology device. The LeapMotion controller is tiny: a sleek, lightweight device about as long and thick as my index finger and twice as wide. So it takes just a small bit of space in front of my keyboard. [see more photos here].
Leap Motion image of their new controller: image usage believed to be consistent with their Social Medial Guidelines at

I will be writing more here as I try out my Leap Motion controller. Stay tuned!

Image credits:

  1. TED 2010 image by Steve Jurvetson on Flickr, via Wikimedia, under the Creative Commons Attribution 2.0 Generic license
  2. Leap Motion controller image used in accordance with their social media guidelines

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.