How applications for 3D user interfaces are like the ocean

So what’s 3D navigation really good for?

After learning how to fly around the world, I stepped back to think about where the innovations might lie – what the ‘low hanging fruit’ and the ‘holy grail’ might be for making the most of Leap Motion and similar 3D interfaces.

Touch-free

In some environments, not touching a control device is a Good Thing. A few scenarios that quickly came to mind:

1. wearing gloves (e.g. for insulation or protection) which could make it hard to use conventional devices (like mice and keyboards)
2. not wanting dirty hands (or gloves) to crud up control devices, and
3. not wanting clean hands (or gloves) to be contaminated by potentially dirty control devices.

To me, these exploits seem like the low-hanging fruit – quick wins. (I’m not saying they’ll be easy to accomplish – for instance, how much will a normal pair of winter gloves interfere with recognition of hand gestures by people working outdoors? But they’re somewhat obvious applications to target.)

Beyond touch-free

Thinking about ways to exploit touch-free user interfaces with existing applications is fun and cool, just like Tom Cruise’s gloved computer navigation in Minority Report. It’s like having a touchscreen in the air, and some useful things can be done with just that.

Yet it’s a far bigger, but more interesting, challenge to think of completely new ways to use 3D touch-free controllers to do things that maybe we don’t yet dream of being able to do. That’s where the real innovations to rock our world will come from. But how can we discover them?

Trying to ‘see the water’

We humans are capable of being “perfect filters”: our assumptions can be so deeply ingrained that often we don’t even realize we’re making them. As Alan Willett so nicely analogized at SEPGNA 2012:

“Most of us are like fish in the water – we don’t see the water.”

Just as ocean water is hard for the fish swimming in it to see, applications where 3D control (not just a 2D virtual touchscreen) is useful can be hard to see. Finding the killer apps for 3D interfaces will require questioning ourselves about assumptions we probably don’t even realize we’re making about how we interact with our computers.

3D spreadsheets FTW?

As an example, look at how we use spreadsheets to store and manipulate data. We are so accustomed to a 2D paradigm that thinking about navigating our row,column oriented spreadsheet tools with a 3D interface like Leap Motion seems silly. (At least, that’s what many of my colleagues said when I asked if or how a Leap controller could help them do their daily work on computers).

But what if 3D control became commonplace? What kinds of workbook tools could we have, and what new insights might we gain, if we could explore data cubes instead of sheets?

Look Ma, two hands

What sophisticated controls might be possible with two coordinated hands instead of one? In my experience this is very hard to do with devices, yet humans can learn to do things with two independent hands all the time, like playing the guitar. (Seriously, have you ever tried to use two joysticks or mice at once, one per hand? If you’ve ever gotten it to work, let me know! I’m not a hard-core gamer, but to me, it’s like patting your head and rubbing your tummy at the same time.)

The holy grail for gestural control

A perfect storm for gestural controls would seem to be be combining all three benefits: to have touch-free, two-handed, 3D navigation. Now that I want to see – and do – someday soon.

---

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.

Photo by DAVID ILIFF. License: CC-BY-SA 3.0

Advertisement

Leaping around the world

My next adventure after Touchless app for Windows was flying through Google Earth with Leap Motion.

I quickly got GE configured for 3D control and, like everyone else, promptly had the earth spinning wildly! I also kept seeing the ‘dots’ on the screen from Touchless, along with GE’s unexplained fat and thin up/down and left/right arrows. After disabling Touchless from the system tray, the dots went away, but my control didn’t improve.

The support forum FAQ article showed two distinct user populations: those that loved flying around the world via Leaping, and those who found it frustrating. Videos seemed to help many people, so I visited the company’s channel (it had 13,741 subscribers as of my visit).

I found lots of videos, mostly 2 weeks to 2 months old. The initial promo video provided by the company for leaping with Google Earth was cool-looking, but only 29 seconds long, and pretty useless for helping someone learn how to fly like that: it was impossible to judge correct hand position relative to the controller because the controller wasn’t visible in the video. More promising was a playlist of 40 videos on how to #LeapInto Google Earth, plus some direct video links from the support forum.

Fortunately, it became clear that some early adopters who tried it out had started documenting. Julia’s #LeapInto Sweden was short but helpful. By far the most useful link I found for guidance on flying without spinning was Leap Motion – Google Earth instructions. This video showed correct hand orientation and control, and explained what the fat and skinny arrows meant and how to navigate.

With that start, within an hour I was able to navigate comfortably around the buildings of NC State Centennial Campus and around Raleigh. Relaunching it an hour later required a bit of re-learning, but I was soon able to fly around San Francisco and under the Golden Gate Bridge, just like another demo video showed. I spent another 30 min or so flying to Vatican City, Sweden, the Statue of Liberty, and some places I used to live in the US.

Now I actually feel comfortable enough to consider showing it at an internal technology demo tomorrow. I can’t wait to hear what people think!

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.

All other trademarks and registered marks are the property of their respective owners.

Leaping into Windows 7

Next adventure after Painting by Leaping: the Touchless app for Windows – can I wave at my laptop to scroll and click instead of using a mouse?

I found a developer video (based on v0.7.3) illustrating Visualizer and other functions on Windows 8, and another video which seemed to show using Touchless instead of keyboard and mouse on Windows 8. Great, but I have Windows 7.

One thing that perplexed me after installing my Leap Motion software was the appearance of a new “Flicks” system tray item. Microsoft’s “What are flicks?” article says flicks are for a pen or finger only. It seemed logical though that the ‘flicks’ conventions for navigation (e.g. upward flick to scroll down) would apply to the Leap Motion controller.

Today (August 8) I launched the Control Panel to try out the Visualizer and was prompted to download another update. I did so, then tried launching Touchless from Airspace. It threw up the User Account Control dialog, I granted permissions, then – nothing. It just stopped. I saw one brief popup from the system tray about a smudge and needing to clean the window of the controller. I did that, but still no dice.

Restarted just for fun and relaunched Airspace and the Touchless app. No apparent change – nothing happened when I moved my hands over the sensor. This time I looked for hidden system tray icons, and found a new one. It didn’t actually say “Touchless” – it just said “Interaction”. “Open Help Guide” launched IE8 (my default browser for corporate reasons) with the tutorial. It doesn’t run well in IE though. Switching to Chrome let me see the tutorial. Back in the system tray icon, I looked under the Interaction submenu and saw that it was “Disabled”. I changed it to “Basic” and now I was able to see fingers on screen. Yay!

One immediate observation: the mapping of gestures to monitor only works on the primary monitor – no obvious way to navigate or control apps on the second monitor that I use when my laptop is in its dock.

Next I tried practicing scrolls and clicks and zooms on the tutorial page in Chrome. Clicking in the vertical scroll bar worked. The scroll and zoom gestures didn’t. Tried a Word document and had trouble scrolling in that, although a light overlay of a mouse kept showing on screen. I had some luck “clicking” the mouse buttons, but that’s not really what the sensor is good for! The keyboard icon popped up sporadically too – it wasn’t clear what I was doing with my gestures to trigger it. I opened a PDF (saved earlier from the PC Magazine article “10 Best Apps for the Leap Motion Controller“) and had similar results trying to navigate in Reader.

I revisited the Support Forum at this point to look for clues. I found one topic on what to do if the error “Touch Emulation Driver Corrupt or missing” appeared. Well, I wasn’t seeing that error, but touch emulation certainly wasn’t right. So I figured it wouldn’t hurt to try the advice – manually installing MultiTouch from the Leap Motion directory. After a User Account Control dialog, I got a dialog for about 15 seconds saying “Installing, this could take a few minutes…” When it finished, I switched back to Reader, and now two-finger zooming worked. Yay!

All in all, some progress and some frustrations. I tend to work with multiple apps and windows open at once, and I frequently made accidental clicks on other windows on the screen. I clearly need practice – and will look for more crowd-sourced tips before I try it again.

For fun, I tried turning ‘flicks’ off from the system tray. It didn’t seem to affect behavior at all.

Next stop: world travel with Leap Motion and Google Earth (the 11th item listed in the PC Magazine article).

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.

Painting by leaping

Adding my first Airspace app

After getting started with my Leap Motion controller, I proceeded to ‘got’ the free Corel® Painter® Freestyle™ app. The prerequisites listed on the page looked OK, although the page warned that it would need 79mb and ‘more space might be needed’ for .NET 4.0; so this app will make an interesting test of whether .NET 4.5 would meet the dependency.

When I clicked “Get App”, the Airspace store page for Corel Painter® Freestyle™ prompted me for my password, then quickly switched to ‘Purchased’, and my computer got a bit sluggish. For a few minutes, I assumed the app was being downloaded in the background to my laptop.

I did something else for about 15 minutes, but my Airspace home page hadn’t yet refreshed to include a new app button, so it wasn’t clear what was going on. [Based on later experiences with app ‘purchases’, no apps seem to actually download until Airspace is refreshed by exiting and re-launching.] I tried exiting the Airspace page by File|Exit, to no effect. Closing it by clicking the X at top right worked, though.

On restarting Airspace from my desktop shortcut, it detected that my LM sensor wasn’t attached, and offered me the option of continuing with mouse only. Again, this feature probably originated as a convenience for development and testing, but leaving it in for customers is a sign of thoughtful error handling in the design of the software. I selected that option and continued.

My Airspace page now loaded with the 5 previous apps plus my new Painter Freestyle app which began “Downloading …” About 15 minutes later, the download had finished. This seemed quite slow, even given that the app needed about 150 mb for its application files. Then I got a new Dependency page that didn’t tell me exactly what was needed, only asked me to install the “Painter® Freestyle™ Dependencies”. I always like to know what’s inside those boxes before I install them – so for now, I didn’t.

Painting in Airspace

Once I got the Orientation app working (see next story), I was able to launch this app without any difficulty, and without installing any additional prerequisites. It launched pretty fast and ran well (unlike Cut-The-Rope, which still doesn’t run).

Actually painting something with the app clearly takes practice. On my first tries, I found it hard to move to the toolbar above the paint area (to choose a new brush or color) without unintentionally painting. This needs more trial time before I can ‘draw’ any conclusions.

Saving a Painting

Saving my crude paintwork as .RIF (the default) yielded a 2644kb file (for a 1280×800 canvas). Windows, of course, didn’t know how to open it from Explorer. I was pleasantly surprised to be able to open the file from the app and save it as .PNG without using the controller (.JPEG, .TIFF, and .PSD formats are also supported). The PNG file wasn’t much smaller, but saving the JPG as ‘good’ quality or 65% yielded a compact 181kb file; at 80% or ‘high’ quality, the file was 265kb.

I’m not much of a non-digital artist, so I don’t expect to be creating any masterpieces with the app. 🙂 Strictly for the curious, here’s a half-size image of the example:

First try at using Corel® Painter® Freestyle™ with Leap Motion

More adventures to follow …

---

1. Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.
2. Corel®, Freestyle™, and Painter® are trademarks or registered trademarks of Corel Corporation and/or its subsidiaries in Canada, the United States and/or other countries.

Orientation to leaping

Getting Orientation.exe to stop crashing

After getting started with my Leap Motion controller, I had stopped at the point where the Orientation.exe app was crashing. A few days later, I was ready to try again.

Checking for prerequisites

First step: reconfirm prerequisites. As of May 19, 2013, the minimum system requirements were:

• Windows® 7 or 8 or Mac® OS X 10.7 – Windows 7 Enterprise (64-bit) with SP1
• AMD Phenom™ II or Intel® Core™ i3, i5, i7 processor
• 2 GB RAM – I have more than 8gb
• USB 2.0 port – check
• Internet connection – check

No problems there.

Next I looked in the Airspace Control Panel for software updates, and found one which said it fixed startup issues with Orientation.exe. But still no luck. I checked the support forum and troubleshooting guides and didn’t find anything useful. At this point (on July 29), I set my Leap Motion controller aside to work on other things.

First leap, take two

Absent any replies to my support email requests and twitter pings, I revived my investigation this past weekend (August 4) by considering possible unstated prerequisites. One that came to mind was the “.NET 4.0” that Cut The Rope apparently needs. Maybe Orientation.exe and other apps need it too?

Using Aaron Stebner’s .NET verifier, I confirmed that my prior . NET 4.5 installation did provide a functional version of .NET 4.0 ( both Client and Full).

Next I re-checked for updates (none found), and scanned the troubleshooting pages. Here I found a new Support entry from August 2 on “How to update your graphics driver“. Aha! (Should have thought of that.)

Rather than following their instructions to the letter and trying to figure out exactly which NVIDIA Quadro FX 880M driver I should manually download, I used the Windows “Update Driver” function. That worked beautifully (after about 20 min on a bandwidth-constrained connection). All I had to do after installation was reset the driver configuration to 1280×800 (listed as the minimum by many Leap Motion apps) and full width.

Getting Oriented!

Now Orientation.exe ran beautifully. I actually didn’t expect it to, so I still had my laptop on mute! I grabbed a pair of headphones to plug in and restarted it. It was way too loud at 50% headphone volume level – I reduced it to 15% and went through the app once. (Mute would have been fine. Orientation has no audio or aural instructions, and I found the music kind of annoying – more distracting than inspiring. YMMV.)

In ‘where it sees’ of Orientation, I noticed that the colors of the visualization changed towards orange or red when I made quicker moves – that was cool, and promising.

In ‘what it sees’, I quickly figured out how to get it to recognize both of my hands, and most of the time, all of my fingers. But some simple gestures that I expected to work well didn’t – e.g. gradually closing a thumb and finger into a pinch, or bringing a left finger and right finger together. In both cases, the two fingers would ‘disappear’ soon before the tips actually touched. I have not yet gotten this to work, but will keep experimenting with it.

‘How to draw’ was a good introduction to apps that use depth for ‘touching’ and ‘lifting’ from a surface. I definitely need more runtime on this too, but will probably just experiment with Corel® Painter® Freestyle™ app from now on. Trying to draw something of any complexity in Orientation, only to have the first parts erased, got to be a bit frustrating.

Who’s training who?

My techie husband saw me doing these Orientation experiments and got curious. One of his first questions was, “Are you training it, or is it training you?” It’s a great question. He shares my interest in understanding how well user interface technologies adapt to people, not vice versa. In the long run, technologies that require careful user training are not likely to ‘stick’ or be productive for sporadic use scenarios. (Key architectural quality attributes: learnability, memorability, and operability!) But for purposes of exploring a first version of a new UI technology, I don’t mind being trained a bit.
 

More adventures to follow …

---

1. Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.
2. All other trademarks and registered marks are the property of their respective owners.

First Leap

Some notes from my “first leap” – getting started with my new Leap Motion device:

Resources

Before jumping in, I read through the FAQs, legal terms, a tutorial on using it on Windows, and some ergonomic hints on the website, as well as the customary Twitter, Google+, and Facebook links. My overall impression was of a professional organization that has thought carefully about its users – an encouraging start.

Installation

Their startup web page auto-detected my operating system correctly, and the software download (about 57 mb) was fast, under a minute (kudos to whoever at Leap Motion planned for scalability of their web portal). Installation of “Leap Motion software and Airspace” was simple, was properly signed with a security certificate, and didn’t require any special permissions (more kudos on all counts).

I plugged its USB cable into my laptop (Windows 7 64-bit Enterprise). After a firmware update and quick installation of device drivers, it was ready to use. So far, the road looked smooth and fast.

Startup

Their “Welcome” app launched after installation finished. Here I hit a glitch after viewing the “Orientation” hints and repositioning my LeapMotion sensor in front of my keyboard as they indicated: I clicked “Continue”, and Orientation.exe promptly crashed (see image). But the crash appeared harmless, and their software continued introducing me to “Airspace”, their app store (iTunes equivalent).

The next speed bump was registering for an Airspace account. I tried to register and leave birthdate blank, but registration failed. This was annoying – I’ll readily acknowledge that I’m WAY over both 18 and 21, so I’m annoyed by spurious requests for the month and day (fodder for identity thieves). I took away a few kudo points for privacy concerns, and continued. Then when I re-tried the registration with a birthdate, I found the page had wiped out the password I’d already entered and I had to put it in again (twice). Grr. But moving on … I made it to the Airspace Home page.

Five app buttons loaded on the page, and then another browser window popped up in front of it. I knew from reading the Leap Motion blog that way more than 5 apps were available, so I wondered if the home page didn’t fully load.

The popup window indicated that the fifth app, “Cut the Rope” (a game), has dependencies I didn’t yet have installed: Microsoft .NET Framework 4.0, and Microsoft XNA Framework 4.0. This puzzled me a bit because I already have .NET 4.5 installed. Shouldn’t the Leap Motion framework and “Cut the Rope” be upward compatible with v4.5?

Being a software hacker (the good kind), I did a bit of poking around at this point. The dependency installation page pointed to C:\Users\[myID]\AppData\Local\AirspaceApps\CuttheRope\Dependencies\README.html, so I took a look in C:\Users\[myID]\AppData\Local\AirspaceApps\. It contained only 3 folders: CuttheRope, Lotus, and Shimsham. (I guess the “Airspace Store” and “Orientation” don’t need folders.) CuttheRope contained an .EXE, some sub-folders, and LeapCSharp.NET4.0.dll. (So I’m guessing maybe it’s not just CuttheRope that is going to have these dependencies.) Its “Dependencies” sub-folder contained installers (one .MSI, one .EXE) for both installations. Kudos for the self-containment, but this didn’t fully offset the jolt of the dependency interruption.

My two cents: Seeing the first home page display get stuck on such dependencies is annoying and weighs down the up-til-now pretty lightweight installation process. Components required by the pre-installed LM apps should be automatically installed along with the Leap software and Airspace”.

I decided to ignore “Cut the Rope” and its dependencies, and try to run one of the 4 visible apps that did appear to load.

Airspace: The Leap Motion app store

Clicking the Airspace Store app button just took me to their website, https://airspace.leapmotion.com/, carrying with it my account information. I explored my profile a bit and found just Username (already populated), Email (already populated), and Payment Type (blank). Hmm. This left me wondering how many apps useful for my investigation would be free or reasonably priced. Of the Top Picks in the store, only a few (including Cut The Rope) appeared to be free, with most priced between $1.99 and $4.99. I’ll be further exploring the “free apps” category link.

Google Earth looked promising, and was included by PC Magazine in their “10 Best Apps for the Leap Motion Controller“. But the link almost completely failed to load in IE8 (all I got were the 3 tumblr links). It loaded much better in Firefox, so I plan to do most of my further app explorations there instead of IE.

Stuck at Orientation

Orientation.exe again crashed, this time with Problem Signature:

Problem signature:
Problem Event Name: BEX
Application Name: Orientation.exe
Application Version: 0.0.0.0
Application Timestamp: 51ea9edd
Fault Module Name: StackHash_0a9e
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 00000000
Exception Offset: 00000000
Exception Code: c0000005
Exception Data: 00000008
OS Version: 6.1.7601.2.1.0.256.4
Locale ID: 1033
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409

If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt

I tried using “Check online for a solution”, to no apparent avail. I also tried searching Airspace for the Orientation app, but the only match was Corel Painter Freestyle (which is free, so I expect I’ll give it a try).

At this point, I unplugged the USB from my laptop to get ready to go to a meeting. Almost immediately, I got a system tray popup that said “Smudge detected. Your Leap Motion Controller may function better if you clean the window.”. Um, no, that’s not the problem 😉

Troubleshooting

After my meeting, I did a little investigation in the Leap Motion control panel before ending my experiments for the afternoon. First I checked the Software Log looking for insights on the Orientation.exe crash, and found at the end of the log:

[15:00:16] [Info] Leap Motion Controller disconnected: LP81576857099
[15:00:24] [Warning] Smudge detected. Your Leap Motion Controller may function better if you clean the window.

So the disconnection was detected correctly, but the message was overridden by a spurious smudge detection – a minor adjustment might be needed in that part of the software.

I didn’t find any clues to the Orientation crash in the remainder of the log, or in the FAQ. I’m going to inquire with Leap Motion support before installing anything further.

I did find a few perplexing log entries, such as “[12:21:05] [Warning] The device has entered ‘Robust Mode’ to compensate for lighting conditions.” and many more messages about the device going into and out of “robust mode”. That probably reflects the fact that I sit and move around between a big window and my keyboard & LM sensor. I may need to adjust my window blinds to darken my space a bit.

I also had one log message “[Info] Reversed tracking orientation”. By default, the Tracking tab of the control panel has “Auto Orient Tracking” checked by default, so moving the device during setup probably explains it.

I may run those two dependency installations to see if it fixes the Orientation.exe crash. And I expect to try out a potentially useful “Diagnostic Visualizer” I found in the Leap Motion control panel.

Then I want to see how well it lets me navigate Windows (via the free Touchless for Windows app).

Running Windows post-installation for a day, without the Leap Motion sensor plugged-in, is a bit “off”. Two observations so far:

• My mouse seemed to be moving and snapping relative to an unseen grid (wouldn’t move diagonally). Unchecking the option to launch the tray item at startup and rebooting seemed to solve this.
• I need to be able to use vertical scroll bars with my mouse or touchpad while reading or showing PDF files in a meeting. My Adobe Reader is now stuck in ‘touch’ mode every time I open a new PDF file (no scroll bars, just a ‘hand’). Attempts to change the touch mode setting in Adobe Reader don’t persist. The multi-touch driver that was installed with Airspace appears to be stickier. I will definitely be looking for a solution to this.

More on my further adventures in future blog entries …

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.

Leaping forward?

Got a Xbox Kinect, Playstation Move, or similar 3D gaming interface at home? Isn’t it cool to not have to hunt for a remote or wand or controller to be able to participate in a game? And it’s nice to not worry whether our devices have power, or whether kids (or adults) will spill food, drinks, crumbs, or dirt on them and make them unusable.

Wouldn’t it be even cooler if we had touch-free control of our desktops, laptops, and enterprise computers, the way Tom Cruise did in “Minority Report”? (see WIMP video here)

Technology has been getting closer to making this vision a reality. For instance, this demo at TED 2010 showed a spatial user interface, but it required the user to wear special gloves.

A new device by LeapMotion proposes to make the vision of accessory-free, touch-free navigation a reality. I pre-ordered a Leap Motion controller last year, and watched as beta stories like NASA controlling a space robot with a Leap Motion sensor began to surface.

Today I received my new toy technology device. The LeapMotion controller is tiny: a sleek, lightweight device about as long and thick as my index finger and twice as wide. So it takes just a small bit of space in front of my keyboard. [see more photos here].

I will be writing more here as I try out my Leap Motion controller. Stay tuned!

---

Image credits:

1. TED 2010 image by Steve Jurvetson on Flickr, via Wikimedia, under the Creative Commons Attribution 2.0 Generic license
2. Leap Motion controller image used in accordance with their social media guidelines

Leap Motion and the Leap Motion logo are the trademarks of Leap Motion, Inc. and are used here by permission.