The latest from Ultraleap and exploring all things hand tracking
The current state of hand-tracking
--
Hand tracking has come a very long way.
If you look back even two years ago, you’ll find a plethora of “why is hand tracking so bad” Reddit posts and articles like “Quest 2 Hand Tracking Is A Neat Trick, But Far From Game Ready.”
Hand tracking is complicated. Ridiculously complicated. Feel free to dive into the minutia here.
So it’s still not perfect, and I don’t think I’m bursting anyone's bubble saying that.
But, and it’s a pretty big but, the gap between available tech and ridiculous video renders is closing.
The Latest Hand Tracking Tech
I got to play with one of the most advanced hand tracking systems on the market called the Ultraleap Gemini. As the fifth edition, Gemini is now available for multiple platforms, camera systems, and third-party hardware.
Here’s why I’m amped and think you should be too.
First Impressions
The second I got it up and running, it was obvious that this is the best hand tracking I’ve ever tried. It is the closest experience where my virtual hands felt like my real hands.
- It’s fine-tuned — It can recognize between different hand sizes and that matters for accuracy
- Stable — your hands stay in view and there isn’t any annoying flickering. I tested with their Ultraleap Stereo IR 170 which has a decent field of view
- Low lighting condition — I was still able to play in a dimly lit room and it worked well (I live in a woman cave)
- It’s precise — it knows when you’re trying to trick it, like me introducing two left hands in the second half of my video below
During testing/playtime, it made me think of an experiment called the “Rubber Hand Illusion,” where a user becomes convinced a ‘fake’ hand is actually their own.
The best way I can describe it is, if you get hurt while in the Matrix, you get hurt in real life. Your brain can’t tell them apart.
Science and even magicians have proven that multisensory perception can influence how we perceive our own body, and this is very much the direction hand tracking is going. Especially within immersive augmented reality (AR) and virtual reality (VR) experiences (more on this below).
Downsides of Gemini? One of the ultimate hand tracking tests is giving the middle finger. This is something it had some issue tracking. I noticed other people were able to do it fine.
You can check out industry icon SkarredGhost’s more technical take here.
The other general downside with hand tracking is that without controllers, haptics gets put to the wayside, which has a major impact on immersion.
FYI, I’ll be talking more about haptics and immersion at AWE2021 with SenseGlove, Manus VR and BeBop Sensors
As an FYI, the Ultraleap Stereo IR 170 hardware and Gemini software is enterprise-level, which means it requires technical know-how to set up and use. If you’re not a developer, I don’t recommend this for you. If you're a developer, here’s an early access look at the requirements and specs.
Gimme the Good Stuff
Getting back to those immersive experiences! Hand tracking has the potential to be better than controllers but we’re not there quite yet.
For one, content needs to be designed for hand tracking specifically, otherwise, it’s like teleporting a PC game into VR — it’s half-baked.
Luckily the content scene is heating up. Here’s a look at some of the best hand tracking games and experiences out there right now:
Best Hand Tracking Games for Oculus Quest 2
The Best in VR/XR Are Betting on Hand Tracking
While diving deeper into hand tracking in the latest AR/VR releases, I kept seeing Ultraleap pop up over and over again. Here are a few of those companies and some other notable mentions in the hand tracking space.
Varjo XR-1
The Varjo Aero was announced 5 days ago and utilizes Ultraleaps’s new Gemini V5 system. Note: Ultraleaps’ tracking is featured in the Varjo XR-3, VR-3 and VR-2 Pro headsets, but there is no hand tracking in the Varjo Aero.
Pimax 5K VR
The Pimax Vision 5K PLUS utilizes the Ultraleap hand tracking module in its 8K X and 9K Plus headsets.
This is an unofficial video (6 minutes) by NextGenVR reviewing the hand tracking system.
Lynx R-1
This video was released 1 day ago by Lynx. This headset utilizes Ultraleap’s Gemini v5.
Qualcomm XR2
Back in 2020, Qualcomm and Ultraleap signed an agreement to use Gemini in their standalone Snapdragon XR2 5G reference design. Here’s a link to the developer blog. It’s hard to find much more information about the XR2 online.
Oculus Quest 2
Oculus has been a big promoter of hand tracking. In terms of what tech is being used for hand tracking, it’s not Ultraleap and it does not support it. You probably could have guessed that though.
For all the Guitar Hero fans, this month a game called Unplugged Air Guitar was released on the Quest platform. Apparently, it's “the best conceptual and technical hand tracking game on Oculus,” according to Upload VR.
Other Notable Mentions:
- Pico Neo 3 — You can use Ultraleap’s hand tracking solution with the Pico Neo 3 Pro by adding Ultraleap’s Stereo IR 170 camera module and utilizing Ultraleap’s Gemini software for XR2. Check it out in action here.
- VIVE Flow mentions hand tracking but I wasn’t able to find any tangible information on it yet.
- There’s much speculation around hand tracking and the Oculus Quest 3. There are a lot of leaks but it’s not out yet. So, there’s nothing else to say.
Just Do It Yourself
These are the corporations and startups in hand tracking. There are so many amazing makers, developers, and hackers too. Some of the best, craziest ideas come from people who build things because they don’t exist yet.
Here’s a glance at some of the coolest stuff going on (that I could find):
Wrist-based Tracking with Haptics
Pinch-Based Tracking
Hand Tracking Using Computer Vision
Standalone Hand and Feet Tracking
Looking Ahead
The buzz around the future of hand tracking has been in speculation for over 10 years. If other cool/mainstream technologies tell us anything, it’s that cool tech has to be ready ‘at the right time’ and the journey requires evangelists and pooh-poohers along the way.
In my humble opinion, not every experience needs to be a ‘key use case’ or groundbreaking though.
The iPod gave us the same thing we already had but did it in a different way. I think we forget that sometimes. It didn’t really have a totally new killer use case. It was just done differently. At the start, it wasn’t super intuitive either, but it became a standard interface.
What’s the future for Ultraleap look like?
According to Matt Tullis, Ultraleap’s Director of Product Management, Spatial Computing “Hand Tracking will be to VR what touchscreen was to mobile.” I’ll be at AWE2021 next month to hear this bold claim out!
Thanks
Special thanks to Ultraleap for access to the Gemini system and shipping me Ultraleap Stereo IR 170 for free. I love new tech and am always up to test it out!
