Interesting Advances in Consumer VR, Making VR Apps - Issue #5
What is the state of consumer VR devices and how easy is it to write VR apps today? I was curious about these questions and this newsletter issue summarizes some of my high level findings.
As a researcher in the HCI and Applied ML space, virtual reality continues to be an increasingly relevant topic. VR is used extensively to simulate the real world with applications in reinforcement learning (see AirSim from Microsoft), robotics, and other fields. Advances in software and hardware solutions also make VR participation democratized, leading to potentially more consumer adoption.
And then, there are increasing conversations on the MetaVerse (see announcements by Facebook 1 and Microsoft 2 ) and how it can help make collaboration more personal and fun2. There are also many interesting HCI implications of VR in terms of engagement, learning, task performance, stress/fatigue and even addiction (see the journal of virtual world research 1 as a potential starting point). Interestingly, some research 4 I conducted a while showed that virtual worlds help users acquire complex tacit knowledge and are perhaps amenable to learning tasks where the subject is complex/tacit knowledge.
All of these reasons make VR particularly interesting!
Recent Advances in Consumer VR Devices
The last time I actively used VR systems was in 2017, working the HTC Vive controller for some experiments with 6 Degrees of Freedom tracking as part of a room scale interactive application (think pointing at/interacting with screens and physical objects in a room). Back then, HTC Vive used its lighthouse tracking system which required base stations installed a room; the base stations emit rapid non visible light in multiple directions, the tethered head mounted display and controllers have photoreceptors which detect the light and uses triangulation to infer the position and orientation of controllers in 3D space (this is also referred to as outside in tracking ). That system was clunky and not portable across space.
I recently tried the Oculus Quest 2 and .. well, things have changed! A few interesting advances over the last few years are worth noting.
Inside Out Tracking - New devices like the Oculus Quest 2 provide a fully untethered experience - tracks head motion, controllers and hands - wireless, no cables, no base station. A big part of this huge shift is enabled by a shift from outside in tracking (e.g., using external sensors for location/orientation) to inside out tracking (e.g., using sensors on the device itself to establish location/orientation). A combination of improving sensors (cameras,IMUs), better SLAM software and improved processor power enable this shift. The Quest essentially digitizes its environment in realtime and uses this model as anchors for tracking its position in space. This is also known as markerless inside-out tracking.
Passthrough Rendering - One drawback of VR headsets was that the user was completely immersed in the VR space and could not see their actual physical space. This meant that to move safely in the physical world, you had to rely on calibrated markers in the VR screen, or take off the headset. With Passthrough, you can see your physical space in the VR space - enabling Mixed Reality applications (sort of).
Realtime 3D Handtracking - Advances in machine learning models enables 3d handtracking, effectively allowing users to use their hands as an input device for interating with VR objects. Facebook originally announced handtracking in the Oculus line of devices in Sept 2019 in developer preview. It still not enabled by default (performance constraints).
For my dev exploration, I chose the Oculus Quest 2; it appears to have the best developer/user experience (untethered, inside out tracking), excellent resolution (1,920 by 1,832 px per eye), great performance all while being the cheapest (comparable devices can cost 3 times as much).
More on this here - https://victordibia.com/blog/vr-intro/
Making a Hello World VR App
I had 3 days (mostly evenings) over the winter break to explore , and was able to build a simple 3D environment where the user can navigate (a large plane and a few cube based walls), use the Oculus controller to grab one of 3 balls and move them around the environment.
My high level findings are summarized below:
Developer tooling for VR is mature enough and great documentation exists such that you can get fairly productive in 3 days. The bulk of my time went to learning the Unity 3D engine and dealing with some outdated documentation/tutorials on the Unity XR Interaction Toolkit. Tooling appears to be better for windows machines compared to a Mac; atleast the Oculus Link which enables live debugging appears to be not supported on a Mac.
Unity 3D works well with the Quest 2 for developing 3D experiences. An alternative is Unreal Engine which is also supported.
XR Interaction Toolkit (XRIT) is Unity's effort to standardize 3D and UI interaction for AR and VR applications built with Unity (e.g., mapping head mounted display to camera view, mapping controller inputs etc, with support for multiple hardware platforms like the Quest 2, Vive, etc.). However, XRIT is currently in active development (preview). I noticed significant changes to the api over the last 12 months, making many tutorials hard to follow. Luckily, this aspect of creating your Unity app only needs to be setup just once, the bulk of effort still goes to building the game/experience.
If you are new to VR (and Unity3D), there are many terms that are worth getting familiar with - scene, game objects, camera, camera rigs, rigidbody, scripts, player controller, prefabs, etc. Believe it or not, the Unity ecosystem is quite mature that they have documentation on all the terms you'll need to know. Great resource to peruse.
More details here - https://victordibia.com/blog/vr-intro/
Some 3D Modeling in Blender
I also experimented with learning to use the free open source tool Blender for 3D modeling. I followed the classic donut tutorial by Andrew Price and created the donut below - sumptuous heh?
Interestingly, 3D model assets created in Blender can be exported rather easily for use on the web, rendered using Three.js.
How to Render Blender 3D Models in React with React Three Fiber
Also, here is a link to something I am calling my Blender Playground - A place to keep track of any interesting Blender projects I am working on. Each model is rendered in the browser, so you can interact with them (drag, rotate, zoom).
Oh, and I wrote a short year in review post discussing some of the things that got done last year!
Happy New Year!