Watch Meta perform a full-body VR tracking demonstration with only a Quest headset.

With the most recent developments, Meta Reality Labs is making significant progress in avatar rendering by fusing machine learning (ML) with sensor data from Quest VR headsets to display your entire body, including arms, legs, the chest, and the head. As a result, the stances and motions of a wearer of a Quest 2 headset are very realistically and precisely portrayed.

 

This is fairly remarkable considering that the only information used to estimate leg motion and position was the positions and orientations of the two Quest VR controllers and the headset itself. This amazing technique works without the usage of external cameras or tracking bands on the legs. Alexander Winkler, a Meta Research Scientist, tweeted links to various videos, a more in-depth YouTube video, and a scholarly paper on arXiv.

 

In a scientific sense, this is known as motion tracking from sparse sensors and ML is particularly adept at extracting meaningful information from very little, if there are enough dependencies between what is known and unknown. When walking or sprinting, we swing our arms for balance, therefore arm movement provides a fair indication of what the legs are typically doing. The ML system can properly forecast the majority of human motion when combined with head tilt and direction.

 

More conventional methods of tracking limbs require additional hardware, such as bands with wireless beacons worn at different positions on your legs to relay position and motion data, or reflective markers placed on your legs and torso that are detected by external cameras.

 

Although these techniques are effective, they are often supplied as expensive accessories and aren’t fully supported in most apps and video games. Developers will be more willing to incorporate body tracking into their games and applications if Meta can provide such impressive results with a Quest 2 headset.

 

In spite of these noteworthy developments, Meta Reality Labs acknowledges that more work remains to be done. If you move quickly enough, the ML model cannot identify your stance properly. The computer has trouble estimating unusual postures, and the movement won’t match if the virtual environment has impediments that don’t exist in reality. Although the overall impression appears to be excellent, it would be a pleasant addition to be able to view a whole body when speaking with pals in virtual reality as opposed to floating torsos.

 

Ideally, this technology will be accessible very soon. The release of full-body avatars seems to coincide perfectly with Meta’s anticipated unveiling of the Quest Pro headset in the coming weeks. A better sense of presence for your friends, family, and coworkers may soon be considerably improved thanks to the Meta Quest Pro’s ability to track eye movement and facial emotions.