avatars, AR and brain-computer interface

Image: Meta

Der Artikel kann nur mit aktiviertem JavaScript dargestellt werden. Bitte aktiviere JavaScript in deinem Browser and lade die Seite neu.

At Meta Connect 2022, Meta presented new research results in the field of virtual and augmented reality. An overview with video examples.

Meta’s research is designed to last ten years or more and push the boundaries of what is possible today in technologies such as virtual reality, augmented reality and artificial intelligence. At Meta Connect 2022, the company previewed research in many areas, from Meta’s AR headset to neural interfaces and 3D scanning to photorealistic codec avatars.

Augmented reality

Metas aims to launch a sleek, visually appealing yet powerful AR headset in the coming years. Since the technical challenges in terms of miniaturization, power, battery capacityand waste heat are great, Meta pursues a dual strategy in its development.

“Glasses have to be relatively small to look and feel good. So we approach the construction of augmented reality glasses from two different angles. The first is to take all the technology we need for full-AR glasses, and then work to adapt it to the best glasses form factor possible. The second approach is to start with the ideal form factor and work to incorporate more and more technology into it over time,” said Mark Zuckerberg in the keynote.

The first effort is codenamed Project Nazare, while the second is a joint project between Meta and EssilorLuxottica, the world’s largest eyewear maker. This partnership has already resulted in one product: the Ray-Ban Stories, which offers several smart features but does not have an integrated screen.

During Meta Connect 2022, Meta and EssilorLuxottica provided an update on their data glasses project and cooperation:

  • The Ray-Ban Stories will soon have the ability to call contacts hands-free or send an SMS via a software update.
  • Another new feature is a feature called Tap Spotify. “You’ll only have to press and hold the side of your glasses to play Spotify, and if you want to hear something different, press and hold again and Spotify will recommend something new,” Meta writes.
  • EssilorLuxottica Head of Wearables Rocco Basilico announced during the keynote that his company and Meta are working on a new helmet this will open a “portal to the metaverse”. Will the next generation of Ray-Ban Stories come with a screen? Zuckerberg and Basilico left that open.

What about Nazare Project?

At Meta Connect 2021, Meta simulated what a view through Project Nazare might look like. This year, Zuckerberg delivered another teaser of the AR headset without showing it.

Meta’s CEO walks down a hallway with the device and controls it using an EMG wristband. Apparently you can see a see through the Nazare project.

Zuckerberg messages Meta Research Manager Michael Abrash and records a video, both using micro-gestures. This is made possible by the EMG Wristband, which intercepts signals from the motor brain at the wrist and converts them into computer commands using AI. Meta considers this type of interface to be the most important AR operating concept of the future in addition to voice control and hand tracking.

Zuckerberg did not say when Project Nazare might appear. According to a report, Meta plans to unveil it in 2024 and market it in 2026.

Neural interface

Another block in Meta’s search update concerns the aforementioned EMG bracelet. Meta relies on a combination of this technology and personalized AI support for the AR interface of the future, which recognizes the context of a situation and an action and proactively supports spectacle wearers in their daily lives. This should enable an intuitive, nearly frictionless interface between humans and computers.

“By combining machine learning and neuroscience, this future interface will work for different people while accounting for their differences in physiologies, sizes, etc., through a process called ‘co-adaptive learning,'” writes Meta.

A video illustrates this. We see two Meta employees playing a simple arcade game via an EMG bracelet and movements of their fingers. Note that they use slightly different gestures – the artificial intelligence learns signals and movements and generates an individual model.

“Each time one of them performs the gesture, the algorithm adapts to interpret that person’s signals, so that each person’s natural gesture is quickly recognized with high reliability. terms, the system improves over time to understand them,” writes Meta.

The more the algorithm is trained, the less the hands and fingers have to move. The system recognizes the actions that the person has already decided upon by decoding the signals at the wrist and converting them into computer commands.

AR navigation for the visually impaired

Meta is working with Carnegie Mellon University (CMU) on a research project to help visually impaired people navigate complex indoor environments.

University researchers used Meta’s Project Aria detection goggles to scan Pittsburgh airport in 3D. They used this 3D map of the environment to train AI location models. Thus, the NavCog smartphone application, developed by CMU, can guide users more safely through the airport by relaying audio instruction. The following video explains the technology.

Simplified 3D scanning

Mixed reality headsets like Meta Quest Pro display the physical environment in the headset. They cannot yet scan objects and save them as 3D models. If that were an option, it would be possible to bring real objects into virtual environments.

“It’s hard to create 3D objects from scratch, and using physical objects as models might be easier and faster. But there is no transparent way to do this today, so we are looking at two different technologies to help solve this problem,” writes Meta.

The first uses machine learning, called Neural Radiance Fields or Nerve in short, to create an extremely detailed 3D object from a few photos.

The second technology is called Reverse rendering. Objects scanned with this method react dynamically to lighting and physics in VR environments.

A disadvantage of both technologies is that they do not yet work in real time. However, Meta sees them as important steps on the way to simple 3D scanning of physical objects.

Codec Avatars

Photorealistic digital dating – for Mark Zuckerberg, this is the flagship application for virtual and augmented reality.

To this end, Meta has been working for many years on so-called codec avatars: digital alter egos that hardly differ in appearance from the human original.

During Meta Connect 2021, Meta showed off second generation codec avatars and full avatars. This year there was another technology update.

Avatars 2.0 codec can now switch between virtual outfits and are even more expressive. To demonstrate the improved expressiveness, Mark Zuckerberg had a Codec avatar created for himself. The following video shows what technology is doing now.

One of the biggest problems for the commercialization and appropriation of codec avatars is their complex creation: users would have to be scanned in a special 3D studio.

To simplify the generation of a personal codec avatar, Meta is working on Instant codec avatars. All it takes is a two-minute face scan with a smartphone. The following video illustrates the registration process.

The downside to this process is that the finished avatar doesn’t look as realistic as Zuckerberg’s, and it still takes hours to get the avatar created and ready to use. However, Meta is trying to speed up the process.

Meta Connect 2022: Watch the Youtube Search Update

Meta points out that the projects show the research and the technologies don’t have to end up in the products. “Still, it’s a glimpse of the direction the technology will take over the next five to 10 years,” Meta writes. You will find below the video extract which presents the innovations presented in this article.