Apple invents a new type of user interface for its upcoming mixed reality headset that uses floating interactive orbs

Mr. Tomlinson Mr. Holman is an American film theorist, sound engineer, and inventor of motion picture technologies, including the Lucasfilm THX sound system. He developed the world’s first 10.2 sound system and has been employed by Apple since 2011 as a “Distinguished Engineer”. Today, the United States Patent and Trademark Office published a patent application from Apple titled “Method and Apparatus for Presenting a Synthesized Reality User Interface”. Mr. Holman is listed as one of the inventors. For this reason, our report will focus on the user interface relative to a music application interface, although Apple explains that this new concept of user interface could apply to documents, video files, Safari / web pages and Moreover.

Overall, Apple’s patent covers methods of present a user interface in synthesized reality (SR), comprising moving a data item in the SR user interface, grouping data items in an SR user interface, selecting groups of data items in an SR user interface, and playing represented audio files by data elements in an SR user interface. SR is best known for representing VR, AR and Mixed Realty environments.

Although the invention may relate to an iPhone or iPad, the majority of the patent relates to interfaces for Apple’s future MR headset.

Apple notes that navigating through a large number of data items (such as audio files, video files, document files, or web pages) to locate a particular data item can be cumbersome. The invention covers an SR user interface that can be used to open one data item among a plurality of data items. In various implementations, presenting the SR user interface includes displaying an SR volumetric environment including SR group representations at various locations in the SR volumetric environment around the user.

Apple’s patent FIG. 1B below is a block diagram of an example operating architecture, with the user wearing an MR headset; FIG. 6A illustrates an SR #600 volumetric environment from the perspective of the user wearing the MR headset. It also illustrates a user interface using tooltips or orbs. Although the examples provided below are for music, the interface can be used for documents, web pages, video files, and more.

Apple’s patent FIGS. 6B and 6C continue with the concept of a musical user interface using orbs or bubbles of information with which the user can interact, as shown in patent FIG. 8A where the user uses a two-handed gesture to enlarge the orb as needed. The user can also apply a pinch gesture with one hand.

3 Apple HMD GUI using orbs or tooltips that the user can interact with

You could spend hours reading and reviewing the dozens of examples of this new HMD UI concept. For the curious, see Apple’s patent application 20210208688.

Considering that this is a patent application, the timing of bringing such a product to market is unknown at this time.

10.51FX - Patent Application Bar