As with Memoji, a wearer’s real face could be mimicked by a cartoon avatar
It was a surprise when Apple revealed that Apple Vision Pro would display an image of the wearer’s eyes on its outside displays, but that’s nothing compared to what else Apple has considered showing.
In Russell T. Davies’s drama “Years and Years,” teenagers wear devices that float large digital avatars in front of their whole faces, for instance showing them as cartoon cats. Apple seems to have seen the show because a new Vision Pro patent application even mentions cats.
Apple Vision Pro already has this EyeSight system where a view of the wearer’s eyes are shown on the outside of the headset. The patent application, “Wearable Device For Facilitating Enhanced Interaction,” details how this works, but then goes further and specifically to avoid reducing “the ability of external observers to interact and engage with the wearer.”
It’s a quite short patent application, relatively speaking, running only around 6,000 words and with 20 or so illustrations. But, it is replete with proposals that would extend EyeSight much further.
“[If] the wearer is playing a game in which they are a cat,” it suggests, “the user’s eyes or face (as displayed) may be modified to appear as a cat.”
“[Instead] of merely displaying a live video feed of the wearer’s face or eyes,” it continues, “the HMD [head-mounted display] may display a cartoon or other digitally rendered face or eyes that track and/or mimic the user’s eye movements, expressions, and the like.”
“As another example, the display of the wearer’s face or eyes may be modified based on aspects of the virtual environment being experienced by the wearer,” says Apple. “Thus, if it is raining in the virtual environment, rain may be visible on or in front of the user’s eyes or face.”
Detail from the patent showing a Vision Pro headset having a better time than everyone else
Apple is big on cartoon representations of Vision Pro wearers, and so many of the descriptions sound like more elaborate versions of Memoji. “[If] the user expresses an emotion such as surprise (as detected by visual cues, biometric data, user-command, or any other suitable technique),” it proposes, “the displayed eyes may be enhanced, modified, exaggerated or the like to show a cartoonish, embellished indication of surprise.”
“Similarly, fi the wearer is angry, the eyes shown in the display may be modified, exaggerated, or the like to appear red or flaming,” says Apple.
Soon you won’t have to look at people you’re having an argument with.
There’s also one odd suggestion about what the wearer of Vision Pro might be shown when someone approaches them. “The processor may be further configured to determine whether the observer matches a known contact of the wearer,” says the patent application, “and, in response to determining that the observer matches a known contact of the wearer, modify the captured first images by obscuring identifiable physical features of the wearer.”
So if you’re embarrassed to be seen wearing a giant headset on your face, Vision Pro could help disguise you. Remember to deepen your voice when you talk.
Aside from this one element, the purpose of the whole patent application is to facilitate “increased interaction between the wearer and the external observer.” Even though sometimes, the proposal suggests that the headset could show a “do not disturb” graphic to those pesky external observers who just want a go on the Vision Pro.
Extended future Vision Pro headsets
In Apple’s TV spot for Vision Pro there is a clip from “Star Wars” with Luke wearing a blast helmet. It’s a regular helmet but there is a solid wall of metal or plastic covering the whole of his face.
Apple has seen “Star Wars.” Perhaps this is an old idea or one that was later scrapped, but this patent application’s drawings contain a version of Vision Pro where it did obscure the whole face, like a blast shield instead of swimming goggles.
With the blast shield down, you can’t see anything. But external observers can.
“[This example HMD] may be substantially the same as [Vision Pro] and may include the same and/or similar components… but may be configured to cover a greater portion of the wearer’s face,” says Apple. “Because [it] covers a greater portion of the wearer’s face… the outward-facing display may be larger… and may therefore display more features of the wearer’s face, including for example the wearer’s nose and mouth.”
Even in a reasonably short patent application, there are practically countless variations of the ideas. As well as the cartoon eyebrows or the blast-shield-like EyeSight, many ideas are more subtle.
For instance, the EyeSight screen showing a user’s eyes could also subtly alter the lighting of that image to match the ambient lighting in the room.
So far Vision Pro has gone for quite subtle effects such as this, and yet EyeSight has already been criticised for how it presents a false image of eyes to observers. Maybe Apple sees Vision Pro as our first step into a wider world of all kinds of things being displayed on the outside of the headset.
This patent application is credited to eight inventors, including Jony Ive and Alan Dye. Ive is listed as the first inventor, but it’s unlikely that he worked on this after his leaving Apple.
Instead, his work is probably in the roots of this patent application, which has history of versions dating back as far as June 2017.
