Vision Pro: Eye tracking theoretically allows password reconstruction

There is a saying that the eyes are the window to the soul. At least you can tell from them what mood someone is in and how they feel at the moment. American IT security researchers have now shown that you can use your eyes to find out what someone is typing and thus gain access to sensitive data such as passwords, PINs or the contents of private messages. The only theoretical attack currently occurs on Apple’s mixed reality headset Vision Pro.

Advertisement

one under the name GAZEploit The published attack uses Apple’s Persona function. They did not need access to the headset. Instead, they analyzed eye movements of virtual 3D avatarswhich the Vision Pro captures via internal cameras. Personality avatars can be used, for example, in Zoom, Microsoft Teams, Reddit, Slack, Tinder, X, Skype or FaceTime calls. They replace the missing video image of the Vision Pro user;

Avatar reveals keystrokes

Reconstructing the input was possible because you can also use eye tracking on the Vision Pro as an alternative to typing directly with your hands in the air. If you have the right letter in view, it can be selected with a finger gesture (tap with thumb and forefinger), which is more pleasant for some users.

Knowledge of what the user is actually doing is reserved for the headset wearer; however, anyone who taps their eyes with their avatar while participating in a Zoom call, live stream or FaceTime call is clearly revealing more than expected. Users’ virtual avatars reflect the users’ facial and eye movements.

Biometric data as an attack vector

The attack relies on two pieces of biometric data that the researchers extracted from a video recording of an avatar: the aspect ratio of the eyes and the estimated direction of gaze. First, the researchers used a neural network to identify patterns that could be used to tell whether someone is typing. To do this, they fed the deep learning model with video recordings of avatars of 30 different people typing with their eyes on a QWERTY keyboard in the Vision Pro.

Mobile operators set up joint venture for network APIs

They then used geometric calculations to figure out where and what shape the users held their keyboards in the air in front of them. If the video footage was good enough to reconstruct the position and shape of the keyboard, it would be possible to recognize the following keystrokes, said one of the researchers involved for the American technology magazine Wired,

The gap is closed, the avatar is hidden

Apple has identified the vulnerability, which has been assigned CVE ID 2024-40865. It is ok now And the fix was delivered to users with a software update to VisionOS 1.3. However, the company is taking a brutal approach: if you use the virtual keyboard while participating in a video call or live stream, the glasses simply interrupt the sharing of the avatar, meaning it becomes invisible. Apparently Apple couldn’t think of any other solution initially.

Although the attack was developed in the laboratory and no Vision Pro users were hacked as a result, malicious attackers could theoretically exploit the glasses’ biometric data leak when developing a similar method. The researchers’ work shows only one specific risk with gaze-based typing. Cheng Zhang, an assistant professor at Cornell University who studies wearable devices that interpret human behavior, told Wired that this exploit is just one possible example. Users of wearable devices often overlook how much information the devices can collect and reveal about them and the data security risks that this poses.

(KST)