Apple announced on Monday at WWDC 2023 its much anticipated Vision Pro augmented reality device, which CEO Tim Cook promised would be a revolutionary advance.
Apple made a pitch for how Vision Pro enhances remote work productivity by lifting collaborative interactions and workflow from a purely 2D, on-screen environment into a 3D spatial experience. The implication is that rather than putting the wearer into some virtual representation of an office or a collaborative environment, Vision Pro delivers tasks, information, productivity tools and collaborators into wherever space one happens to be in, whether it’s their actual office, their home or somewhere else.
During the keynote, Apple also announced iOS 17, macOS 14 Sonoma, iPadOS 17, new Apple Watch features and an array of hardware upgrades. Find out what I consider is the real star of WWDC 2023.
Jump to:
At WWDC 2023, Apple announced iOS 17 and the new experiences it enables across iPhone, FaceTime, Messages and AirDrop. The mobile OS also includes a journaling feature and a new way to view time and other information when an iPhone is set down and charging (Figure A).
Figure A
In addition, Apple touted the virtues of iPadOS 17, which include a redesigned interactive Lock Screen; widgets; and new features in PDFs and Notes, FaceTime and Safari.
Apple announced lifestyle-focused features for Apple Watch enabled by watchOS 10, including mapping topography, fitness and health features.
The company’s virtual reveal of the Apple Vision Pro, which will go on sale in early 2024, at $3,499, included a rundown of myriad new technologies driving a highly compact architecture. These include two postage-stamp sized visual fields with millions of pixels, all designed to engender an entirely immersive experience for work, entertainment and commerce without sensory isolation.
Cook characterized Vision Pro as the first Apple product designed to be looked through, not looked at.
“In the same way Mac introduced us to personal computing and iPhone introduced us to mobile computing, Apple Vision Pro will introduce us to spatial computing,” Cook said during the keynote address. The focus was not on how the goggles would usher wearers into a world of avatars in meta real estate perched on Maui’s cliffs, but on how this new headset has the versatility to augment reality and traditional applications by grabbing from a 2D screen and throwing them into space and letting your eyes and fingers do the walking, instead of a mouse.
Vision Pro can replace reality, the company was quick to note, demonstrating how the device could turn the sensory ordeal of a long-haul flight into a fully immersive cinematic experience — viewed from, say, the top of Pikes Peak.
Alan Dye, Apple’s vice president of human interface, said during WWDC 2023 that the key was not to isolate users from people around them as the device uses optical wizardry to display users’ eyes when someone is nearby. He said, “It reveals your eyes, providing cues to others, so it helps you stay connected.”
Apple executives demonstrated how the Vision Pro has technologies onboard that achieve these goals while providing security and privacy guardrails arising from captured facial and gesture data: numerous sensors, computation and AI, state-of-the-art circuitry, encryption and privacy caching of data.
Among the many sensing capabilities is LiDAR. It 3D maps surroundings for audio and visual optimization, high-performance eye tracking and supercomputer power using Apple Silicon tech in a dual-chip design, with the new M2 chip at the core. That chip runs in parallel with a brand new purpose-designed chip called R1, which is designed for real-time sensor processing using data from the headset’s 12 cameras, five sensors and six microphones.
An issue with VR headsets has been their tendency to induce nausea due to image delivery latency. Mike Rockwell, the vice president of Apple’s technology development group, said during the WWDC keynote that the system delivers images in 12 milliseconds, eliminating latency nausea. “So experiences feel like they are taking place live,” he said.
The EyeSight feature that allows others to see users’ eyes and vice versa uses a curved OLED panel. Rockwell explained that when using FaceTime, the system creates a representation of users during the initial setup.
Rockwell said developing a user avatar was critical for video conference functionality since no on-device camera could capture the user’s face. The company resorted to AI.
“We used machine learning to create a novel solution,” he said, explaining that during the user-enrollment process, a machine learning model trained on facial images creates a representation based on the initial scan of the user’s face. The ML model tracks users’ facial movements and expressions to build a realistic video-conference-friendly representation for FaceTime and non-Apple conferencing platforms.
VisionOS, the new operating system behind Vision Pro, is the first such system designed from the ground up for spatial computing, according to Rockwell. “Nothing else is like it. It is more than a new product — it’s the start of a new platform.”
Some experts worry that extended reality devices pose privacy issues because they capture physiological information about users, such as eye and head movement, that can be used to gain insights into individual preferences or biases.
Apple said there would be no such issues with Vision Pro because it locks away eye movement and physical behavior and the AR headset does the same with data about a user’s eyes garnered from the device’s unlocking system called Optic ID, which authenticates a user by scanning their irises.
The company said Optic ID is encrypted, never leaves the device, and works with features like Apple Pay and App Store purchases.
“In Apple Vision Pro, where you look stays private,” said Rockwell, explaining that data is stored in a separate background process. “Camera data processes at system level, so apps can’t see your surroundings,” he said.
En Kelly, a senior engineering program manager at Apple, said the Vision Pro platform includes Apple’s first 3D camera with spatial audio, delivering a 3D immersive experience or spatial video. “It also delivers an extraordinary entertainment experience,” she said, explaining that a turret dial allows control of the visual field and the ambient setting, as mentioned above.
“You can turn any room into a movie theater,” she said. “It’s built to give you a fantastic viewing experience. 3D movies… feels like you are looking into the movie. No other device can do this.”
Apple said there will be more than 100 arcade titles available on day one of the launch of Vision Pro. Disney CEO Bob Iger said Disney will create experiences “to bring our fans closer to characters they love in ways that were previously impossible.” He added that Disney+ will be available on day one of launch.
Although Apple is as well known for aesthetics and user experience as it is for computational versatility, the real star of Apple’s WWDC event was the invisible-to-user lineup of super-fast Apple Silicon system-on-a-chip platforms advancing under the M2 and now the R1 monikers — the former powering new MacBooks revealed during the event and the latter the computational powerhouse in the Vision Pro.
As undercards to the Vision Pro announcement, Apple revealed many hardware and software products, including a new Mac Pro ($6,999) and a new 15-inch MacBook Air ($1,299).
Kate Bergeron, the vice president of hardware engineering at Apple, touted the latter as the world’s thinnest laptop, at 11.5 mm top to bottom, noting that the new M2-powered computer is an animation powerhouse supporting a palette of some one billion colors. She said the computer is 12 times faster than the fastest MacBook Air.
Earlier this year, Apple launched the M2 Pro and M2 Max chips, according to Jennifer Munn, the director of engineering program management at Apple. But the “monster of a chip” and the one powering the new Mac Pro computer is the new M2 Ultra, which she said is the most powerful chip ever created for a personal computer. It is essentially a pair of mated M2 Max chips with a 24-core CPU that has 20% faster performance than the M1 Ultra (Figure B).
Figure B
