Meta offers a glimpse of its supposed iPhone killer: Orion

For years, Silicon Valley and Wall Street have questioned Mark Zuckerberg’s decision to invest tens of billions of dollars in Reality Labs. This week, Meta’s wearables division revealed a prototype of its Orion smart glasses, a form factor that the company says could one day replace the iPhone. This idea seems crazy…but maybe a little less crazy than it was a week ago.

Orion is a headset prototype combining augmented reality, eye and hand tracking, generative AI and a gesture-sensing bracelet. Thanks to LED microprojectors and silicon carbide lenses (which are quite expensive), Meta appears to have solved a long-standing AR display challenge. The idea is that you can look through Orion – you know, like a pair of glasses – but also see application windows projected onto the lenses that appear to be integrated into the world around you. Ideally, you can use your hands, eyes, and voice to navigate the environment.

But to be clear, Meta’s Orion smart glasses are bulkier than your average readers, reportedly cost $10,000 apiece, and won’t be available for sale anytime soon. We’re talking years from now. All of Orion’s technology is relatively young, and it needs to get cheaper, better, and smaller to make its way into a pair of smart glasses you can buy at the mall. Zuckerberg says the company has already been working on Orion for 10 years, but there is still no path to a sellable product.

However, Meta isn’t the only company trying to replace your smartphone.

This month, Snap unveiled its latest generation of Spectacles smart glasses, which are larger than Orion and have a more limited field of view. A former Snap engineer called the latest Spectacles “blatantly bad” – although you can order them. Google hinted at its I/O conference in May that it was also working on a pair of smart glasses, perhaps a rehash of its failed Google Glass experiment from the last decade. Apple is reportedly working on AR glasses that look a lot like Orion. And we can’t rule out Jony Ive’s new startup, LoveFrom, which he recently confirmed is working on wearable AI with OpenAI (although we’re not sure if it’s glasses, pin or anything else).

What’s brewing is a race between Big Tech’s richest companies to create a sleek pair of smart glasses that can do everything your smartphone can do — and, hopefully, something more. Meta’s prototype clearly showed two things: There is something there, but we’re not there yet.

These devices are a notable departure from the Quest virtual reality headsets that Meta has offered for years now and Apple’s Vision Pro. There are a lot of similar technologies involved, like eye tracking and hand tracking, but their usage is completely different. VR headsets are bulky, uncomfortable to wear and make people nauseous when looking at the screens. Sunglasses and eyeglasses, on the other hand, are relatively comfortable to wear and millions of Americans use them daily.

To Zuckerberg’s credit, he’s been pushing the glasses form factor for quite a long time, when it certainly wasn’t popular to do so. Meta’s CEO has long been reported to hate having his popular social media apps accessible through Apple’s phones (which may have led to the ill-fated Facebook phone). Now Meta’s competitors are also getting into IT when it comes to glasses.

Meta’s first investments seem to be paying off. Zuckerberg gave an Orion presentation Wednesday that we won’t soon forget, filling a room full of skeptical journalists with electricity and enthusiasm. TechCrunch has yet to demo Orion, but early reviews have been very positive.

What Meta offers today is the Ray-Ban Meta: a pair of glasses with cameras, microphones, speakers, sensors, an LLM on the device and the ability to connect to your phone and the cloud. The Ray-Ban Meta is much simpler than Orion, but relatively affordable at $299 – actually not much more than a pair of regular Ray-Bans. They look a bit like the Spectacles 3 launched by Snap a few years ago, although the Ray-Ban Meta glasses seem more popular.

Despite the big differences in price and capabilities, Orion and Ray-Ban Meta are more related than you think.

“Orion is truly the future, and we ultimately want to have the full holographic experience. You can think of Ray-Ban Meta as our first step there,” Li-Chen Miller, Meta’s vice president of product who leads its apparel team, said in an interview with TechCrunch. “We really need to nail down the basics, like making sure it’s comfortable, that people want to wear it, and that they find value in it every day.”

One of the things Meta is trying to achieve with Ray-Ban Meta is AI. Currently, smart glasses use Meta’s Llama models to answer questions about what you see in front of you, taking photos and running them through the AI ​​system alongside a user’s verbal requests. The Ray-Ban Meta’s AI features today are far from perfect: latency is worse than OpenAI’s advanced voice mode; The meta-AI requires very specific prompts to function properly; he hallucinates; and it doesn’t have tight integration with many apps, making it less useful than just getting my iPhone back (perhaps by Apple’s design). But Meta updates coming later this year attempt to address these issues.

Meta has announced that it will soon launch live AI video processing for its Ray-Bans, meaning the smart glasses will stream live video and verbal requests into one of Llama’s multi-modal AI models and output real-time verbal responses based on this input. It also gets basic features, like reminders, as well as more app integrations. This should make the experience much smoother, if it works. Miller says these improvements will be applied to Orion, which runs on the same generative AI systems.

“Some things make more sense for one form factor than another, but there is definitely some cross-pollination,” Miller said.

Likewise, she says some Orion features could apply as her team focuses on making more affordable AR glasses. Orion’s various sensors and eye trackers are not cheap technologies. The problem is that Orion needs to improve both And more economical.

Another challenge is typing. Your smartphone has a keyboard, but your smart glasses don’t. Miller worked on keyboards at Microsoft for nearly 20 years before joining Meta, but she says Orion’s lack of a keyboard is “liberating.” She says using smart glasses will be a more natural experience than using a phone. You can simply talk, gesture with your hands, and look at things to navigate Orion; all things that come naturally to most people.

Ironically, the iPhone is another device criticized for its lack of a keyboard. Former Microsoft CEO Steve Ballmer mocked the iPhone in 2007, saying it wouldn’t appeal to business customers because it didn’t have a physical keyboard. But people have adapted and his comments seem naive more than 15 years later.

I think making Orion feel natural is definitely more of a goal than a reality at this point. The Verge notes in its hands-on review that the windows sometimes filled the entire glass of the glasses, completely obstructing the user’s view of the world around them. It’s far from natural. To get there, Meta will need to improve its AI, typing, AR, and a long list of other features.

“For Ray-Ban Meta, we narrowed it down to a few things, and it does them really well,” Miller said. “Whereas, when you want to build a new futuristic computing platform [with Orion]we have to do many things and do them all very well.