Experience Strategy
Amazon Lumen wordmark

An Experiment in Projected AR for the Smart Home



MY TOOLKIT: Millumin projection mapping / LeapMotion / AdobeCC / Projector






Web 4.0 — the ambient internet — exists, as humans do, in 3D space. But our current interfaces are either audio-only smart speakers, or redundant, anchored smart screens.

We can do better by using projection to augment a voice UI when needed.

ALLIES: Minimalism, mobility, universal design
ENEMIES: Screens, tethers, smart fridges



Using the latest advances in sensors, laser-based projection and machine vision, any surface can become a display, and any 3D space an interface.

Diagram explaining that a laser projector is connected to 3D scanning sensors, which are linked to an Amazon echo and cloud-based machine vision.


Amazon Lumen Onboarding screens, showing the initial set up of the device, 3D scanning of the environment, manual adjusting of the results of the scan, and a welcome screen

Amazon Lumen is a hypothetical member of the Echo family of smart home products. It allows the Alexa smart assistant to project context-specific visuals and interactions to the user. These visuals can address specific pain points, increase accessibility, provide ambiance and offer assistance.

Because any surface in range can host a projected visual, users don't need to be tethered to a screen to have the internet at their fingertips.



Use Case: List Management

Opportunity: 43% of smart speaker owners use the device to manage shopping lists, but few use them to place shopping orders.

Pain Point: List management is currently cumbersome, and it is still necessary to use a screened device to manage items or place orders.

This projected concept depicts gestural control over list content, digital Amazon Dash buttons, a quick compare pop-up and a one-”click” ordering system.


Use Case: Visualization

Opportunity: 64% of users ask their smart assistants for weather forecasts. Large percentages also seek other live information, such as traffic updates or news.

Pain Point: It can be difficult to retain complex information conveyed audibly.

This prototype explores how projected visuals convey information quickly while showing how a visual mood can be presented through animation and ambient light.


Use Case: Browsing

Opportunity: 70% of users listen to music via smart speaker. Audio book consumption is also popular, with 30% of frequent audio book users using smart speakers to access them.

Pain Point: Audio commands for music and books work well, but browsing is impossible without a paired screened device.

This projected, browsable bookshelf allows a quick look at available content and presents a purchase opportunity.


Use Case: Task Assistant

AI assistants are marketed for their utility in providing recipes, timers, and on-the-fly conversions. But even the most sophisticated model behaves as a cookbook that turns the page when you say "next step."

By using machine vision to understand where you are in a recipe, Amazon Lumen can give you timers, instructions, and reminders by anticipating what you need.


In this use case, we see a recipe being followed. Lumen detects the proper ingredient, displaying NEXT by the oil. After the oil is added, a temperature reading is projected on the pan and the instructions advance to step 2, getting a timer ready for a sauté.

Diagram showing the cooking experiment. A projected display gives directions, a timer and an indicator of the next ingredient.


Context = Utility

The initial use case experiments — lists, weather and media — offered some utility but were essentially projected versions of existing Web 3.0 interactions. 

The fourth experiment began to reveal the full affordances of the ambient internet, machine vision and projection. When a system understands the context of the task a user is trying to perform, it is able to offer timely or preemptive directions, warnings or assistance.

Projected AR or Wearable AR?

Not all of the experiments proved enough utility to be worth making the privacy trade-off inherent to sensors.

The future of smart projected AR lies, like that of wearable AR, in workshops, garages and kitchens, where the usefulness is high enough to be worth the installation.

The principles learned in this study, shared below, offer broad guidance for designing for the ambient internet in AR space, whether projected or wearable.

Design Takeaways



Take your accessibility guidelines on contrast and dial them up to 11. My projector was cheap, but even high-contrast projections will encounter issues with ambient light. If you're using lower-end projectors, contrast matters even more. 



A user who isn't physically touching an interface needs visual or audio feedback. For example, increasing scale as a user nears shows they have been detected, while blinking, color changing, or animating can show selection.


Consider projected AR if you need a social or shareable experience, a hands-free or screen-free experience. Choose traditional AR if you need a mobile experience or a high-resolution display. Choose wearable AR for the best of both worlds.


Process documentation and background information


Gesture Testing

An animated GIF showing a projected circle responding to the side-to-side movements of a human finger.
An animated GIF showing a projected circle moving down as a human palm waves in front of it.
An animated GIF showing a projected circle growing slightly as a human finger approaches it.

 I used a LeapMotion connected through Millumin to test gestures and visual feedback.

It is important that gestures feel natural and intuitive, require little memory or complexity, and are broadly uniform across cultures. For these experiments, only very simple gestures were needed. However, personal gestures could be used as shortcuts or as a physical equivalent to a "wake word" for an AI assistant.


The Internet Has Expanded Beyond the Screen

WEB 1.0

The birth and growth of the world-wide web.

The internet is rare.

Ex. Prodigy, AOL, Lycos, Netscape, Mad Maze


WEB 2.0

An explosion of interactivity, ecommerce and social media.

The internet lives on your desk.

Ex. MySpace, Amazon, Facebook, HomestarRunner

WEB 3.0

The internet goes mobile. Smartphones and apps proliferate.

The internet lives in your pocket.

Ex. Instagram, Snapchat, HQ, Candycrush

WEB 4.0

The internet becomes ambient. 5G, edge and cloud computing, and ubiquitous IOT sensors and AI proliferate.

The internet lives everywhere.

Ex. Alexa, Google Assistant

Sample Alt-UI Flows

A map of conversational interface flows, showing where voice UI and gesture UI might work together in three use cases: Setting the mood, showing lists, and helping cook.

Sketches & Ideation

A page from a sketchbook showing scribbled notes about home spatial AR and drawings of various home spatial AR imagery.
A page from a sketchbook showing scribbled notes about gestural design and drawings of potential gesture inputs.
A page from a sketchbook showing scribbled notes about home spatial AR and drawings of various possible product designs, including ceiling mounted projection and table based smart speakers.


An animated GIF showing the wireframe of a hand showing Will It Snow Today in American Sign Language


The same improvements to machine vision and gesture recognition that make Lumen viable can connect users of sign language to the power of the ambient internet.

An illustration showing light mapped onto a house plant in an orange pot.


Laser projectors can generate any wavelength, and machine vision can detect different plant species, auto-mapping the right amount of light to individual plants. No wasted energy, no starved herbs.

An image showing an arm with the bones and their names projected onto it.


Social use-cases for this include family-friendly games that use projection and machine vision to offer uniqe educational AR overlays.


The future is weird.