AR Display for Astronauts

2x national finalists in the annual NASA SUITS Challenge

TIMEFRAME

Sept 2022 - May 2023 (9 months)

ROLE

UI UX designer

TOOLS

Figma, Unity

DISCIPLINES

AR design, Interaction design, Usability testing

The Problem

01

How might we design an AR interface for NASA astronauts/engineers to explore the moon?

THE MISSION

Spacesuit User Interface Technologies for Students (SUITS) challenges students to design and develop an augmented reality (AR) display for NASA’s researchers to test for the sake of astronauts on future Artemis moon missions. The AR interface is built into the spacesuit and should provide an intuitive, seamless experience.

Microsoft HoloLens 2

Rover robot

Rock scanner

THE EQUIPMENT

THE TEAM

LEADERSHIP

Ashley Fan

Michael Wang

Jessica Young

DESIGN

Hermione Hsieh

Ryan Lee

Keya Shah

Dong Yoon Shin

Bill Xi

Bryce Yao

Linlin Yu (Me)

DEVELOPMENT

Julius Beberman

Jamie Chen

Danielle Kim

Martin Ma

George Xu

FACULTY

Michael Lye

Skye Ray, Evaluator 1

Kelly Mann, Evaluator 2

THE FEATURES

We’ll be implementing 4 sequential procedures to protect the user during their spacewalk.

Egress

Prepares astronaut’s exit from lander onto moon.

Navigation

Guides astronaut across the lunar surface.

Geological Sampling

Displays scientific info of lunar geology.

I designed this procedure!

Rover Commanding

Drives the moon buggy vehicle.

Physical display

THE USER

Our AR program is centered around NASA design evaluators assuming the role of an astronaut. Evaluators will test our program in a simulated environment called the Rock Yard, which is similar to the conditions on the moon.

The Solution

02

Task progression

Egress

Map and compass

Nav

Placing a user waypoint

Placing a ROVER waypoint

Rover

Performing sampling (Click to enlarge)

Geo

The Research

03

Less Information, More Confirmation

2022’s interface required tediously tapping buttons for access to info, which overwhelmed users and blocked their vision during testing. However, the previous year’s research helped us start off strong.

Physical > Virtual

The moon landscape limits the ability to walk, so astronauts jump around. We needed to design a display that complements bodily actions.

James H. Newman

Former NASA Astronaut

Steve Swanson

Retired NASA Astronaut

Peter H. Schultz

Geological Sciences

One way to empathize: Mocking up UI on transparent acetate paper and holding it in front of your face!

Jonathan Levy

Cartographer

Isabel Torron

UX Designer

Alejandro Romero

VR/UX Specialist

Jim Head

Geological Sciences

James Russell

Planetary Sciences

Empathize with Experts

Confidence drives decisions in a life-or-death situation. We went over the interviews with eight specialists from the 2022 challenge to launch our project:

(Click the arrows to see more)

“What’s closer to the user is more important, which means it should be higher in the visual hierarchy”
-Romero
“Be as minimal as you can. It’s not great for all controls to disappear [on the AR display], but it can help with organization”
-Levy
“With bulky gloves, there's no tactile feedback”
-Swanson
“The main challenge is the gloves because they are airtight and large, so mobility is tough”
-Torron
“Use bigger hand movements!”
-Romero
“A procedure list and suit status are necessary”
-Torron
“A checklist relies on memorization and the current one looks like a poorly designed book”
-Newman

Prototyping

04

As a team with 4 subteams, we had a hard time explaining our ideas to non-designers and grasping the idea of spatial limitations. We learned to write callouts on Figma so developers can better understand our decisions and to wear the HoloLens directly to test our solutions.

01 Sketches

“How do we design a moving display in a 3D space?” Draw it on paper and hold it at arms length as you walk.

02 User Flows

“How do we divide work but ensure a linear sequence for the user? Brainstorm key features in 4 teams & regroup.

03 Wireframes

“How do we validate our ideas to user testers and devs?” Lay out lo-fi frames into one clickable wireframe.

04 Unified Design

“How do we make a minimum viable product?” Iterate for 3 months and implement hi-fi design into MRTK3.

Interviews

05

We conducted preference research by asking RISD faculty to walk through our clickable wireframes. We asked them to “think out loud” and had to intervene/skip tasks due to a bad understanding of the simulation. We learned the importance of briefing - to not go into it blind!

Matthew Bird

Senior Critic, RISD Industrial Design

Cheeny Celebrado-Royer

Assistant Professor, RISD

Leah Beeferman

Assistant Professor, RISD

Professor Insights

“I don’t know what’s real and not real…Am I supposed to click on this?”
-Bird
“What’s the red triangle? Warning? Arrow? Volcano?”
-Celebrado-Royer
“It would help to merge ROVER command into the navigation map, since both you and the vehicle are navigating”
-Bird
“The key thing is about making the buttons consistent in word and icon choice to make it easily understandable every time”
-Celebrado-Royer
“It’s difficult to see the white icons. There’s so many”
-Beeferman
“Icons have no outline. Words are too small, line weight is too thin. Warning signs & top right notifications are too big and block my view”
-Bird

What do these symbols mean?

Do these visuals inspire confident decisions?

LINKED USER FLOW

(Click the arrows to see more)

Lo-Fi to Hi-Fi

06

AR DISPLAYS

I worked with two designers and two developers to implement Geo Sampling. We made Figma wireframes and uploaded them into MRTK.

DESIGN SYSTEM

I helped design the compass, hazard button, and geological sampling components. We had originally used MRTK’s default assets as placeholders to test usability on the headset.

Development

07

Our devs worked tirelessly to implement the designs into the Mixed Reality Toolkit 3 (MRTK3). I met with devs Danielle Kim and Jamie Chen twice a week while communicating via Zoom, Slack, & GitHub. .

The HoloLens supports both the Unreal & Unity engines. For the challenge, the devs used Unity.

Using MRTK3 to implement a range of functionalities such as spatial mapping, gesture recognition, and object manipulation.

User Testing

08

“It’s very finnicky”

MARCH 2023 in the dark

To extract errors in our design and code, we conducted in-environment testing called “Human-In-The-Loop (HITL)” at two local state parks to simulate the moon environment. (Click to see more)

Testing color contrast & font size.
Mimicking the astronaut on uneven terrain.
Both designers & developers making refinements after trying on the headset.
Going over errors in development.
Testing in a pitch black environment with extreme light sources.
Using car headlights to provide harsh lighting.

“I wish there were backup options for when things fail”

MAY 2023 at NASA

My team and I were invited to Houston to test & present our design at Johnson Space Center from May 18th to the 23rd. (Click to see more)

TESTING
On-site testing was conducted at the JSC Rock Yard. Testing was sectioned into briefing, testing, & debriefing.
TESTING
Our design successfully loaded in the HoloLens! Team member Danielle Kim guides Skye Ray through the 1st procedure.
DEBRIEFING
Our pain points include lack of confirmation buttons, button and info differentiation, and backup options for vital actions.
BRIEFING
The developers connect to the telemetry stream to view live feedback in our design. When a switch is flipped we get a notif!
BRIEFING
As one of five participants in the 1-week marathon, I revamped the design and prepared our exit pitch to panelists.
BRIEFING
Kelly Mann is equipped with our high-beam lights, a piece of handmade hardware that assists the user during testing.
BRIEFING
Team member Danielle Kim teaches evaluator Kelly Mann the action that corresponds to our palm menu feature.