OVERVIEW
Context:
Live digital performance creators have endless creative ideas that they wish to test during workshops and rehearsal, but limited time to test all the possibilities. Our client is looking for a digital solution to save rehearsal cost and time.
Challenges:
1. User research in a niche market
2. Designing user experience for cutting edge mixed reality technology
3. UX Design in COVID-19 outbreak.
Outcome:
Our team designed a high fidelity prototype for RingMaster Virtual Production System — a Mixed Reality unreal engine extension tool on the Magic Leap which enables show creators to alter placement, timing, and other properties of objects and effects during rehearsal.
My Role:
Lead UX researcher and designer
- User research
- Design UX Structure and Flow
- Usability Test
Client:
Lifelike & Believable (L&B) Animation Design
Project Duration:
A 14-week industry project from concept development to tested prototypes.
Tools:
Adobe Illustrator, Adobe Photoshop, Adobe XD
PROJECT DETAILS
Introduction
RingMaster Virtual Production System (VPS) is a set of open-source production tools which will enable show creators to design, implement, preview, and control a live performance integrating real-time motion capture, rendering, visual effects and projection along with stage lighting and practical on-stage effects. It is part of a two-year project to produce Shocap - the world’s first immersive in-game 3D simulcast of a live, real-time motion-capture circus performance.
Our Approach
Our lack of domain knowledge in live digital performance meant we needed to understand the industry thoroughly and quickly. We invested our time heavily on on-site observations, off-site interviews, and research.
We adopted a design-sprint approach at the beginning to brainstorm ideas and gather inspirations. As the project progressed, we used an evaluation heavy human-centred design approach to create better design and reduce uncertainty.
Background
What is live digital performance?
In the context of this project, live digital performance refers to a high-tech performance that involves live performers (dancers, acrobats, etc.) whose on-stage movements are tracked and rendered in real time to drive the movement of avatars in a digital production, complete with virtual props, effects, and scenery displayed on a projector screen behind the live performers.
An aerial hoop artist is pictured above, performing while wearing a motion capture suit in a rehearsal space with multiple infrared cameras tracking her movements. The tracked movements drive a fairy avatar in a colourful forest as shown on the projector screen in the background.
Live digital performance has two type of audience, live audience and remote audience.
Live digital performance is a niche type of performance art that employs real-time technologies to enhance the creativity and dramaturgy of a performance whilst maintaining the human connection between live performers and live audiences. In this way, the live audience is the primary audience that the director creates the experience for.
The Problem
Show creators have endless creative ideas for live digital performances that they wish to test during workshops and rehearsal, but limited time to test all the possibilities.
As a team, we visited Animatrik film design during our client’s workshop to gain more insights about the workflow and the challenges people were facing.
Based on our observation, I outlined the current workflow for workshop/rehearsal. The problematic steps are highlighted in grey:
- the director has to physically move back and forth between performers on stage and programmers working from the stage-side computer in order to accurately communicate desired changes.
- It takes several minutes to program and render a change to the virtual production. In this time, the production crew and performers are usually standing by for a cue to resume.
- The director and programmer must establish a level of mutual understanding of the creative vision to avoid getting trapped in the loop indicated at step 09, where the change that the director wishes to make is not understood or implemented correctly and must be re-done.
Problem validation and building empathy
The problem mentioned above is validated through:
Interview with show creators:
In the interview, one show creator mentioned the workshop has to stop about 15 times in the day to reset things and figure out what was working and what was not.
In another interview, one staff mentioned that renting the venue and equipment as well as having all crew on site can be costly. Reducing downtime could greatly improve cost effectiveness.
User test:
We set up a scenario, which resembles the one in the workshop at Animatrik Studio, with a projection screen on the wall and one “developer” in the project room. Users were given two tasks.
In the first task, they were asked to give verbal directions to the developer, and the developer will translate their instructions to 3D design.
In the second task, they were asked to use an AR application to design the scene themselves.
80% of the users prefer to design on their own rather than relying on the developer. 100% of the user expressed that communication with the developer was inefficient, inaccurate, and time consuming.
Our Goal
The goal of the RingMaster VPS tool is to streamline the director’s process of experimenting with, previewing, and saving changes to performance design during rehearsal. The modified workflow with use of the tool is shown below:
The director can be more self-sufficient in making desired changes.
The programmer is still needed to implement larger-scale changes, but would be more free to participate in the creative dialogue with the director during the rehearsal.
If rehearsal downtime can be shortened by enabling the director to preview changes in Mixed Reality, they will either be able to shorten their total rehearsal time, or test more possibilities in the allotted time to raise the quality and creativity of the performance.
The Process
Challenge 1. niche market
Within the field of performance arts, live digital performances occupy a niche market, as there is a lot of risk involved in using real-time technologies for large-scale productions. Accordingly, there is only a small user population that is creating live digital performances with motion capture technology. This limits the amount of user research and testing we can accomplish with primary target users.
Overcoming the challenge
Interview
Instead of solely focusing on the show creators working in live digital performance industry, we expand our research to people who are interested in but yet to enter the industry either due to the barrier of cost or technology.
Persona
We noted from our interviews that a spectrum of affinity for real-time technologies exists for show creators, with users like Claire being more technology-averse due to perceived complexity and risks involved in creating live digital performances. In comparison, users like Francois are more inclined to take risks and use complex technologies to explore new storytelling possibilities. For this reason, technology-curious to technology-obsessed show creators are our primary target user population.
User Journey Map
A user journey map was created to illustrate the goals, pain points, emotions, and touch point of our persona, Francois. It highlighted some key opportunities for our solution to streamline his current workflow.
Agile Statement
Based on the research findings, we reframed our problem statement, and refined our goal:
UX Architecture
Build on the requirements arising from user journey maps, I designed the preliminary ux architecture for RingMaster VPS.
Due to time and manpower constraints, we focused on developing a prototype of a vertical slice of the application as highlighted below.
Challenge 2: Technology
Mixed Reality is necessary to allow users to see both the actual performers and the virtual environment, objects, and effects overlayed on top. Magic Leap One Creator Edition is the chosen platform to develop the RingMaster VPS tool due to these devices being gifted to our client through the Epic MegaGrant.
However, use of this technology presents several challenges:
First of all, Magic Leap has a very limited field of view. Anything at the edge of the screen will be cut off, which interrupts the immersive experience.
Secondly, the controller does not have an ergonomic design. It has limited input methods, and accessing multiple buttons at once can be very uncomfortable.
Thirdly, we were unable to use hand gesture controls. Our tool is meant to be used during rehearsals when performers are on stage. Magic Leap could easily pick up their or the director’s natural hand gestures by mistake.
Overcoming the challenge
Market Research
A range of XR applications were studied for features and design principles that were applicable to our RingMaster VPS tool, such as Mindshow, Tvori, and Finger Food Studio’s Scenographic Tool for Cirque du Soleil.
TVORI
Mindshow
Finger Food Studio’s Scenographic Tool for Cirque du Soleil https://www.cnet.com/news/microsofts-latest-hololens-partner-cirque-du-soleil/
Detail analysis were made for each case studies, including brief description, interesting features, pros, cons, and possible project implementation.
Design development and evaluation
Ideation
Instead of carrying out traditional group brainstorms, we used Sprint to generate ideas. We started the day with lightning demos and followed by the Four-Step Sketch to generate ideas.
My sketch received the most votes and hence was later used as the starting point of our design.
Low-fi prototypes and user test
I partnered with our UI designer to create paper prototypes to test our ideas.
Building on the feedback, we created two improved iterations of paper prototype and carried out A/B test.
Guided by the user test analysis, we were able to identify the right path to pursue.
Storyboards and flowcharts
The development team was facing a steep learning curve tackling Unreal Engine and the Magic Leap.
To help the development team better visualize the design. I created animated storyboards and flow charts for the overall architecture and each specific functions.
High-fi prototypes and user test
With the help of my teammates, I planned and carried out another around of user test to analyze our high-fidelity prototype.
I presented the user test findings and analysis to the team, and together, we used the KFC (keep, fix, and change) techniques to determine our next steps.
Challenge 3. UX design in COVID-19 Outbreak
We were all thrilled to see the accelerating progress of our project. However, our prototyping and user testing were interrupted by COVID-19 outbreak in Canada. We had to suspend our plan for future user tests and change to remote working.
Overcoming the challenge
To overcome the challenge, I have to shift our approach from user test based evaluation to research based evaluation for the rest of the development.
Design Outcome
Load Scene
Upon launching the application, the user should be able to swipe left and right on the touchpad to browse through tile cards representing all saved scenes. By default, the scene in the centre should be the most recently modified scene and should show its file name, the most recent date that it was modified, and a thumbnail representing the scene.
Library
Simple User Interface
Our library menu combines 2D and 3D elements into an intuitive interface. We do not wish for creative professionals to have their creative agenda for a workshop/rehearsal hindered by navigating through an unconventional UI and getting stuck trying to figure out how to access all desired functionality.
Continuous UI canvas
We chose to combine menu elements into a single, continuous UI so that the different UI elements do not float disparately in 3D space.
Customizable Menu location
We then made the decision to hybridize our solution and make the library appear in front of the user at a 2m distance by default (hands-free), and making it easy to reposition both in terms of angle and distance from the user with a simple press of the bumper and drag of a slider. Yet to be tested: slider on menu vs. bumper + push/pull on touchpad.
Organized categories and functions
Transformation
Controls: One Function at a Time
The transformation menu is only activated when an object in the scene is selected. The menu always appears right next to the user’s dominant hand — the hand that holds the controller. Users can activate the various transformation functions by pushing the icon on the menu with the controller.
Users can easily:
Move an object with “drag and drop”;
Push an object further or pull it closer by tapping the front and back of the touchpad;
Rotate an object around any axis by swiping the touchpad in a circular motion;
And scale an object by dragging the corner of its bounding box up and down.
Demo
The Impact
We managed to deliver an evaluated high-fidelity prototype with documentation on time and fulfilled all the requirements from our client. Although our deliverable is only a vertical slice of the project, the UX research and analysis/guidelines provide a solid foundation for future development.
Throughout the process, with the help of entire team, I managed to overcome three major challenges, i.e. conducting user research for a niche market, mastering Magic Leap user experience, and carrying out UX design in COVID-19 outbreak. I gained invaluable lessons on Mixed Reality UX research and design which could be applied to many similar future projects.
Supervisor
Dr. Rachel Ralph
Team
Jaclynn Wong (project manager)
Shyen-Yi Low (UI designer)
Yuankun Huang (developer)
Julie Moens (developer)
Allyson Zhong (UX designer)
