Capstone project VR & Affective Computing
A VR space that visualizes the flow of emotions.
When it comes to artists, two things stand out: the fact that their work is directly related to their emotional states, and the fact that encountering mental blocks is a common theme. Therefore, we designed Stream, a virtual reality system for creatives to visualize their mental states in order to overcome mental blocks and ultimately spark inspiration in their creative process.
How might we use biometric data to support artists to overcome creative blocks by introducing change in their environment?
Artists and industry experts who we conducted interviews with (not all people are shown):
Below are 6 insights generated from our study, with details presented in the comprehensive research report:
1. There is no universal definition for emotion, but artists mostly interpret it through forms.
“Emotions is like quantum physics and is beyond my visual imagination.” - A1
2. Emotional experience is fluid not discrete.
“People feel certain ways, and they assign words to it. Since you assigned separate words, you assume internally that this thing is separate in its generation. That assumption is only based on some language that you invented.” - E4
3. Manipulating sensory stimuli in a person’s environment directly affects their related emotional experience.
There is a constant loop of perception, cognition and action when one is in an environment. An emotional response comes from this loop. By changing the environment that one is currently in, there is a direct effect on this emotional response, and by extension the artistic output of the artist.
4. Emotional memory can be trained much like muscle memory through repeated practice.
“One of the key insights from mindfulness research is that, you can exercise your internal attention like a muscle, you can exercise introspection and that you have a transfer of those skills.” - E4
5. Overcoming creative mental blocks is about introducing change.
“Being able to change my context and kind of recollect and restructure my thoughts allows me to come back with a fresh perspective and homes me back into the big picture.” - A4
6. Artists want to have control over the tools in the creation of their artwork, but are willing to surrender control of their emotions during the creative process.
When using creative tools, artists want to have control over the end product of their art as much as they can in order to precisely convey their thoughts and feelings. However, during their creative process, they want to be able to have their emotions be free within themselves for more innovative and fresh explorations of expression.
Design Principles & Low-fidelity prototypes
To guide our design solution, we created 6 design principles based on previous research insights:
Low-fi Prototype Ideation
After grouping and selecting concepts based on our design principles, we narrowed down to four design concepts to develop low-fi prototypes.
1. Reflective Mirror
User's affective data will be visualized as microorganism on a petri dish. Elements in the petri dish can be used as colors and patterns for creation.
2. Collaborative Drawing
Users will listen to a piece of music and collaborate in drawing. Each user will be assigned to a shape (eg, a circle or a line) and can only draw with that shape. Size of the shape is dynamic based on the user's affective data.
3. Mind Knob
The user is immersed in a painting, either created in 3D, or a classical work converted to 3D. The user is able to control the flow of elements in the painting using their brainwaves.
4. Affective Room
The user is immersed in different scenes in VR and is able to use their brainwaves to change the environment, such as weather conditions
Low-Fi Prototype User Testing
Feedback from Low-Fi Prototype User Testing:
1. Reflective Mirror
Pain point: Drawing tools are forced on the user.
Though having dynamic color based on the user's EEG data is a cool idea, it does not provide users with full and precise control over their creation tools.
"I will get really frustrated if I cannot get the exactly color I want." - p1
2. Collaborative Drawing
Pain points: Limiting free expression; low incentive to participate; VR as platform
"Free expression is more enjoyable. This is limiting my expressions" -p2
"This is kind of meaningless … I don’t know why would I do this. Unless this is a game or competition" - p1
"For people who don’t know how to draw, this is not fun and can even be scary" -p3
3. Mind Knob
Pain point: Interaction is too monotone.
"What I can do is very limited, I think I will be bored soon" -p1
"I want to bring more changes in the art using my body... It will also be nice if I can interact with a music piece." -p3
4. Affective Room
Participants liked this prototype the most, and said that it will be more fun if the affective room is more surrealistic.
"I don’t really need the environment to calm me down, I will just take a pill. Instead I want another universe." -p1
Mid-fi Prototype & Testing
Based on feedback from the low-fi user testing, we developed and tested the midlevel-fidelity prototype - an affective space where the user can draw, explore others' work, and reflect upon their mental states.
Photos from mid-fi user testing:
Feedback from Mid-Fi User Testing:
Added color pallets, eye drop tool and opacity for easier access; Added Eraser and Clean Canvas button; Made all brushes colorless before color is selected
Home Portal for Better Affordance
Previously, we decided not to have home portal, so that once the user enters the application, the user can directly interact with the space. However, testers were confused about their current state and were looking for affordance for navigation. So we added home portal where users can choose different functions, rather than find them on the controller panel.
"There is no affordance in the space, so I feel lost and feel like I need to navigate to something more editable. Maybe this is just the first time user feel. But I still want something more controllable." - P2
Previously, we designed a training function where the user can record a desired mental state (ie, a clip of the affective room) and try to match that state to train their mental muscle strength. In this design, a small window is shown on the corner to replay the recorded mood, and a percentage to tell how much the current mental state has matched to the desired mood. However, during user testing, some problems of this function are revealed.
“I need context to understand the emotional values or visualizations, and what triggered that.” -p3
“It is a little bit cold in showing my emotion as number.” -p4
Therefore, we changed the training mode to reflection mode, where the creation process and the affective states are both replayed to help users understand why they had a specific mental state, and thus give them more clues in trying to match that state. In addition, a reflection ball showing the difference between current and the previous mental states is shown to help users bridge the gap.
Specification & Next Steps
Demonstrate to public @Seattle Public Library
Looking forward, we want to propose a question that our design provokes. What exactly does it mean to have a system that is emotionally reactive to you?
Affective computing opens up a potential landscape of emotionally intelligent interfaces that will be able to provide insight into ourselves. We will be able to learn about ourselves in novel ways that were not possible before. Within our design, the creative activity we have present is 3D painting, similar to something like tilt brush, but that is not the focus of our design. We chose painting to keep the idea focused, however having feedback on any creative activity is possible and valuable.
Furthermore, we only explored one option of how the immersive aspects of VR can be used in the future. Virtual reality is usually framed as “how can we make it close to real life?” - better graphics, better facial tracking, etc. This is a narrow way of looking at what VR has in store. With virtual reality, we have the opportunity to meaningfully interact with the virtual world in ways that we cannot in regular life - see things that we can’t see, go to different places, and so much more.
We are in the beginning of both virtual reality and affective computing and as time passes d their presence in our technological ecosystems will only increase. Stream is just one step in the direction of an imagined future that we have the opportunity to create.