Introduction

(T)racing Eyes and Hearts is a research project where we are working towards creating an installation to explore the physiology of empathy. Our physiology responds to our environment in ways beyond our immediate awareness. The visualizations produced here present the heart rate, respiratory changes, and sweat response primarily using non-intrusive sensors. By tracking and visualizing data, we investigate and reflect upon the physiology of emotions experienced in response to human stories. This project was funded by GVU Small Research Grants.

Principal Investigators: Dr. Nassim JafariNaimiDr. Anne Pollock, and Dr. Lewis Wheaton.

My role: UX Design, Visual Design

Research

Our initial focus was to try and abstractly visualize the way a human empathizes with others’ stories and experiences. We hoped for insights on whether the person would ‘feel’ the same emotions as the person telling the story. We also were curious as to whether people are conscious of their true responses to such human interactions.

For our research, we began exploring existing projects and installations that have tried to use human physiological measures as input for artistic pieces. We came across various art installations that visualized human responses to music or other sounds through light, color, and sound.

We explored different stimuli that we could use for our project. Each stimulus came with different implications:

  1. Music would definitely change the pace of heartbeats and modify the respiratory pattern.
  2. Advertisements would evoke emotional responses, but that would seem manipulative as these ads are designed to do so.
  3. Conversations would evoke genuine emotional responses. But these responses may not produce as extreme or interesting outputs, especially if one of the persons isn’t completely engaged in the conversation. Also, it is difficult to have casual conversations when you know that you are being observed in some way.
  4. Listening to inspirational stories or speeches could evoke strong emotions. Such stories are generally captivating, and since they are true, the element of manipulation of emotions disappears.
  5. Movies and videos could captivate an audience, be it scripted or not.

Correlating the changes in physiological patterns to emotion is one of the main challenges we faced. For example, the heart may race because of happiness or fear. Distinguishing the happiness from fear would require more expertise in the subject.

Ideation

Next, we entered our brainstorming phase. We considered the physiological input that we would measure, the type of stimulus (whether it would be some sort of media or a conversation), the output, the set up of the physical space where this experiment would take place, and how we could interpret the output.

Sensors that could be used

Initial brainstorming session

Quick revision of sketch of environment

Proposed environment with the participant sitting in a partially enclosed area, while others can see an abstraction of their ’emotions’ on a timeline outside

Prototyping

In the next steps of this project, we concentrated on creating the visualizations from the data that we record. We were inspired by a few current art pieces as well as the children’s geometric drawing tool, the Spirograph. I was responsible for designing how the visualization would look.

Ideas for visualization which consider color, amplitude and frequency of the waves, and the dots depict the eye tracking.

Current visual representation, created using MATLAB

Choosing Videos

We wanted to select a good variety of stories to show our audience. These videos were called by our code randomly and hence through the responses, we got a good variation of results that we could explore. We were always transparent with our audience about the video selection and that they shouldn’t compare their result with anyone else’s as the media they were engaging with could be different. In our playlist of videos, we had advertisements, real life inspirational stories, puppy videos, high intensity animals-in-the-wild videos, etc. This initial playlist and the responses it got will now help us shape the media (or any other prompt) that we use for further iterations.

Colors

A lot of research has been done on colors and the emotions that they relate with. During our research phase, we read through a lot of these papers but as it wasn’t our expertise, we currently use a default set of colors, and explicitly mention that there is no correlation as of yet. In future iterations, we will be looking into this.

Iteration

Until this point, we were focusing on how we could visualize interactions of any kind between more than one person, whether that is conversations between two people or even multiple people engaging with the same video or music. But we realized that the visualization that we were producing was more appropriate while discussing a person’s reactions to the stories compared to their own baseline and exploring the points in time when they were relatively more activated by the video or more engaged with its content.

Presentations and Response

We have presented our work at multiple Georgia Tech events. We were also selected to present at the 4S (Society for Social Studies of Science) conference in Boston in August, 2017.

Future Directions

Creating a Final Artifact

At multiple conventions and meets, a constant response that we got is that the people would like to take away some physical artifact from this experiment which they could share with their loved ones and start a conversation. Therefore, we decided to print out these visualizations along with a brief legend on a post card size paper and share it with our audience. We plan to convert this visualization into some 3D form, like projections on the floor and curtains around the participant (as discussed in our brainstorming phase) or collaborating with artists who work with lights, lamps, and other structures.

Application

Through our presentations at conventions and other meetups, we encouraged our audience to engage with the visualization as much as they could. Many of them suggested applications of such an artifact beyond being an artistic expression for starting conversations. Such a project could be used by actors and other artists to explore whether the emotion they are trying to convey is being received accurately. It can also be used in environments like cars in real time where if the results show a decrease in engagement, it could imply that the driver’s attention is waning, and this could trigger another element (like loud music) to make the driver attentive again. These ideas and responses to our project will be explored further in the next steps.

Final Poster

Our poster used for these presentations, further describes our project and how to read the visualizations.