What are emotions? How do we design for an affective diary?

(This was really an email in the project we have together with Microsoft Research on designing for an affective diary).

A set of given, labelled, ready-made expressions for a range of fixed “emotions” is not going to be good solution. It will limit users horribly and might even reinforce some simplistic model of emotions that we would not like to see being spread in our culture. (I am thinking of how certain emotion expressions, like smilies and abbreviations used in chats and similar, spread outside the on-line contexts into our everyday lives. Not that I believe that our systems will be as influential as those are... ).

Our perspective on emotions is that they are:

1. Processes - not states.

2. A mixture of many different emotion processes at the same time. I can feel happy but also tired and in general depressed. Very seldom will my whole being be emotionally hijacked by only one singular emotion feeling.  (Although it does happen in extreme situations).

3. That our experience of emotions goes through many changes as time passes. When I am in the middle of something I understand it and experience it in one way, right after in another, and two weeks later in a third way. Thus, if I want to ”re-live” (or re-experience) an emotion experience, there will not be a true state that needs to be reinstated. It will be a different experience from what happened the first time around. Because there is no “ground truth” about these things. They are subjective.

4. Most of our emotion expressions as well as probably our emotion experiences are quite bland. Very rarely do we get into strong emotional experiences. There is this cool study by Campbell in Japan who made people carry tape recorders for a long time. They write that:

”First, the definition of ‘normal’ appears to be highly context-dependent, as the speaking style varies according to both familiarity with the interlocutor and type of conversation. By far the majority of speech falls under this category and there are remarkably few angry or sad tokens in the corpus (which now contains more than two-years of speech). Normal seems to be ‘moderately happy’, but rather than expressing pure emotion (which is perhaps just an extra-linguistic aspect of the speech, irrelevant to the discourse), ‘speaker involvement’ and ‘discourse intention’ appear to be the main dimensions of paralinguistic variation. [..]

Second, the ‘emotion’ labels have proved to be over-simplistic. It is not at all easy to classify a given utterance into one of the above basic classes without first making clear whether we are referring to the speaker’s subjective emotional states (both short-term and long-term) or to the emotional colouring of the utterance itself (and whether intended or not).”

When we made people try to express what they were feeling while interacting with a computer game, they kept pointing to the part of a Russell circle that indicate mild confusion or not emotion whatsoever. Once they managed to complete some “task” in the game, they would become happy or frustrated or something else, and then right back to the mild, bland, slightly confused/frustrated state. Afterwards they could also give another story of their experiences. But in life, as we muddle along, we are mostly expressing (and experiencing) a mild state of happiness.

5. But our experience of an emotion as it is happening is not the same as what we want to express to others or what we want to tell ourselves about that emotion experience. Thus, in the affective diary, I might want to exaggerate or completely change the story of what really happened. An affective diary might change the way I remember my life. Here I really like the work by Sengers et al: http://www.sics.se/~kia/evaluating_affective_interfaces/Sengers.pdf (from our CHI-workshop which was SO COOL!)

When we worked with eMoto (our extended SMS-service) we tried to address this illusive nature of emotions (vs emotion experiences vs the “story told” afterwards) through creating ambiguous expressions where different emotion expressions were blending into one another. Our idea was that if we avoided “emotion labels” (such as smilies), and instead went for expressions that can be interpreted in many different ways, we would allow for users to express themselves as they want to through the given expressions. At the same time, we wanted to provide a set of given expressions so that they could start using those, negotiating their meaning over time with the receiver of the messages, starting to understand their meaning based on how they are used in their communication, etc. Thus the expressions were not entirely random - they were just fuzzy at the edges. Also, as those expressions were ”created” when the sender of messages were performing some affective gestures, the interpretation of such a message would also involve imagining what the other person had been doing (gesturing) in order to achieve the expressions. The animations in the expressions were there to, in a sense, mirror those gestures in another modality.

I am a great believer in creating technology in which users’ appropriation of the technology will be reflect in the technology itself. Either in terms of making the system change over time as users do things with it (as in social navigation) or in terms of allowing for interpretation and use of the system in ways that fit with what users themselves want to do. In a sense I want to leave ”surfaces” open for users to fill them with content and meaning during use.

There are of course problems with this perspective. Designers will always build their own intentions into the systems they produce. Users will sometimes (perhaps not as often as we would like to think) appropriate technology in ways that the designer had not anticipated. So how do we deal with this? I do not think that we should give up and say ”we can never know anyway how stuff is used, so let’s just provide technology and then people will appropriate it to their needs anyway”.

My current position is that we should aim for good, open-ended, tool-based kinds of designs where the underlying technology or interpretations done by the designer is not hidden but revealed and thereby possible to appropriate by end users. But then we need to study the practice that arises around these designs and *change* the technology as we see how it is really used. In a sense, I believe that technology will ”reveal” practices and get to the kernel of what people are trying to do (and it can of course also hinder them and make their lives miserable and ”reduce” the wonderful complexities of what we are and do - and this in turn may influence us and our culture in bad bad ways). It may be that we open certain “surfaces” to be read/written on (in the terminology of Suchman) while we should have been opening others. Design is difficult!

We’ve built another system named MobiTip based on some of the same ideas. There we designed the system from a perspective of seamfulness (rather than seamlessness which is all about hiding the underlying technological framework). Users in our study did indeed appropriate the technology and made sense of the seams as they revealed themselves in the interface! But not only in the ways we expected them to, but also in other ways that we did not expect them to. And those were of course crucial and we need to cater for those as well in order for the system to work properly as they want it to do.

There is BAD design around even if users find ways of getting around those limitations and expressing themselves as they want anyway. But I think we have a responsibility to at least try and build better design for new practices that make sense in people’ lives and that does not make simplifications where life is complex. Thus, we with the affective diary I think we should aim for:

• providing the user with expressivity that makes sense to them, that they can use, play with, change, have fun with etc.
• providing the user with such expressivity that does not reduce their experiences to only the ”nice”, cuddly feelings, but also allows for the complexities of real life
• providing users with accounts of events that makes them reflect on their experiences and maybe makes them disagree

Maybe the last is the most controversial. I am moving closer and closer to Phoebe Sengers position here who believes that you can make a computer into a ”being” that is not a tool but something that speaks back to you. This way you’ll not only mirror whatever the user wanted to collect, but perhaps also provoke our understanding of what happened? But I am still very much into the tool-based, user-centred view that says that it is the user who should express herself as she wants and sees fit.