* concepts

* design

* prototype

* user study

* publications

We have designed, implemented and evaluated a wearable prototype for exploring user experience and musical interaction. The prototype senses the user's context and actions when walking through the city, maps this information to the audio processing of live urban sounds in real time, and outputs the resulting music through headphones. It is an open-ended platform for iterative prototyping of sound content and musical interaction that enables testing in real-world settings.

* system overview

The prototype consists of a small laptop computer, a BasicX-24 microcontroller, a USB-MIDI converter as well as biometric and environmental sensors worn by the user. Sensor input is collected by the microcontroller, which sends them in MIDI format to the laptop (carried in a shoulder-bag) via the USB-MIDI converter. The data is then reconverted and processed for context and action recognition of "if-then" type, and mapped to musical parameters in a modular program created in the interactive environment PD: urban sounds captured by the microphone are processed in real time and turned into music based on sensor input. The music is output through headphones as the user is walking.

* hardware

Current sensors used in this implementation are a metal detector, an IR-sensor measuring proximity to walls and objects, a light intensity sensor, a microphone measuring sound level, and an accelerometer sensing stops, starts, and the starting user pace that determines the music tempo of a whole session. We have also experimented with sensing pollution and temperature and plan on adding a heart-rate sensor.

Low-level sensor input such as light intensity or presence of metal are continuously measured and mapped to the music, whereas the context recognition of high-level parameters such as "standing still at night" are updated every other beat.

The sensors can be plugged in and out of the platform, thus individual sensors are easily isolated, combinations quickly tested, and damaged components simply replaced. The way the sensors are worn and positioned on the body can influence on what is being sensed and thus on the kinds of musical patterns generated. Therefore, their positioning is designed to be flexible in order to allow for more user control and expression through customisation.

In the final version of the system, the sensors will be wireless and the context recognition done on hardware level, allowing for more flexibility and robustness, as well as the possibility to conduct more long term evaluation with user in context.

* software

After a layer of context recognition, the sensor input is mapped to music parameters in a program written in the envrionment PD. Structurally, the PD program is composed of small modular units that construct the music algorithmically according to what factors trigger them, and to the values of the data mapped to them. Concrete urban sounds input through the microphone take manifold parallel and serial paths through sound processing objects. These paths can be flexibly deviated and redirected in response to multiple incoming sensor values, resulting in a highly dynamic sonic output. This modularity and the flexibility of the mapping models enable us to easily test various types of musical output.

* sound & mapping details

The accelerometer senses stops, starts, and the starting user pace that determines the music tempo of a whole session. On a high level, enclosed environments have twice as fast tempo as open ones, loud contexts more intricate rhythms than silent ones, and bright contexts are more atmospheric than dark ones. Standing still mutes most of the music except for a metronome sound. Turning changes the patterns of the rhythm layers randomly. On a low level, sound level is mapped to the amount of rhythm layers, and light intensity to pitch. The metal detector plays back the latest random samples of urban sounds recorded by the system. Samples can also be scratched with the proximity sensor.

[audio clip]