* concepts

* design

* prototype

* user study

* publications

* input

Rather than relying on a fixed infrastructure, such as sensors deployed in the environment, we opted for an entirely wearable solution in order to support user mobility. To determine interesting input parameters, we re-examined characteristics of walking in the city and carried out some limited ethnographic studies of specific sites and pedestrians paths [observations]. From these observations, characteristics of pedestrians and surroundings were categorised in terms of action and context. High-level descriptions, such as 'indoors' and 'crossing the street', were broken into measurable cues that the system could use for context and action recognition. From this, possible input parameters from sensors emerged:

 - body-related input: heart rate, arm motion, speed, pace, compass heading, ascension/descent, proximity to others/objects, stopping and starting

 - environment-related input: light level, noise level, pollution level, temperature, electromagnetic activity, enclosure, slope, presence of metal

Some types of input involved a range of continuous values fluctuating over time, e.g. the outside temperature or a pedestrian's heart rate. Other types, for instance a car horn, only occured momentarily, in a way that could be described as discrete.

* sound design

The sound content had to be consistent with how people already perceive and experience the environment of the city. We considered peripheral versus foreground aspects of the experience and musical possibilities ranging from ambient to rhythmical. As we wanted to maintain a close experiential relationship between the sound content and the context of music creation ­ (namely the existing city soundscape), we decided to use real time audio processing of urban sounds as a basis for the sound design. The musical output is shaped in real time by sound processing objects such as filters, delay loops, envelopes, sampling, playback, mute, and echoes.

As a follow-up to an iterative sound design process, we used video simulations as a method for developing and validating our design decisions. These simulations consisted of videos filmed by us in the city, to which we added soundtracks created with the prototype. The video's audio was input into the prototype and we used sensors input to simulate interactions suggested in the video.

In those simulations, the following mappings were developed iteratively. Incoming urban sounds were processed through filters that opened and closed rhythmically at a tempo corresponding to the pace of the player while walking. The frequencies of these filters was determined by the intensity of light detected. The noise level of the local environment was mapped to the amount of rhythm in the music, such that rising volume increased the number of rhythmic layers that are overlaid on each other. Proximity to metallic objects or surfaces triggers brief echoes, the delay of which depend on the pollution level. At night, samples recorded randomly were echoed and filtered in relation to sudden flashes of light detected from street lamps.

* mapping strategy

The mapping had to be both transparent to the user and complex enough to sustain interest if the system were to be used day after day. We also considered it essential that the mapping would reflect scales of time and distances covered while walking in the city and maintain the distinction between continuity and discreteness of input. Sources of input selected during our urban observations were therefore classified into two levels: low-level discrete and continuous factors coming directly from the sensors, and high-level factors of general context and user actions resulting from abstractions of the low-level ones.

In the mapping we developed, high-level abstractions of context and actions are mapped to structural composition parameters. Low-level discrete and continuous factors are mapped respectively the triggering of short musical events, and to spectral variables. Thus, on a low level, rhythm patterns are algorithmically generated and sonically shaped based on sudden user actions and localised urban events and ambiances, while on a high level, the overall structure of the music is based on patterns of actions, path over time, and overall urban context.

Within this general framework, decisions about details of the mapping were carefully made one at a time to insure coherence and pertinence.
 

* interaction space

When considering questions of user perception and control over the music, we asked ourselves the following questions: How 'in charge' of the experience a user should feel? How explicit should means of controlling the system be? Should some degree of randomness be built in the system to maintain interest? How should it sound in situations of unvarying sensor input values over long periods of time? For everyday use, how similar could the same walk sound day after day without becoming boring? What should the balance be between the influence of user and environmental factors? How would 'invisible' factors (whether sensor-based such as pollution or processing-based such as randomness) be perceived?

Rather than starting form a target audience or from ourselves, we have used scenarios to imagine alternatives for the sound design, control factors, and aesthetics [scenarios]. They are deliberately extreme in order to represent a wide range of possibilities and design implications, and were based on interviews with people that perceive the city or music in diverse and extreme ways, f.ex. practitioners of parkour, a french artistic sport making use of urban infrastructures. For instance, we held a workshop in Paris with parkour practitioners to serve as inspiration and potential driving force for the development of the sonic city prototype. [parkour workshop]

The scenarios highlight differing personal relationships with the city. We considered balances between active/passive, peripheral/foreground and ambient/rhythmic aspects of the experience, and were able to define a design space for the amount and nature of user control supported by the system. This control space is a territory of possibilities and locates the scenarios in relation to one another. Two axes describe the predominant factors that influence the music: the vertical axis shows the balance of body or user input versus environmental or city input; the horizontal axis describes the span of possibilities from unpredictability to user-deterministic control.
The scenario that we chose to implement was 'Joanna'. Balancing both active engagement and urban discovery, Joanna would use Sonic City to rediscover her environment as a poetic and aesthetic practice. Representing the essence of our intentions with Sonic City, this scenario provides a foundation for testing other variables and possible experiences.

* wearability

A garment was crafted to easily 'try on' the experience of early versions of the prototype and wearabilty options with users in the city. Rather than a design solution, the garment was a sort of working paper prototype – a participatory format for provoking discussion, improvisation and iteration. [wearable design]

The garment helped testing the robustness of the prototype implementation. After performing a cognitive walkthrough, in which we tested different interaction options and usability issues ourselves in the lab, we staged the use of Sonic City during controlled experiments in urban settings, using the modular jacket. This enabled calibrating the prototype and determining optimal sensor placement for the user study.

In the future, the system could be integrated in clothing accessories such as earrings, badges, belts or bracelets.