The Mimoid and Blob Projects Demonstration for MultiMedia '98
Bino - bino@sics.se
Lennart Fahlén - lef@sics.se
Pär Hansson - par@sics.se
Mårten Stenius - mst@sics.se
Jonas Söderberg - jas@sics.se
Anders Wallberg - andersw@sics.se
Karl-Petter Åkesson - kalle@sics.se
MIMOID is an interactive mini opera based on ideas from Stanislaw Lem's
novel Solaris [Lem] and created for a multi-user mixed reality environment
and focusing on new forms of artistic experiences and social interaction.
In the opera real singers and sensor controlled avatars are acting against
each other in a performance being simultaneously presented in two different
CAVE's. Furthermore, MIMOID is planned to be distributed to the audience
in two or more ways. The Blob was developed as an initial part of the Mimoid
project and is the first steps of designing the scene for the play. First
Mimoid will be described, then the Blob and finally what we would like
to demonstrate during MultiMedia '98.
The Mimoid project is a combination of a research-project and a small-scale
opera-production.
-
The research-part includes the developing of both tracking- and behaviour-controlled
avatars, a new kind of visualisation of the World Wide Web and interactive
control paradigms for a spatialized soundscape.
-
The artistic part contains both the graphic designs of the virtual stage
and actors, and the music.
-
The design is based on the contrast between a sterile, brushed metal like
interior and the organic, colourful outside. These two themes are also
reflected in the cast’s costumes.
-
The music (and sound) is a mix of modern chamber-opera and EAM (Electro-Acoustic
Music).
-
The actors is one "live" singer (baritone) and one virtual avatar (pre-recorded
mezzo-soprano). In the second version both of the singers will perform
live, but in different locations. The orchestra consists of a string quartet
and a piano-player.
The project was initiated in January 1998 and is intended to be performed
inside the prolongation of the EC i3-project eSCAPE.
The Mimoid story
In Stanislaw Lems novel Solaris from 1961 the psychologist Kris Kelvin
arrives at a space station orbiting a remote planet, Solaris, completely
covered by ocean. His mission is to investigate some psychological and
social "difficulties" experienced by the three researchers currently working
at the station. The Solaris ocean has been an enigma to the scientific
world for more than a century. There has even evolved a special branch
of multi disciplinary science called " Solaristics". The semi organic fluid
covering the planet shows no regularities in response to different kinds
of experimental stimulus and tests. Early reports of fantastic visions
and supernatural experiences indicates that the ocean or the cloud filled
atmosphere either is mildly hallucinogenic or it really can perform visual
metamorphosis.
When Kelvin wakes up after his first night at the space station there
is a woman laying next to him in the bed. He recognises her, it is Harvey,
the woman he lived with back on earth. But the real Harvey is dead. She
killed herself ten years ago as a result of a crisis in their relationship.....
The woman awakes. She is disoriented and asks Kelvin where they are.
She apparently thinks she is the original. In the scene that follows she
gradually becomes aware of the real situation, she starts to "remember"
her own suicide and what led up to it. In desperation Kelvin helps her
to once again commit suicide. The next morning she again lays next to Kelvin
in the bed. She doesn't remember anything from the day before. She is "new".
Kelvin ( and the audience) realises that the scene will happen again and
again for ever....
Harvey and Kelvin in the bed and the view outside the ship flying
over the surface of Solaris.
Is Harey a ghost? Is she a dream created by Kelvins longing and bad
conscience? Or is she a manifestation of the Solaris ocean? Are the researchers
the ones actually been researched and experimented with? The Solaris ocean
could be the conscience of the man, an alien life form, or the real sea.
The woman could be a male fantasy, a real women, or a tool for alien researchers.
Who is researching whom? Is the man more real than the woman?
The story is inspired by The Odyssey, Pygmalion, Kalevala and Stanislaw
Lem and the above scene is just one chapter from the Solaris novel but
is an excellent basis for a 30 minute mini opera.
Accomplishment
In the initial version Mimoid is intended as a dialog between a real actor
and an avatar. The man (a baritone) performs his part live, the woman’s
voice (a mezzo-soprano) is pre-recorded. The performance space (the stage)
is a CAVE and the scenography is computer generated, as is the woman. The
performance behaviour of the avatar towards the man is governed by rules
extracted from the dramaturgical structure of the libretto. This behaviour
can range from a very close tracking of the mans every move to more complex
stances. The mans movements are therefore being tracked by various means.
As a second step a distributed version running in two (or more) caves
will be developed. In the first performance space the man acts against
a virtual actress, just as in the single site version. At the other space
it is the opposite situation. The woman is real while an avatar directly
controlled by the male actor in the original performance space represents
the man. The female avatar is of course also controlled by its physical
actor so there is two complete tracking systems capable of reading the
physical actors positions and movements.
Of course, in this second setting both of the voice parts will be live
but one of the voices (the mans or the woman’s depending on position of
the listener) will be heard directly and the other one is distributed over
the connecting network and played over loudspeakers.
The music in both versions consists of a motion-controlled multi-channel
"soundscape" composed in "classical" electronic music style (i.e. electroacoustic
music) combined with a parallel score being performed live. In the initial
version a pianist plays the live instrumental parts. In the second, the
two-CAVE-version, there is a pianist at one site and a string quartet at
the other
Some themes:
-
The original and the copy:
When the story begins the avatar-woman thinks she is a real human being,
and when the story continues she evolves a personality of her own, not
a replica of her prototype. But does this mean that she automatically got
the right to be treated as a human?
If a copy (a clone) doesn't differ in any way from the original, do
these concepts have any relevance? Will the humanism of the future incorporate
androids, robots, AI's cyborgs, clones etc... Or is there a need for a
whole new set of ethic rules, an extended humanism.
-
Inside and outside:
The room is an extremely confined space, but the panoramic observation
windows make the presence of the ocean constantly felt. The scene setting
is therefore being experienced both as claustrophobic and vertigo inducing
(probably like being in black space itself) The interior is barren, functional
and probably rather boring. The ocean on the other hand, being half-organic
and constantly shifting in form and colour is both frightening and inviting.
-
The Ocean:
To graphically depict the ocean is one of the major design challenges
of the project. The ocean is fluid like and in constant metamorphosis.
Its shifting are completely unpredictable but at the same time giving a
hint of some hidden systematic activity being at work. In the modelled
virtual sea there has to be sets of behaviours that maps to the scenic
action without ever becoming predictable.
-
Subject and object:
During all these years of mankind studying the outer and inner spaces,
maybe the real situation has been that it is the humans that’s been the
subject of research and experimentation. Maybe humans are too blinded by
megalomania and ideas about their own superiority to see and understand
the encounter with an equal or superior intelligence or being.
In the two-CAVE-version of the opera the switching back and forth between
different views could be made even more explicit by the second site having
a female actor meeting her fantasy of a man. The audiences at the different
performance spaces will then see two different "realities" that somehow
seem to be connected.
The Blob
The design of the Blob was the first steps of creating the Solaris planet
with its organic and chaotic ocean. A first version of the Blob was shown
at the CeBit fair. It was developed jointly by technicians, artists, and
musicians. We tried to get the technology to meet the demands of art. The
Blob uses a virtual desktop to allow a person to manipulate it for generating
sound and graphics. The user stands in front of a table, a touch screen,
uses stereo shutter glasses and puts a finger on the screen which shows
the Blob object surrounded by a ‘star filled sky’ environment. We also
include a videotape from the CeBit fair with this submission.
Two screenshots from the Blob.
As the user touches the screen the Blob moves and changes shape and
texture and emits sound depending on how it is touched. The inspiration
for the Blob textures is a beautiful organic underwater fantasy world,
fragments of cloned memories, and jellyfish like movements, which are interactive
and change shape and colours depending on different influences and reactions.
The textures are sorted into three groups depending on the richness of
the colours in the image. The more the visitors drag or deform the Blob
the stronger and brighter the colours get, and less interaction will cause
the images to be more transparent and plastic wrapped. Sometimes there
is an image of an object hidden in the surface texture.
Users experiencing the Blob during the CeBit fair.
The Structure of the Blob – Triangles, Crystals, and Colour
The Blob looks like an underwater space flower with its perfect formed
crystal like tri-mesh architecture. It is constructed of tri-meshes, triangle
formed polygons, the triangle is the strongest construction form, because
it does not deform under stress. The downward pointing triangle stand for
the element water and the upward pointing triangle for fire [Tyson].
As Stanislaw Lem wrote in his novel Solaris:
" The meeting of ocean and sky drowned in blood red mist. The symmedriad
blossomed like a gigantic crystal flower"
In 1666 Isaac Newton discovered, by the use of a prism, that all the
colours exist in solar light and can be separated from each other through
refraction. This link between the geometric shapes underpinning the Blob
and the breaking down into a series of colours provides a motivation for
considering the use of basic primary colours within the Blob.
Vibrant colours (Goethe, the theory of colour):
| Yellow |
- Purple |
| Orange |
- Blue |
| Green |
- Red |
If the complimentary (vibrant) coloured textures are mapped next to each
other on the Blob a vibrant effect is achieved. These complimentary colours
can be experienced by for example looking at a strong red coloured object
for a while. If you then remove it you will see a green shadow.
Interaction with the Blob - AKUPRESSURE
Stanislaw Lem describes, in his book Solaris, the Mimoid (the ocean) as
a child:
"It was a glossy shiny child with big blue eyes, it rose and fell with
the waves but it was making other movements with no meaning and they were
horrible"
The Chinese discovered that pain on the body could be healed by akupressure
through a theory about representation of the body of a child, [Harvey].
The idea being that if you for example wanted to releave a pain in your
back, you could apply pressure on for example your ear at points where
the back of an imaginary child mapped on the ear would be. One idea is
to find the Blob’s akupressure points and map interactions to behaviours
based on where on the mapped imaginary child the interaction occurred.
Further interaction with the Blob - BUDDHIST FINGERMAGIC
In the present application people interact with the Blob through the index
finger with the water like behaviour and textures, but for a future version
it could be possible to use a sensor glove for switching environments,
sound, textures, etc. connected to different elements for a possibility
to change the virtual world’s element depending what finger you use, Buddhist
finger magic for touch screen:
| thumb |
spirit |
white |
| index finger |
water |
blue |
| middle finger |
earth |
black |
| ring finger |
fire |
red |
| small finger |
air |
yellow |
Using a sensor glove these old buddhist concepts can be used to change
the Blob’s behaviour, parameters, textures, soundscapes, etc. In effect,
a simple way to move between blob worlds or change the Blob landscape.
Future development of the blob.
The Blob used at CeBit was only controlled by a single user, i.e. from
one site. We will use the distribution in our DIVE(Distributed Interactive
Virtual Environment) system to make it possible to manipulate it from several
sites simultaneously. A second version of the Blob is under construction,
for a multi-user interface, distributed between two different virtual desktops.
The screens are touch sensitive and are meant to work in multi-user mode
so many users can share the exciting experience and interact with the Blob
at the same time (for example, there can be 10 users around each table
at each site). Through sound and design, it will be possible for each user
to see and hear personally how he controls the application and at the same
time experience the collective influence upon the object. We also plan
to see how far we can take it in terms of behaviours. Morphing between
different geometries will be added and synchronised with behaviour changes.
Organic behaviours in virtual environments
The modelling behind the Blob can be generalised and used to breathe
life into virtual environments. Apart from dynamics generated by tracking
equipment and simple scripts more organic motions can be added by applying
this technique on for example clothes, skin, and plants with swaying branches.
Another interesting thing to explore would be to, with a ‘surgical knife’
in the VE, cut certain springs in the model resulting in a new way of sculpturing
objects. One could also implement gesture recognition which enables the
user to use a gesture vocabulary to perform more complex operations.
The MultiMedia '98 Demostration
Our intentions with the demonstration at MultiMedia '98 is to show the
current status of the Mimoid project as it will be in September. The project
is on going and therefor we can not tell how far we will have got
in September. What we at least will have is the distributed version of
the Blob as it is almost finished today. Another quastion mark today is
what kind of computer equipment we will be able to bring with us, but at
least we will have ordinary desktop computers with which we could run a
distributed interactive performance over the Internet.
References
[Lem] Lem, S.: ‘Solaris’, Harcourt Brace Jovanovich, 1987.
[Tyson] Tyson, D.: ‘New millennium magic’, Llewellyn Publications,
1996
[Harvey] Harvey, C. G., Cochrane, A.: ‘Vibrational healing’, Thorsons,
1998.
Last Modified, April 29, 1998
Karl-Petter Åkesson, kalle@sics.se
