In its current form the ACCORD toolbox is made up of a
number of software and hardware components that are intended to be integrated
into the home for the purpose of managing the home environment. These
components are closely associated with a set of compositional editors that
allow these components to be combined to form applications. The general model
is that developers and users would construct applications by assembling a
number of these components to work in tandem.
The basis of the software is the notion of a shadow digital
world that acts as a “virtual” representation of the physical environment. This
virtual environment is closely tied to the real world and changes in the
virtual world have an effect on the real world and vice-versa. Essentially,
electronic household items and services are manifested as software components
that transform information between the physical space and this digital
environment.
The broad Accord vision is to allow home inhabitants to
create customised services by using configuration mechanism that are adapted to
the know-how of family members. The toolbox encompasses the software
infrastructure for connecting the different services and items available around
the home and editor facilities for building new home services through
combinations of these.
We describe several applications or services that have been created
which transform input from the physical world into digital data objects. These objects provide access to and from
their physical counterparts. Connections between these objects may be created
via editors that are based on either electronic or physical interactions. We
present the design and implementation of examples of both types of editing
facilities as well as initial observations on their use.
In order to understand the role of the components and
editors it is worth briefly reviewing the various concepts that are part of the
basic toolbox and which have been previously described to some extent in
deliverable 2.1. Specifically we briefly consider the following:
-
The component bean model - this defines the standard
practice in designing components for the envisioned home system.
-
The shared Equip data space - this is the chosen infrastructure for interconnecting
components. However the toolbox is
versatile to make use of other models.
-
User profile information and definition of access
rights for objects in the data space
Rather than seek to develop another component model the
Accord model builds upon an existing standard. In particular, the
implementation of transformers and services adheres to the Java Bean model [1],
and they are referred to as beans in this document. The bean model proposes a set of coding standards for identifying
constituents of objects and a compatible interface for accessing and modifying
their internal data.
In principle all physical devices will have a bean
counterpart that will populate the data space.
The data space is also comprised of other bean types, such as
transformers, which aid in the interconnectivity of different objects [2]. As long as future developers adhere to the
bean model, which is good coding practice for object-oriented environments [3],
the toolbox will accept and seamlessly make use of any new service. Also, our
use of the bean model allows us to gain leverage from the existing body of code
that already adheres to the beans component model.
The data space model is currently implemented with the EQUIP
system [5]. However, the management
mechanism handling the bean interconnection activity is designed to be
independent of any particular system.
Thus the toolbox implements its own event model so as to be able to make
use of other data space solutions or forms of networking. The use of this
architecture will allow us to exploit a range of existing data spaces including
the development of JavaSpaces [6] and Tspaces [7]. However, the current
performance characteristics of Equip are such that they offer considerable
advantage to the Accord toolbox.
Visual and physical editors [see Editors, p 4] use the toolkit event model to propagate and
subscribe to activity in the data space.
This is useful for coordinating the state of the different connections
and activities between editors and also promotes extensible modular
architectures.
In addition to the transformers and services adhering to the
bean model, specific objects representing users and access rights to bean
components have been introduced. These
access objects are not part of the bean model and are intended to act as
generalized access rights components for any kind of service. The toolbox makes use of these objects to
distribute the user access rights across the space for the case when there are
multiple editors and users.
In this section we will proceed to describe the types of
editors implemented as part of the toolbox.
The editors are the main point of contact for the toolbox and are the
principle manner in which applications and services are assembled. Editors are
categorized into two basics types:
·
Visual editors – these provide a graphical
representation of the interconnectivity between components and utilise
graphical manipulations for the management of links
·
Physical editors – these provide tangible
mechanisms for editing the interconnectivity in the data space but do not
necessarily visualize it in its entirety.
We decided to begin by implementing a developer oriented
visual editor for creating different household services, before attempting to
develop physical mechanisms for interconnecting components. It was felt at the
time of construction that this form of simple developer oriented editor would
provide itself useful for determining and testing the practicality,
effectiveness, and robustness of the envisioned and implemented services. Its use helped us to improve or discard
services based on demonstrations and focus-group sessions, where it became more
apparent which of these were of enough practical interest to include in future
concrete studies (or future releases of the toolbox).
A graphical library was implemented to support the creation of different
graphical elements and to amend the overall look and feel of the
interface. In this design we allowed
the inner workings of the system to be made available at different levels of
abstraction. This gave us maximum flexibility in determining the appropriate
level of abstraction to expose to the user.
The net result was a versatile system to support a user-centred approach
to interface development.
The graph editor was the first configuration facility to be
implemented as it based on one of the most straightforward approaches for
connecting transformers and services together [8]. Each bean and its properties are represented as nodes in the
editor. Each node represents a bean and
its properties are textually visible as an item list. Properties from different beans can be connected by dragging
directed edges from property to property (see fig. 1). This kind of representation
is closely related to the underplaying implementation of the toolkit. As such,
it is very useful for developers of the infrastructure and of transformers for
testing purposes. However, such an environment is less appropriate for home
inhabitants, our targeted users, although we could envisage an editor of this
form being used by future service providers.
Our second graphic editor provides a more user-oriented
approach to assembly. This editor is based on a recreational puzzle metaphor.
User connect component through a series of left-to-right couplings of puzzle
pieces. It provides a recognizable
mechanism for connecting services together.
Constraining puzzle connections to be left to right also provides the
illusion of a pipeline of information flow.
Each service or transformer [conforming to the beans model]
is represented by a puzzle piece template.
The editor detects the beans in the data space; a puzzle piece
representation is created and added to a list of available templates for the
end user to use. Each bean has the
option to provide an icon for its representation in any of the visual
editors. Puzzle pieces are rendered
with these, along with a descriptive text.
The control panel or puzzle editor is composed of a number
of other panels, a list of templates (of available components) and an editing
canvas. The templates can be dragged and dropped into the editing canvas or
workspace. When a puzzle piece template is dragged onto the editing canvas it
clones itself and becomes a symbolic link to the service or transformer. The
editing canvas serves as the work area for connecting pieces together and
visualizing their activities.
Unconnected isolated puzzle pieces in the workspace are trash collected
[removed from the workspace] automatically when not handled for a certain
amount of time. This is to maintain a
tidy workspace.
Figure 2 Puzzle editor
Connecting puzzle pieces together works by dragging a
particular piece in the vicinity of a fitting target piece. When a puzzle piece is first dragged
non-compatible pieces in the workspace are disabled and shadowed, indicating
which pieces are plausible target connections.
As previously described, the connections imply that the internal
variables of the source bean are connected to the compatible internal variables
of the target (or sink) bean. However,
since a bean has the option of containing several variables multiple
connections are possible. For
one-to-one matching the connection is done automatically. For multiple choices a dialog window is
presented to the user asking for the preferred matching for each variable. The design issue has been whether to design
beans from the beginning with constrictions of single input and output
variables – as to avoid the extra dialog box step - or to allow any number of
such. Preliminary PD sessions
demonstrated that although the puzzle mechanism is easy enough to grasp, the
extra layer of accessing internal bean parameters is on the boundary of easily
understandable interaction.
Specifically, poor naming and descriptive conventions for the internal
variables (properties) made it elusive for most test users to comprehend their
function.
The editor not only provides audio feedback on interaction,
but also traffic feedback to some extent. When properties related to puzzle
pieces in the data space are updated, the corresponding puzzle piece changes
colour and a short audio clip is played.
In addition to the two graphical editors we have developed
two examples of physical editors that provide users with tangible techniques
for creating new home services. These are the linker device and the paper
puzzle editor. They are described in this section.
The linker device uses physical manipulation within the home
environment as a means of exploring the properties a device makes available to
the data space and as a means of linking properties on one device with
properties on another [2]. The basic device consists of an iPAQ that talks
directly to the shared data space and a barcode reader that can read barcodes
placed on interactive devices (figure 3). When the user scans a barcode the
linker editor displays the properties that this device has made available to
the data store. The user can then scan a second physical device bringing up the
properties whose type match the selected property. The user can then select the
destination property making the link
This design has the advantage that it relies on in context
physical manipulation within the home space and as such it is provides users
with a much more specific understanding of the components involved in a new
home service. However, the current design suffers to the same extent from the
same problems of presenting users with property data as the puzzle editor.
Figure 3 The linker device
The idea of the Paper Puzzle Editor is to use real 8x8 cm
puzzle pieces as a physical version of the previously explained graphical
Puzzle Editor. The puzzle pieces have
the same graphics (as their graphical counterparts) printed on top and can, in
the current version, be combined three at a time on a shelf or other specially
designed place in the home. This shelf or place contains the reader unit for
the puzzle pieces. The technology uses a very short reading distance; for this
application it is almost in contact. However, it does not need electrical
contact.
Each puzzle piece has
a unique identity in the form of a printed code underneath. The code doesn’t
have to be visual for the eye. The code can be printed either on the printer
used for the PC at home or in a print shop, and then even with invisible ink.
The printing cost is under 0.1€ per piece.
In the near future a
puzzle piece with a changeable code is going to be created. Just touching a
certain area on top of it will modify the code. This can for example contradict
the action to be taken when using the puzzle piece.
Figure 4. The
picture shows a prototype to the puzzle editor and the shelf containing the
reader for the puzzle pieces where you can e.g. get a shopping list for
groceries as an SMS to your mobile phone by the use of three puzzle pieces.
This reader technology, has been integrated into other of
Acreo’s research projects, where it will be explored further.
User profiles and user access rights to the different items
are both distributed as objects across the data space. They identify general and individual rights
to a particular item or a set of these.
The toolbox package includes visual components for creating and managing
user profiles in the system, as well as managing the access rights to the data
space objects.
At the moment, user profiles are graphically handled as
identity cards, either for editing or just for viewing purposes. They are accessible through the visual
editors. User profiles include fields
such as a name signature, description, picture, and global access rights.
Figure 4 Prototype of profile
editor panel
When editing the data space, either by introducing new
elements or interconnecting existing components, it is assumed that an
identifiable user is “logged in”. The
scheme for determining the current user is to choose from a list of registered
users or create a new one provided one has enough security rights for it. The means for intelligent user identification
and verification (the system recognizing the individual) has not been
implemented at this stage. When
created, user profiles become objects that reside in the data space [9]. There are
persistency mechanisms that ensure that user profiles and access rights objects
are always present in the data space.
Beans themselves have corresponding access objects for
individual rights settings, and editing panels of such. The creator of a bean or the user which
first introduces the bean to the data space is viewed as the bean owner. Basically, the owner assigns a general
access configuration for all potential users of the bean in the home. From then on, the owner or other users with
sufficient rights, has the option to restrict or open access to individuals provided
their user profiles exist in the space.
These restrictions can manifest themselves in a wide variety of subtle
ways, but we have implemented a set of simple schemes. For example, a user with limited access
rights editing the space with the puzzle editor would be able to see but not
interact with a restricted bean. On the
other hand, entirely concealing restricted items might prove inconvenient when
trying to adopt an overview of the current state of the system.
We begin this section by briefly recapping the purpose and
different types of transformers. We then present a set of components that we
have built for use in the home, along with example scenarios of use.
The
fundamental aim of components in our arrangement is to ensure the convergence
of the physical and the digital environment. Thus each component can be though
of as a digital/physical transformer that provides a conduit between the
real and the digital. Each component has a set of properties that it uses to
make information digitally available. The values of these properties are shared
through a distributed data space. The data space is dynamic and reacts to
changes in the values of properties; propagating these changes to all the
components that share these properties. There are three main classes of
components.
Physical to Digital Transformers take physical
effects and transform them into digital effects. Any particular device may make
use of a number of transformers. Essentially each transformer measures a
physical effect and transforms it into a corresponding digital property that is
shared through the data space.
Digital to Physical Transformers represent the
complement set of transformers. Their job is to make digital information
physically manifest in the real world. This class of components transforms the
values of shared properties to drive some sort of physical device.
Digital Transformers act upon digital information and
effect digital information. This class of components provides a way to present
to users deeper semantic reactions to changes in the environment.
Let us illustrate some of the components developed to date
using the Puzzle Editor where each transformer is represented as a puzzle
piece. However, it is worth stressing that components are available to all
editors and can be seen and assembled through these different editors.
Depending on the purpose of a specific transformer, each
puzzle piece has different numbers of ingoing and outgoing connections, and
therefore a different shape. The transformers listed below have all been tested
and demonstrated in a mock-up apartment at a SICS lab. Most of them are grouped
according to a example scenarios, which hopefully helps in explaining their
functionality
GroceryAlarm
GroceryAlarm generates names of missing groceries in the
cupboard. It detects groceries moving in and out and if one is away more than
30 seconds it is said to be out.
AddToList
AddToList takes an element string and adds it to the list it
publishes into the data space.
SMSSend
SMSSend takes a message string
and sends this as SMS to the phone number supplied as an input string.
Scenario
1: Grocery item is missing from a kitchen cupboard for some time.
Using the pieces above, GroceryAlarm is connected to
AddToList, so that the missing item is added to a list. GroceryAlarm reports
the missing item after a certain time interval. AddToList is then connected to
SMSSend, so that the list is sent via SMS to a mobile phone.
TableCommands
Reads physical "shaker" objects and generates
corresponding commands, such as NEXT, PREVIOUS etc.
News
News takes a news command string and a profile string, and
outputs a new URL to a new news web page.
KitchenTableDisplay
KitchenTableDisplay controls a web browser via a set of
commands and can also display web pages by giving the URL to it.
Scenario
2: news articles on the web can be read using phicons we call ‘shakers’
(since they are handled like salt and pepper shakers).
Connecting the pieces above, the TableCommands receives
commands (e.g. “Next”, “Previous”) from the physical interaction objects
(shakers) and controls the news service, which in turn is displayed on a
kitchen table display.
MailReceive
MailReceive takes a mail preference string as input and
outputs "To", "from", "subject", and
"body" strings of new incoming emails.
MailToBubbles
MailToBubbles listens to email "To" and
"Subject" string properties and for each change the new values are
converted to a BubbleTower command string.
SICSWebHits
SICSWebHits connects to a server checking the web log
running on the web server, which sends configurable strings for every web
access.
WebHitsToBubbles
WebHitsToBubbles listens to a web log string property and
each time it is changed the new value is converted to a BubbleTower command
string.
BubbleTower
BubbleTower listens for a bubble command change and sends to
a micro controller controlling bubble tower.
Scenario
3: Web server hits or incoming email are presented through an ambient
display – a bubble tower. Coloured lights and bubble burst sizes give extra
information.
A server application connected to SICSWebHits, on a web
server watches the web access log and reports these to the data space.
WebHitsToBubbles processes the web server access info and generates commands
for the bubble tower, which are connected to BubbleTower. In a similar fashion,
MailReceive is connects to MailToBubbles, which in turn is connected to
BubbleTower.
Reminder
Reminder presents a reminder input GUI, manages the reminder
alarms, and publishes reminders as URLs when reminder is due.
Scenario 4: A Reminder application lets the user
enter textual or auditory reminders using a touch display and microphone. This
can be connected to a display with speakers, sent to a mobile, etc.
The Reminder piece can be connected to the
KitchenTableDisplay or SMSSend.
SMSReceive
SMSReceive publishes phone number and message of last
received SMS.
Speech
Speech converts a text string into speech.
TextToWeb
TextToWeb converts a text string into an HTML file and
outputs the URL to it.
WebToText
WebToText converts the content of a URL into raw text.
During preliminary design of the puzzle editor, there were a
couple of focus group sessions to present the implemented ideas to potential
end users and obtain some focus on the direction of development of future
services and editors. At the time
several services where devised and embedded into a mock-up apartment in a SICS
lab. Our main points of observation were:
-
To obtain feedback about how intuitive the puzzle metaphor
editor was, as it was to be the primary form of configuring services for the
home. This would help sort out primary
services and configurations to address on future forms of editors.
-
To obtain information on which services fit into a real home
environment.
-
To gain some knowledge on how to fragment services so as to
allow freedom of configuration while at the same time keeping the level of
complexity in reach of end users.
-
To obtain preliminary views on concerns on issues of home
security, alarm systems for accident prevention, and access to the editing
services.
The puzzle editor was run on a touch sensitive 52” flat
display, mounted on a customized portrait frame. It had the format of a full-body mirror, in an attempt to blend
the technology with the home environment.
Embedded services included among other things: a voice reminder, a
grocery alarm, the kitchen table display, SMS services, and a water bubble
tower. Each of these services could be
configured together through transformer beans in the puzzle editor [see
Transformers, p Error! Bookmark not defined.].
Figure 5 View of ACCORD apartment
with puzzle editor embedded in mirror frame.
Three families where introduced to the ACCORD apartment, and
were briefly shown a demonstration of the puzzle editor and embedded
services. A discussion was carried out
while the families where allowed to freely make use of the apartment technology.
Upon making use of the puzzle editor immediate reactions
where on difficulties understanding icons on the puzzle pieces, and it was not
immediately apparent which pieces [hence services] one could connect
together. There were suggestions for
default and ready-made macros to choose from, perhaps borrowing/importing
macros from friends or neighbours.
Also, there were suggestions for improving feedback from activating
services or connections; such as audio feedback when actions are generated or
error sound when someone tries to join two incompatible pieces together. This demonstrated that although the puzzle
metaphor was easily grasped, poor design choices for the icon graphics and
textual descriptions of services made it difficult to couple the real world
services to their bean counterparts.
Some of the services where very fragmented at this stage: E
g creating an SMS reminder for missing groceries in the kitchen cupboard
required the coupling of four bean transformers. This, along with all the other design issues where improved on
for the current puzzle editor version.
Fragmented services where packaged together into smaller and more
clearly understandable versions.
Needless to say, there was a vast list of suggestions for
services. Although our main focus is on
configuring the services and not necessarily creating them, we will briefly
mention some of the suggestions.
The most recurring suggestion was a calendar and agenda
function for the entire family. One
that is easily understandable by the children and elders. It should be complemented with an automated
flow of different information sources into the agenda (school notes, phone
messages, mail, etc). Control over
family time being the issue, all introduced technology should aim at increasing
their free time in an otherwise full schedule. The calendar events should be
able to reach the children. Children
should for example have small toys or devices that receive text or image
messages to inform them of events (dinner ready, football training, etc). A single calendar reminder is obviously
preferred over multiple instances manifesting themselves throughout the house,
as this may cause confusion.
The second major requirement was the ability to communicate
with outside relations. Locating
friends and other family members, as well as establishing direct communication
to them ranked high on their priority list.
We should point out that a major concern is that security,
trust and function comes far ahead of any particular finesse an appliance might
have. One should be able to depend on a
service if it is to become a useful pattern in their
everyday life.
1. Java Beans - http://java.sun.com/products/javabeans/
2.
Åkesson, K-P.,
Bullock, A., “A Toolkit for User Re-Configuration of Ubiquitous Domestic
Environments”, UIST 2002.
3.
“How to be a good
bean” - http://java.sun.com/products/javabeans/docs/goodbean.pdf
4.
IBM Alphaworks - http://www.alphaworks.ibm.com/alphaBeans
5. Chris Greenhalgh, "EQUIP: a Software
Platform for Distributed Interactive Systems",
Submitted to UIST
2002.
6. JavaSpaces
http://java.sun.com/products/javaspaces
7. TSpaces - http://www.almaden.ibm.com/cs/TSpaces/
8.
OpenJGraph - http://openjgraph.sourceforge.net/
9.
Accord
Deliverable 2.4 Final Report on Toolbox Conceptualisation, Rodden, T., Koleva,
B. May 2002
