Accepted for the Poster session at the SIGIR 97,
20th International ACM SIGIR Conference on Research and Development in
Information Retrieval, Philadelphia, PA, USA
July 27 -- July 31, 1997
The IR Interaction and HCI problem
We are constantly involved in various interactions with the environment through different
communication mechanisms or processes. Information seeking is such a process, where users
in different ways interact with the information environment. Participants involved in the
interactive information seeking process, are the user of the information, the information
retrieval (IR) system, and the intermediary.
Recently, we have seen a growing interest and examples of interdisciplinary research within the information science area and the computer science area (Hewins, 1990, Sugar, 1995, and Koenemann and Belkin 1996). Current approaches to IR interaction research has created studies based on the assumption that the traditional IR does not support or reflect interaction (Belkin et al, 1995, Ingwersen, 1992, 1996, and Saracevic, 1996).
Studies in user behaviour
and individual differences (Borgman, 1989), and user interface in information
retrieval (Belkin et al 1995, Marchionini, 1995) have proved that this area
is of great interest. Questions that have attracted growing interest are:
how do we make a better adaptation to users' different preferences such
as their tasks, goals, abilities, individual differences and to support
these in the redesign of the user interface? We consider IR as an iterative
and interactive communication process between the information user, the
IR system, and the user interface with the purpose to transform the information
need into query formulations.
This means that the user interface should
have a supportive task. It is obvious that research on IR techniques solely
cannot provide the understanding, support and interaction between the user
and the IR system. A central focus of research, as Allen (1996) points out,
is the need to establish a link between research within IR and the design
of user interfaces.
A major problem inherent, and recognized, is that the methods to evaluate information retrieval systems, under a long period, have been focused on recall and precision, but not at the usability of the user interface and how well users can accomplish their goals through the system. Our present approach will be to acquire knowledge for a redesign, and we take an approach which combines the user-centered IR interaction perspective (Ingwersen 1992) and the user-centered design methods in HCI (Nielsen, 1994) on ) to evaluate information seeking interaction and tasks in a hypertext system; to identify characteristics of the user population; and finally, to make suggestions for supporting user characteristics and needs in the interface design.
We have formulated the following hypotheses: Since the current user interface to the Dienst database does not provide the users with a good support for their information seeking task. Different users show different preferences i.e. knowledge, goal, information need and expectations. Different information seeking strategies and behaviour can be observed and identified in the users' information seeking behaviour.
IR Interaction and User Interface Design
To accomplish our task, we adopt a user-centered approach, in which, based
on a modified model by Bryce (1996, p.24), we identify the following framework
of methods:
| COMPONENT | METHOD | TASK |
| Resource Analysis | Description of information system functionality | Describe resources used to complete the tasks. |
| User Needs Analysis | ||
| Task analysis | Hierarchical Task Analysis | Users tasks, goals and activities that they accomplish when meeting their needs. |
| (User Modeling) | ||
| Designing for usability | Requirement lists (qualitative data) | Requirements for user interface redesign |
The design of the user interface should, among other things, take in consideration following aspects: to support user interaction with texts and other components of the system; to support evolution of goals, and change of knowledge; and to support a variety of types of interaction.
Evaluation
Generally, one of the main task for evaluating IR systems is to obtain information
concerning the satisfaction of the userŐs task in a specific work environment.
This will include subtasks like the userŐs information need, their knowledge,
the systems functionalities etc. In many of the subtasks users are involved.
How each task is performed, and how the different tasks is connected will be
influenced by the user interface.
Within the HCI research, there has been extensive work within the usability
evaluation area. This includes highly controlled user tests in a laboratory
environment, where subjects are performing specific tasks that are observed
using different techniques like GOMS.
We have applied HCI evaluation techniques to our of IR evaluation to make a
connection between the traditional IR evaluation and HCI evaluation.
The formar usually measures precision and recall, while the latter involves
task analysis, domain description, and sometimes user evaluation and analysis.
This is already performed within the IR community, that is within the library
and information science research area.
| Data collection methods | Types of data collected | Data analysis methods |
| Internet-based evaluation
questionnaires before and after information seeking task. |
5-point Likert scale from questionnaire. Written data to the 5-point Likert scale. |
|
| Download of search log history. | Quantitative data analysis. |
The study set-up
This study is based on an previous project, initiated by the ERCIM , in
which we participated. A Dienst (Lagoze 1995) server was installed and SICS
research reports were stored, indexed and made accessible through the WWW
interface. We used an experimental online evaluation methodology setup,
combining questionnaires and log statistics of database search logs to examine
users actions, time estimates concerning the whole search and individual
actions made by the user.
To accomplish our task, we used the following
methods to collect data: Questionnaires: two questionnaires were used, one
before and one after using the system. The pre-search questionnaire explored
the user's preferences, intentions and goals and contained five questions.
The post-search questionnaire contained questions like satisfaction with
search result, navigation support issues and design issues, and contained
nine questions.
The questionnaires were set up in a non-controlled situation,
i.e. the subjects were asked to participate but not "forced" to
answer every question or comment. Each questions is followed by a Likert
scale of 1 to 5, to be checked by the user. Verbal (comments) data: In addition,
to every question, there is a "comment"-field, where the subject
could clarify or verify his/her statement in the 5-point Likert scale. Database
search log: Data collection for usage investigation will be facilitated
by logging each online user' server request. Each usage record will contain
a users identification, the total search time, search actions and errors
encountered using the system.
Data collection and analysis methods and procedures
The goal of the data analysis was to capture information about the user's
information seeking behaviour and strategies to extract lists of values
and requirements that form the framework for design decisions and feed the
information to the user through the user interface. The data collection
and analysis were conducted as follows: We contacted potential participants
in either three ways: personal contacts, electronic mailing lists and through
recommendations and the selection was based on the potential interest in
the subject domain (computer science). When the participant had conducted
the task, the answers from the questionnaires and the log statistics of
a user were linked together, creating an individual log-file of a user.
After that we created a coding scheme and ran our data through that scheme.
We examined the collected data at three levels: a general level (all users),
at a group level (all users in that group) and at an individual level (a
single verbal statement).
We used both descriptive and statistical analysis methods:
Log statistics
The mean time for a information seeking task was 11,5 minutes, and the mean
numbers of actions was 9,6 actions, which gives 1,1 minute per action. The
users spent a total of 438 minutes in the system and conducted 366 actions.
19 subjects (50%) out of 38 only browsed or searched. 17 subjects used the
search functionalities, and 2 subjects used browsing. This is evidence for
the poor support for browsing within the Dienst system. The analysis of
the user log indicates that subjects used both browse and search to accomplish
their task.
The Users
The subject entered this particular system for the first time. About 150
subjects were asked to voluntarily participate in this study. We wanted
to conduct this study in a real environment and with as real users and situation
as possible. The subjects were asked to perform some information seeking
task based on a real information need. 38 subjects (16 female , 21 male
and 1 anonymous) completed the two questionnaires. The users were separated
into the following groups: Computer Science researchers (CS), Industry (Industry)
and Information Specialists and Librarians (ISL). 37% of the participants
were computer science researchers, 24% worked within the industry, and 39%
were information specialists and librarians.
NOTE that this document only shows the preliminary result. A comprehensive list of result will be presented later. This will include correlation measures as well.
Our basic assumptions was that different users show different preferences i.e.
knowledge, goal, information need and expectations. We also believe that
different information seeking strategies and behaviour can be observed and
identified. Based on our findings, we will make suggestions for a redesign
of the user interface.
To study this, we identified the following main
categories: User background, knowledge and preferences;
User satisfaction; User behaviour; and
User requirements elicitation. The categories were
measured through a set of variables described in each category.
Single Variables examined
We collected information about the users' background knowledge and preferences.
We also made a simple structure of the users into 3 user groups. When analyzing
the written comments, we also created a list of requirements of the users'
expectations before and after using the system. The data also resulted in
a general table of user tasks and goals.
In chapter 5.2, we have examined the category of user background, knowledge and preferences (selected single variables) and results from statistical analysis is presented. We will noe present some conciderations that will influence the user interface redesign. Based on these, we will in chapter 7, make suggestion for a user interface redesign.
Relations between variables
Some preliminary results from the study of the combined variables. We found a strong relation between users with both a high level of previous
experience and a high level of satisfaction with search result.
The level of user satisfaction with search result depends on several factors.
When comparing the variable "high user satisfaction witth search result",
with other variables, we found that there was a close relation
between the variables a high satisfaction with content, satisfaction with
system effectiveness and navigation support and the variable high satisfaction
with search result.
We could also see a relation between high satisfaction with navigation
support to complete a information seeking task and high IR knowledge.
We also found a strong relation between
satisfaction with navigation support to complete a information seeking task
and domain knowledge understanding (terminology used in the system).
It seems that most of the subjects had a good knowledge about the subject domain,
and that this knowledge affected the satisfaction with navigation support.
This means that we need to design the interface according to the knowledge
resource domain.
Also, log analysis of mean time and mean action resulted
in that there is a stronger relation between time spent in the system and
satisfaction with navigation support to complete a task, rather than actions
made satisfaction with navigation support to complete a task. Subjects that
spent more time (+30 sek) in the system had a higher level of satisfaction
with the navigation support to complete a information seeking task.
ISL had the highest score for IR knowledge. The score for low knowledge on how
to formulate queries for the CS group were 21%, and for the ISL group 0%.
This shows that a distinction between the ISL and CS group.
Browse- or search-oriented strategy.
The subjects who had high level of previous experience wanted to use the
browsing strategy (36%), and the subjects who had a high IR knowledge also
wanted to browse (36%). Of all user with high level of navigation support,
59% wanted to use browsing strategies and 24% wanted to use search (analytical)
strategies. We also wanted to see if it was true what the users said that
they wanted to do and what they really did. To do this we examined the answers
given in the questionnaires, and then compared them with the users logfiles.
The result shows that the subject who said that they browsed, used the search
options, and the subject who said that they searched, also did so. The reason
for this could be that the user interface have a poor support for browsing
and that the users were "forced" to use the search functions within
the system. Furthermore we analyzed the log statistics concerning type of
action and categorized them as browse (35%) or (analytical) search actions
(65%). We found that both the CS and ISL group made equal numbers of search
actions categorized. Another observation is that the CS group made 17 actions
to inspect the retrieved item and the ISL group only one (1). The reason
for this could be the obvious: that the ISL group, by profession does not
conduct the evaluation of document, rather the customer.
User Interface Design
The design of effective human-computer interfaces becomes ever more critical to overall system performance. One task for the design of user interfaces
would be to reflect the users' tasks and information problem, and help users
to solve tasks by being adaptive and supportive to a user's knowledge and
goals. We have identified several levels of work that must be understood
in order to understand information seeking activities: the users' specific
preferences, tasks and goals; the users' information seeking behaviour;
and the use of an information system. Suggestions for a user interface design
will be presented later, based on the knowledge acquired from studies above.
Our study has also produced a simple model for knowledge acquisition for
user interface design (and redesign) of information retrieval systems.
References
Allen, B. (1996). From Research to Design: A User-Centered Approach. In:
Ingwersen, P. and Pors, N. O. (eds.). Second International Conference on
Conceptions of Library and Information Science: Integration in Perspective
(CoLIS 2), Copenhagen, Denmark. October 13-16, 1996. pp. 45-59.
Belkin, N. J., C. Cool, A. Stein and U. Thiel. (1995). Cases, Scripts, and
Information-Seeking Strategies: On the Design of Interactive Information
Retrieval Systems. Expert Systems With Applications, 1995, Vol. 9 (3), pp.
379-396.
Benyon, D. and K. Höök. (1997). Navigation in information Space:
supporting the individual. INTERACT'97. Australia. (Forthcoming)
Borgman, C. (1989). All users of information retrieval systems are not created
equal: an exploration into individual differences. Information Processing
& Management, 25 (3), p. 237-252
Bryce, A. (1996). Information Tasks. Towards a User-Centered Approach to
Information Systems. San Diego : Academic Press.
Chen, C. and R. Rada. (1996). Interacting With Hypertext: A Meta-Analysis
of Experimental Studies. Human-Computer Interaction, Vol. 11 (2). Pages???
Hewins, E. T. (1990). Information Need and Use Studies. In : Williams, Martha
E. (ed.). Annual Review of Information Science and Technology (ARIST), Volume
25, 1990. pp. 145-172. Amsterdam, The Netherlands, Elsevier : ASIS.
Ingwersen, Peter. (1992). Information Retrieval Interaction. London : Taylor
Graham.
Koenemann, J. and N. Belkin (1996). A case for interaction: A study of interactive
information retrieval behaviour and effectivness. In : Proceedings of CHI'96
Human Factors in Computing Systems. New York : ACM Press. pp. 205-212
Lagoze, C. and Davis, J. (1995). Dienst: An Architecture for Distributed
Document Libraries. Communications of the ACM, 1995, 38 (4), p. 47.
Marchionini, Gary (1995). Information Seeking in Electronic Environments.
Cambridge Series on Human Computer Interaction 9. Cambridge : Cambridge
University Press.
Nielsen, J. and R. Mack (eds.) (1994) Usability Inspection Methods. New
York : John Wiley & Sons.
Saracevic, T. (1995). Evaluation of evaluation in information retrieval.
In: Fox, E. A., P. Ingwersen, and F. Fidel (eds.). Proceedings of the 18th
Annual International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR'95). Seattle, Washington, USA. July 9-13, 1995.
pp. 138-146. New York: ACM.
Su, L. T. (1992). Evaluation measures for interactive information retrieval.
Information Processing & Management Vol. 28 (4), pp. 503-516.
Sugar, W. (1995). User-Centered Perspective of Information Retrieval Research
and Analysis Methods. In : Williams, Martha E (ed.). Annual Review of Information
Science and Technology (ARIST), Volume 30, 1995. pp. 77-109. Medford, N.
J. : ASIS.
1997-04-12