Fidelity Considerations for Simulation-Based ... - Semantic Scholar

Fidelity Considerations for Simulation-Based Usability Assessments of Mobile ICT for Hospitals

Authors: Yngve Dahl (corresponding author) Telenor Group Business Development & Research Telenor ASA 7004 Trondheim, Norway Phone: +47 905 27 892 E-mail: [email protected] Ole Andreas Alsos Department of Computer and Information Science Norwegian University of Science and Technology 7491 Trondheim, Norway Phone: +47 915 44 825 E-mail: [email protected] Dag Svanæs Department of Computer and Information Science Norwegian University of Science and Technology 7491 Trondheim, Norway Phone: +47 918 97 536 E-mail: [email protected]

Fidelity in Simulation-Based Usability Assessments Abstract: We have conducted controlled usability evaluations of mobile ICT for hospitals. As part of these evaluations, clinicians have acted out mobile work scenarios and used the systems to solve related tasks. The evaluations show that relevant usability factors go beyond that of graphical user interfaces. Some of these usability factors only show up when the realworld context of use is replicated in the laboratory to a high degree of fidelity. The complexity of the context of use for mobile ICT in hospitals has motivated us to explore training simulation fidelity theory. Based on a review of the training simulation literature, we identify a set of fidelity dimensions through which training simulations often are adjusted to meet specific goals. We argue that the same mechanisms can be used in usability assessments of mobile ICT for hospitals. We substantiate our argument by using the identified set of fidelity dimension in a retrospective analysis of two assessments. The analysis explains how the fidelity composition contributed to the identification of relevant usability factors. Keywords: Clinical information systems, mobility, simulation, simulation fidelity, usability assessment, user-centered design.

2

Fidelity in Simulation-Based Usability Assessments

1. Introduction This paper investigates the concept of fidelity and its role in controlled usability assessments of mobile information and communication technology (ICT) for hospitals. Within HCI literature, fidelity has traditionally been used to describe the extent to which a software application reproduces visual appearance, interaction style and functionalities during evaluation (Virzi, 1989; Virzi, Sokolov, & Karis, 1996). In conventional PC-based usability testing, the physical and social aspects of the use situation are of little concern. Interaction with PC-based systems is highly uniform and static with regard to the physical and social aspects of the use situation—one user sitting in front of a PC, with his or her attention directed mainly on the computer screen. As interaction with ICT becomes more mobile in nature and take place in dynamic work settings, such as hospitals, the old standards for usability testing no longer hold. Clinical work is characterized by extensive mobility, rapid context shifts, changing work priorities, and close interaction between different actors (Bardram & Bossen, 2005; Reddy, Dourish, & Pratt, 2006; Sørby, Melby, & Nytrø, 2006). As a use setting for ICT, these characteristics make hospitals very different from office environments – the prototypical use environments for desktop computers. In contrast to relatively static office environments, the usability of mobile ICT supporting clinical work is also likely to depend on factors beyond the GUI and software solution per se, including physical and social aspects of the use situation. Such external factors cannot be addressed through conventional usability testing in laboratories designed for evaluating desktop computer applications. Over the last five years we have done controlled usability tests of a number of mobile ICT systems for hospitals, both formative evaluations of prototype systems and summative evaluations of products ready for deployment. In these usability assessments hospital workers 3

Fidelity in Simulation-Based Usability Assessments have used the products to solve specific tasks during enacted scenarios. The scenarios have been acted out in a laboratory that has been custom made to mimic a hospital ward section. The laboratory has movable walls in a 10 x 8 meter room that allows for full-scale simulations of different hospital settings. The rooms are equipped with patient beds, chairs and tables to create a high level of realism. We have consulted health workers in this process. Gradually, we have become aware that our approach has strong elements of simulation – a method associated with skill training in various high-risk industries, such as aviation, naval shipping, and health care. Fidelity is a central concept in simulation research and in the design of training programs. In contrast to the case for HCI, simulation research often describes fidelity as a multi-dimensional concept, encompassing various aspects of the research setting. In training simulations, the various fidelity dimensions are often tailored to fulfill specific goals (Liu, Macchiarella, & Vincenzi, 2008). We argue that to assess usability of mobile ICT for complex use settings, such as hospitals, evaluators need to carefully consider the fidelity of the research setting vis-à-vis the actual performance context. For simulations to work as an effective tool in the design process, it is critical to identify the right level of simulation fidelity. Similar to the case for training simulations, the fidelity of simulation-based usability assessments can be adjusted to achieve targeted trials that help participants focus on specific aspects of the simulation experience. Drawing on two earlier formative usability evaluations, we first aim first to show that fidelity in usability evaluations of mobile ICT is a concept that extends beyond the prototype GUI being evaluated. Particularly, when addressing hospital settings, where the technology is likely to be used as part of work activities requiring manual labor with hands and feet in addition to high situational awareness, elements of the use context become vital components of the total system being simulated. 4

Fidelity in Simulation-Based Usability Assessments Our second objective of this paper is to demonstrate how fidelity theory from training simulation research can be applied as a guiding framework for composing targeted usability evaluations of mobile ICT for complex use settings, such as hospitals. Research on training simulations has identified various fidelity components or dimensions that can be adjusted to achieve goal-specific training. Based on a set of relevant simulation fidelity components we will conduct a retrospective analysis of the main findings from the two usability evaluations. The analysis provides a rational for why the two assessments succeeded in evoking relevant user responses and behavior. This is conceptualized in a simulation acceptance model. The main purpose of the paper is to draw attention to the relevance of simulation fidelity theories in the design of formative evaluation addressing the usability of mobile ICT for clinical work. The general structure of the paper is as follows. In Sect. 2, we will provide a definition of the term simulation-based usability assessment, and describe the motivation behind the approach. Sect. 3 will briefly outline the limited understanding of the fidelity concept in HCI vis-à-vis simulation research. In Sect. 4, we will draw some parallels and distinctions between training simulations and usability assessments. Next, in Sect. 5, we will present relevant theory from simulation research, and identify a set of fidelity dimensions through which simulation-based training often is adjusted. Sect. 6 will present the design and results from two former simulation-based usability assessment, in which mobile ICT for hospitals have been evaluated. In Sect. 7, we analyze how the fidelity configurations for the assessments helped us identify relevant usability factors, and propose a simulation acceptance model. Sect. 8 will shed light on issues of relevance to the design of simulation-based usability assessment, and identify some limitations of the current study. Finally, a brief summary along with concluding remarks is provided in Sect. 9.

5


2. Simulation-Based Usability Assessments 2.1. Definition A simulation is broadly defined by Beaubien and Baker (2004) as "a device that attempts to re-create characteristics of the real world". Simulations can generally be divided into two categories. The first category consists of computer-generated models of a system or an environment. This models can be digitally simulated according to predefined rules of operation (McGrath, 1995). The second category includes simulations in which a particular system is imitated in the physical world with representative actors of that system as participants. In the current work we will focus on simulations fitting the latter category. The term simulation-based usability assessment is used here to refer to a usability test in which the design concept being evaluated is employed by end users enacting in constructed work scenarios in natural-like physical environments. Generally, such evaluations are designed to reproduce specific contextual factors of the real-world setting in a controlled manner. Kjelskov and Skov (2007) refer to such approaches as in sitro evaluations. A scenario-based usability assessment fulfills the general features of simulations identified by Gagné (1962): •

It attempts to represent the real situation in which actions are performed

•

It provides the simulation participant with some control over the situation.

•

It deliberately omits some aspects of the real situation.

A fundamental challenge related to all simulations, then, is what aspects of the performance context can be omitted without compromising the validity of the results.

6


2.2. Motivation Below, we will provide a brief account of the methodological and practical issues that has motivated us to follow a simulation-based approach. 2.2.1. Methodological Concerns The question of which research setting that is optimal for studying mobile usability has been addressed in earlier studies (Kjeldskov, Skov, Als, & Høegh, 2004; Kaikkonen, Keklinen, Cankar, Kallio, & Kankainen, 2005; Nielsen, Overgaard, Pedersen, Stage, & Stenild, 2006; Rogers, Connelly, Tedesco, Hazlewood, Kurtz, Hall, Hursey, & Toscos, 2007). The studies, however, offer different views. One explanation for the ambiguity in the results is that the question of optimal research setting essentially is a question of which research criteria you want to maximize. McGrath (1995) identifies three desirable, but conflicting, criteria when studying phenomena or events in social and behavioral sciences. These are generalizability of research evidence, precision of measurements, realism of the situation or the context being studied. McGrath argues that different research methodologies prioritize different criteria. Simulation-based usability assessment, which is the focus in the current investigation, corresponds to what McGrath classifies as experimental simulation. Experimental simulation is a compromising strategy in which one attempts to partly maintain the control associated with laboratory experiments, while retaining some of the realism associated with experiments conducted in the field. Some of the main methodological benefits of evaluating mobile usability in simulated contexts rather than in the field include: •

Laboratory settings provide immediate access to relevant situations. Relevant use situations can be created “on demand” and repeated.

7

Fidelity in Simulation-Based Usability Assessments •

Laboratory settings allow evaluators to “pause” situations and collect feedback from participants while the episode is still fresh in their minds.

•

Laboratory settings offer the possibility to do detailed video and audio recordings.

2.2.2. Practical and Ethical Concerns In addition to the methodological motivation described above, there are also practical and ethical reasons why simulation-based approaches can be valuable when studying mobile usability. Field evaluations may have and obtrusive effect on the care situation being studied, and discretion need to be shown with regard to sensitive patient information. The latter may prevent the use of audio and recording equipment, which are valuable data collection tools when studying usability. Simulation-based usability assessments, then, can be seen as a pragmatic approach when field evaluations are infeasible due to security and safety reasons.

3. The Fidelity Concept in HCI Within HCI, fidelity has traditionally been thought of in engineering terms, referring to the extent to which a prototype system is perceived as authentic or realistic by end users (Virzi, 1989). Thus, in the case of high-fidelity prototypes, a user should experience little or no differences between the prototype and the end product. In this sense, the fidelity concept has been considered a one-dimensional feature limited to describing aspects of the prototype. This understanding of the fidelity concept corresponds with traditional models of humancomputer interaction. Typically, these models denote the user and the computer as two information-processing units in closed-loop system (Kaptelinin, 1996). Phenomena existing outside the loop are excluded. Consequently, it has made little sense to talk about the fidelity of the physical and social use context when evaluating usability of software running on

8

Fidelity in Simulation-Based Usability Assessments desktop computers. Recreating the prototypical use context of desktop computers for usability evaluation purposes is generally a trivial issue. As further discussed in Sect. 5, the conceptualization of fidelity as a one-dimensional feature is contradicted in simulation research literature. Within simulation research the concept is used not only to describe the accuracy of devices used during trials but also denotes the accuracy of the context in which the trial are held. This understanding of fidelity is highly relevant in the design of simulation-based usability assessments.

4. A Comparison of Training Simulations and Usability Assessments Within the context of usability assessments, simulation of real-world phenomena in controlled laboratory environments represents a relatively novel approach. Some early usability studies of mobile systems and services, in which key contextual features have been replicated in laboratories, are described by Bohnenberger et al. (2002), Pirhonen et al. (2002), and Kjeldskov and Skov (2003). However, there is little HCI literature describing how to effectively compose such evaluation, and the mechanisms through which they can be adjusted to fit a particular purpose, e.g., to inform specific design issues. This has motivated us to search for guiding principles beyond what can be found in HCI and computer science literature. In particular, we have studied literature from training simulation research. To highlight similarities (and differences) between training simulations and usability assessments we will continue by making a conceptual comparison of the two. Simulation-based approaches have long since been systematically used for training purposes in safety-critical industries. In the education of, e.g., aviation pilots and ambulatory care personnel simulations form an essential part of the training (Patrick, 2002; Maynes, 2008).

9

Fidelity in Simulation-Based Usability Assessments For trained professionals, simulators are often used to maintain (or adapt) skills and practice drills. The objective of training simulators is mainly to develop human work-related skills1. This makes transfer of training, from simulations to real-world practice, a critical issue in training design (Liu et al, 2008). Within research on training simulations there has been an ongoing debate on the degree of realism, or fidelity, that need to be mirrored in simulations in order to maximize transfer of training. While training simulations aim to optimize human task performance, usability assessments are conducted with the intention of measuring and, in the case of formative evaluation, identifying ways to enhance the performance of a product. Similar to training simulations, usability evaluations are specific with regard to participants, the objective of the participants, and the context in which the activities are carried out. Training simulations seek to maximize participants’ skill transfer from the simulated context to the real-world context. Usability assessments typically seek to maximize design knowledge relevant for future development of a product, by gathering data on product acceptance and usability. Thus, the two types of exercises target different stakeholders. The “learners” in usability assessments are not the test participants, but the evaluators. Table 1 sums up the main difference between training simulations and usability evaluations. In many ways, these differences make training simulations and usability evaluations complementary when it comes to optimizing interaction with tools used as part of workrelated activities. Training simulations is about adapting people for specific tasks, while formative usability evaluation is about adapting tools for specific tasks and users.

1

In certain cases, knowledge and attitudes may also be targeted through simulation (Beaubien & Baker, 2004). For the sake of simplicity, the current paper will only consider simulations used for skill training. 10

Fidelity in Simulation-Based Usability Assessments Training simulations

Usability Evaluations

Enhance human skill

Evaluation of product

performance in a specific

performance relative to a

context.

particular context.

Trainee.

Evaluator.

Role of technology

Part of simulator

Product to be evaluated

Role of participant

As trainee

As representative user

Output

Skill.

Product acceptance and

Purpose

Learner/knowledge recipient

usability. TABLE 1

Comparison of training simulations and usability evaluations.

Current training simulation research strongly suggests that optimization of skill transfer is intimately dependent on multiple factors including the type of skill being trained, who the trainees are and their level of experience, the circumstances in which the training takes place, and available resources (Liu et al., 2008). These different parameters typically guide how training simulations are composed. Different simulation fidelity components are typically adjusted to meet the objectives and premises of the training. The position argued in the remainder of the current paper is that much of the same intimacy between objectives and premises on the one side, and how training simulations are designed on the other side, also applies to simulation-based usability assessments. Effective training simulations are tailored to promote acquisition and transfer of specified skills in relation to particular circumstances. We argue that usability assessments of mobile ICT for hospitals are most effective in when they are tailored to promote end user reflection concerning specific aspects of design. The underlying principle of simulations for both training and formative

11

Fidelity in Simulation-Based Usability Assessments usability assessment purposes is very similar. In both cases, the multitude of factors affecting human cognition and behavior in the actual performance contexts makes it infeasible to address them all at once, and understand their relevance and relationship. Systematic decomposition of design problems is by no means a new concept in HCI and usercentered design. Iterative design processes gradually increases the fidelity prototypes from low to high. This is a central feature of the user–centered design process, as defined in ISO 13407 (1999). As pointed out earlier, this decomposition has typically only concerned the prototype (typically the graphical user interface) and to a far less degree been applied to the external use context.

5. Simulation Fidelity Theory Having drawn some parallels and contrasts between training simulations and usability assessments, we will now turn attention to aspects of simulation fidelity theory of particular relevance when designing usability evaluations of ICT supporting mobile clinical work. Drawing on relevant literature, this section will describe some central mechanisms or components through which simulations for training purposes are modified to maximize their effect.

5.1. Simulation Fidelity Components As previously noted, fidelity in simulation training plays an important part in the transfer of skills from exercise to reality. At an overall level, simulation fidelity defines the reality of the simulation (Alessi, 1988; Gross, Pace, Harmoon, & Tucker, 1999). The exact understanding of what “reality” consists of varies. Over the years, multiple fidelity dimensions have been suggested (Rehmann, Mitman, & Reynolds, 1995). Some of these suggestions have been highly simulator specific in the sense that they only apply to a given type of training device

12

Fidelity in Simulation-Based Usability Assessments for a specific domain. Motion fidelity in flight simulators, i.e., the extent to which the simulator realistically mirrors in in-air movement, is an example of this. In the following review, we have deliberately focused on fidelity dimension that are more universal, and that can be used to describe simulations in a more general sense. Simulation fidelity can be understood both in relation to the physical experience on the one hand, and the psychological or cognitive experience on the other. This has motivated different approaches to resolving the transfer problem. Physical fidelity and psychological fidelity can be divided into subcomponents. Figure 1 presents an overview of the components or dimensions of simulation fidelity that we will focus on in the current work.

FIGURE 1

Simulation fidelity dimensions.

5.1.1. Physical Fidelity (Engineering Fidelity) Early transfer research emphasized the need for simulations to duplicate the physical elements of their real-world counterparts – The more the training situation resemble the characteristics of the real task situation, in terms of operational equipment and environment, the more effective transfer of training can be expected. This hypothesis is also known as identical elements theory (Thorndike & Woodworth, 1901). Equipment fidelity and environment fidelity are often referred to as subcomponents of physical fidelity.

13

Fidelity in Simulation-Based Usability Assessments Equipment fidelity. Equipment fidelity refers to the extent to which the appearance and feel of real tools, devices, or systems (hardware and software) that simulation participants operate on is replicated (Zhang, 1993). For example, aircraft cockpit procedures have been trained both with hi-fi representations of aircraft instruments and with lo-fi mock-ups (Prophet & Boyd, 1970). Environment fidelity. Environment fidelity concerns the extent to which physical characteristics of the real-world environment (beyond the training equipment) are realistically represented in the simulation (Kinkade & Wheaton, 1972; Beaubien & Baker, 2004). In the case of flight simulators, this corresponds to visual, auditory, and motion stimulus. Highfidelity aircraft simulators are full-size replica of cockpits that duplicate the operational aircraft environment and motions to great detail (Rehmann, Mitman, & Reynolds, 1995). In flight training environments of less fidelity (e.g., desktop evaluation environments), visual and motional cues are often reduced or absent. Within training simulation, physical reproduction of the actual performance environment has traditionally been considered a prerequisite for maximum transfer. The underlying motivation behind simulations prioritizing physical realism is that the physical similarity between training and performance context will enhance the potential for skill transfer. High physical fidelity, combined with training scenarios based on actual events, has been the main answer to transfer-of-training in domains such as aviation and military. While the physical fidelity approach has been proved effective for numerous tasks, its main challenges is that it can be costly in terms of resources (Liu, Macchiarella, & Vincenzi, 2008), and that it potentially can increase the cognitive workload on participants and thereby impede learning (Alessi, 1988).

14

Fidelity in Simulation-Based Usability Assessments 5.1.2. Psychological Fidelity The idea that physical resemblance between constructed situation and actual situation is essential for training transfer was later questioned, when it was discovered that cognitively oriented tasks could effectively be practiced using mock-ups to represent key elements of the situation. Prophet and Boyd (1970) highlighted this in an early comparative study of aircraft cockpit procedures trained in authentic environments and with low-fidelity representations of aircraft instruments. Similar findings, indicating that high-fidelity environments and devices are not always required for positive training transfer, is described in work by Hays et al. (1992) and Duncan and Feterle (2000). The methods employed in these studies have put emphasis on what within in training design is referred to as psychological fidelity. Psychological fidelity concerns the degree to which the simulation captures key psychological processes of the performance domain (Kaiser & Schroeder, 2003; Kozlowski & DeShon, 2004). It affects the extent to which a trainee psychologically and cognitively engages in the training situation (Kaiser & Schroeder, 2003), and is closely related to how participant perceives the “plot” (actions, events, and changes in circumstances) of the simulation, vis-à-vis the real-world situation. Human perception, attention, decision-making, memory, and action are factors that may influence psychological fidelity (Patrick, 2002). Similar to physical fidelity, psychological fidelity plays a key role in achieving positive reactions among simulation participants. By trigger the central cognitive mechanisms relevant for on-the-job performance, rather than focusing on replicating the concrete performance environment, the psychological fidelity approach represents an alternative technique for promoting engagement among participants. High physical fidelity approaches, on the contrary, tries to implicitly capture the key psychological processes trough realistic reproduction of the performance environment (Kozlowski & DeShon, 2004). 15

Fidelity in Simulation-Based Usability Assessments The psychological fidelity perspective is an emerging approach within training design (Kozlowski & DeShon, 2004). It is in many ways analogous to low-fidelity prototyping approaches in HCI (Kuutti, Iacucci, & Iacucci, 2002; Svanæs & Seland, 2004). Task fidelity and functional fidelity are often considered subcomponents of psychological fidelity. Task fidelity. Task fidelity describes the degree to which tasks involved in the actual environment for a given domain is replicated in the simulation (Zhang, 1993; Roza, 2000; Hughes & Rolek, 2003). This typically affects the extent to which participants experience the simulation as operationally realistic. In high-risk domains this is often a critical component. In full-mission flight simulations, for example, deviation from operational practice is generally experienced as negative user acceptance cues and might elicit different pilot behaviors during training vis-à-vis the real-world performance (Rehmann, Mitman, & Reynolds, 1995). Developing scenarios that replicate task demands of the real-world system is one of the techniques used for enhancing psychological fidelity in training simulations (Beaubien & Baker, 2004). In training exercises, task are sometimes isolated to investigate certain issues (Liu, Macchiarella, & Vincenzi, 2008). Thomas (2003) suggests that task fidelity is often equated with the physical and functional fidelity of the simulation. This tendency, it is argued, has biased technological advancement in development of training simulation, while less focus has been placed of recreating authentic task scenarios. Functional fidelity. Functional fidelity describes the degree to which the simulation reacts like “the real thing”, i.e., that it provides realistic responses on the tasks and actions executed by the participant. This does not necessarily concern only the functional fidelity of the training equipment per se. For example, flight simulators with high functional fidelity does 16

Fidelity in Simulation-Based Usability Assessments not only contain flight control devices that acts like real equipment, but also provide realistic aircraft motion cues in response to interaction with those devices. In order for simulation participants to learn the exact consequences of their actions, high functionality is often required. A flight simulator, for example, must maneuver as a real aircraft in order to for positive transfer of such training aspects. Functional fidelity and task fidelity are both essential for the credibility of training, which is a key premise for skill transfer.

5.2. Factors Affecting Simulation Fidelity Configuration In the design of simulation training there are multiple factors influencing the overall fidelity configuration. Below, we will present some of the key factors. 5.2.1. Goals of Training A fundamental challenge related to training design is identifying a suitable level of simulation fidelity without compromising positive transfer. As suggested in literature on training design, this is intimately dependent on the goal of the training (Maran & Glavin, 2003; Beaubien & Baker, 2004). In maximizing training transfer, physical fidelity and psychological fidelity can play complementary roles (Kozlowski & DeShon, 2004). Studies have shown that when training cognitively demanding tasks and procedures, high transfer can be achieved with simple simulations (Prophet & Boyd, 1970). For such purposes, high-fidelity components can be removed without having negative effect on transfer. This is especially the case if the component is unimportant for the skill being trained (Maran & Glavin, 2003). For inexperience subjects practicing basic task skills, advanced equipment and undesired

17

Fidelity in Simulation-Based Usability Assessments interference can be inappropriate and actually diminish transfer as it can divert trainees’ attention from the goal of the simulation. 5.2.2. Cost-to-Benefit While high-fidelity simulations attempt to minimize the differences between training and transfer situation, such approaches are often associated with high costs. This has made the fidelity issue in training design a question of cost-to-benefit. Drawing on Miller (1953), Alessi (1988) suggested that the degree of simulation fidelity should ideally correspond to the training stage of the learner. Alessi argued that beyond a certain level, increasing the fidelity of the training device would yield diminishing transfer. Thus, increasing engineering fidelity beyond a certain level will not produce sufficient extra transfer to make it worth the added costs. Kinkade and Wheaton (Kinkade & Wheaton, 1972) proposed that the ideal relationship between physical fidelity and psychological fidelity changes as the experience of the simulation participants increases. By focusing on replicating the cognitive demands, rather than prioritizing engineering fidelity, psychological fidelity approaches form a cost-effective alternative to simulations. 5.2.3. Other Factors Hays and Singer (1989) identified multiple factors influencing the relationship between simulation fidelity and transfer, including the abilities of instructors, instructional techniques, types of simulators, measurement techniques, etc.

5.3. Summary of Key Simulation Fidelity Considerations Based on the studies referenced in this section we have identified a number of fidelity considerations that are relevant for training design. Below, we will first present a brief summary of the key points from our literature review.

18


The fidelity concept: At an overall level there are two major simulation fidelity components – Physical fidelity and psychological fidelity. Physical fidelity is the exactness to which real world operational equipment and environment are replicated in the simulation. Psychological fidelity refers to the degree to which the central cognitive processes relevant for the skill being trained are triggered. Each of the two components can be further divided into subcomponents. Fidelity in modern training design is considered a multi-dimensional feature.

•

User acceptance: Reflecting a sufficient degree of fidelity is a necessary for the trainees to accept the simulation as a replacement for the real world performance context. This acceptance is required for provoking realistic behavior and positive training transfer.

•

Effectiveness: There is no direct correlation between the level of fidelity and effectiveness in form of training transfer.

•

Fidelity configuration: Fidelity is only critical in terms the role it plays in the simulation experience. Simulation fidelity (both physical and psychological fidelity) needs to be carefully adapted so that it matches the objectives and the content of the training and training level of the participants.

•

Iterative process: A training program will usually require different levels of simulation fidelity as it progresses.

5.4. Relevance of Simulation Fidelity Considerations in the Design of Mobile Usability Evaluations If we compare the fidelity consideration above with the role of fidelity in usability evaluations, there are some noteworthy parallels as well as distinctions.

19


The fidelity concept: With regard to the general understanding of fidelity, as it is described in HCI literature, it is to a much larger extent treated as a one-dimensional feature, typically refereeing to how closely the graphical user interface prototype matches the final product. The concept is normally not used to describe the accuracy of a simulated use context.

•

User acceptance: Positive transfer of training strongly depends on the users acceptance of the simulation as a credible surrogate for the actual performance context. Similar to training simulations, user acceptance is also critical in usability evaluations, but for different reasons. A central issue in formative usability evaluations is to provoke behavior and feedback for participants that can help inform future design. This means that users need to be able to relate an evaluated design to work-related tasks. If not, one cannot expect them to provide valuable feedback.

•

Effectiveness: Effective training transfer is not a direct function of simulation fidelity. Similarly, the quality of results from usability assessments does not per se depend on whether a low-fidelity or high-fidelity prototyping approach was employed. It is rather that different fidelity approaches in usability evaluations produce different types of usability data. Low-fidelity prototype approaches typically put emphasis on understanding the purpose of a computer system and context in which it will be used. What are people trying to accomplish? Which processes are involved? What are the essential user requirements? High-fidelity prototyping approaches, on the other hand, are far more concrete with regard to the design solution being evaluated. Consequently they tend to gather responses that to a much larger extent are solution specific.

•

Fidelity configuration: Simulation for training purposes attempts to tailor the fidelity level of the trial to match the phase of the training program. Likewise, the level of

20

Fidelity in Simulation-Based Usability Assessments prototype fidelity in usability evaluations typically depends on which design phase it relates to. Training simulations, however, has to a much larger extent than usability evaluations systematically taken into account the performance context in which control devices are used. •

Iterative process: Skill training and design of interactive systems are iterative processes. In both cases the relevant performance context is typically to complex to understand in a one-time effort. Instead, one first attempts to understand the basic requirements, and then gradually addresses more complex issues.

6. Elements of the Use Context Affecting Mobile Usability in Hospitals To empirically ground our previous claim that the usability of mobile ICT in hospitals is determined by multiple factors that are highly contextual, we will present examples from two previous simulation-based assessments of relevance. The respective results from the two assessment are describe in detail in previous work (Svanæs & Alsos, 2006; Dahl & Svanæs, 2008). In the current work, we will focus on aspects of the two simulations that have not been discussed earlier – their fidelity compositions and the compositions’ impact on the results.

6.1. Overall Experimental Design Both assessments aimed to explore alternative interaction techniques supporting mobile hospital workers at the clinical point of care (i.e., at the patient’s bedside). The assessments were conducted in a large usability laboratory configured as a hospital ward section. The main purpose of the experimental design for both assessments was to evaluate product usability in a relevant context of use. According to ISO 9241-11, usability is always relative to the context in which the product is used. In the standard, context of use is defined as 21

Fidelity in Simulation-Based Usability Assessments “users, tasks, equipment (hardware software and materials), and physical and social environment in which a product is used.” Figure 2 provides a conceptual view of the various components of the research setting, including the evaluated prototype, the simulated context of use, and the usability laboratory with technical facilities (e.g., recording equipment).

. FIGURE 2

Conceptual view of the experimental design.

6.2. Assessment 1: Combining Handheld Devices and Bedside Patient Terminals In Assessment 1, we explored the potential for letting physicians use handheld devices (PDAs) as input device for the bedside terminals. Eight different prototypes of this setup were implemented, including a baseline solution where all interaction was done directly on the patient terminal touch screen without using a PDA. The eight alternative designs were tested in a pre-surgical scenario where a physician uses a bedside terminal to show X-ray images to a patient. Figure 3 shows two of the design solutions. The illustrated solutions allowed the physician to use select an X-ray image by dragging it to a terminal icon on the

22

Fidelity in Simulation-Based Usability Assessments PDA (left), or to use the PDA as a remote control to navigate in a menu on the bedside terminal (right).

FIGURE 3

Design solutions for controlling patient terminals from a PDA.

The usability tests were done in a simulated use context replicating a patient room with a hospital bed, a touch screen bedside terminal, and a PDA. A total of five pairs, one physician and one simulated patient, were recruited for the tests. Figure 4 shows a physician interacting with one of the design alternatives.

FIGURE 4.

Test subject interacting with a PDA to select an X-ray image to show to

the patient. After testing all versions, the physicians’ and patients’ opinions about the design solutions were collected in a post-test interview and through a ranking exercise. The comments made during the tests and in a post-test interview were analyzed. This gave insights into the factors that were perceived as influencing the usability of the solutions. 23

Fidelity in Simulation-Based Usability Assessments 6.2.1. Fidelity Configuration for Assessment 1 Regarding the fidelity configuration for Assessment 1, prototype (equipment) fidelity can be characterized as medium. Prototypes were represented with real hardware, but only simplified GUIs and information content (X-ray images) were employed. Environment fidelity can be characterized as medium to high. It was configured using a real patient bedroom a model, but certain physical elements were removed (e.g., patient uniforms, blankets, paper charts attached to beds, medical equipment, etc.). We consider task fidelity to be medium. A physician confirmed the realism and validity of the overall scenario in a pretest interview. However, we realize that low-fidelity information content may have affected how the simulated pre-surgical consultation was carried out. Functional fidelity can be categorized as being medium to high. The prototypes supported all the relevant interaction techniques and provided feedback to users. In addition, the patient actors had been given a brief instruction on how to behave during the assessments and what type of responses to provide the test participants. 6.2.2. Summary of Identified Usability Factors in Assessment 1 Screen size and ergonomic aspects of devices. All participants reported that the patient terminal screen was large enough for viewing X-ray images, while the PDA screen was too small for this purpose. Positioning the patient terminal within the range of the patient and the physician made it easy for both to see the X-ray images. The terminal position made it easy for the patients to use, while some physicians were uncomfortable with the solution, as they had to bend over the patient’s bed to reach it. Some physicians commented that a good thing about the PDA-based design alternatives vs. the baseline alternative was that they no longer had to bend over the patient’s bed to interact with the terminal.

24

Fidelity in Simulation-Based Usability Assessments Shared view vs. hiding information. One recurring issue during the interviews was whether the list of available X-ray images should be on the patient terminal or only on the PDA. Most physicians wanted to hide it because it distracted the patient, wasted valuable screen space, or displayed patient sensitive information. The patient actors, however, felt that the physician was keeping secrets for them when the list was not present. Focus shifts. Almost all physicians commented that the PDA became a disturbing element in the communication with the patient and that the change of focus between the PDA and the patient terminal was quite demanding. The participants preferred less demanding design alternatives that did not disturb them. When the physicians and the patients looked at or interacted with the same screen, they felt that they were communicating on an equal “level”. When the physicians started using the PDA, some of them expressed that it became a disturbing element in the dialog and that they now were communicating on different “levels”.

6.3. Assessment 2: Automatic Identification of Patients at the Point of Care The aim of Assessment 2 was to evaluate and compare the usability of different sensor-based techniques for automatic patient identification during medication administration in a ward. During the drug administration round, a health worker distributes prescribed medicine to ward patients, and signs off on the respective patients’ medication chart after each administration. For simplicity, the chosen test setup involved only two in-bed patients. It was assumed that the correct medicine dosage for the respective patients was carried in the hospital workers’ pockets. The problem being addressed in the developed prototypes was that of identifying the correct patient at the point of care. By adding new ubiquitous-computing technology to the mobile

25

Fidelity in Simulation-Based Usability Assessments EPR, such as token readers or location sensing, patient identification can be automated thereby reducing potential for error during administration. Four different design solutions were evaluated. The four alternatives were combinations of two sensing technologies and two device technologies (Figure 5). The two sensor technologies were barcodes (token-based) and WLAN positioning (location-based). The employed positioning system continuously detected the physical position of all WLAN devices in the simulated patient room to an accuracy of approximately 0.5 meter. PDAs (mobile) and bedside touch-screen terminals (stationary) formed the two device technologies. An implicit assumption in the prototype implementations was that the devices could retrieve medication charts from an EPR system. The user interface for the medication chart was extremely simplified, as the focus of the study was not on medication charts or the GUI, but on automatic identification of patients.

FIGURE 5.

Two of the four design solutions that were evaluated. The left image

shows location-based interaction with electronic patient charts being presented on a mobile device. The right image shows a token-based variant with patient charts being presented on a fixed bedside terminal. Eight hospital workers from a local hospital participated in the assessment. Persons with experience from health care acted as patients during the simulations. The test participants

26

Fidelity in Simulation-Based Usability Assessments were encouraged to interact with the simulated patients in a similar way as they would interact with real patients in an authentic care situation. As in Assessment 1, the participants were asked to rank the design solutions in order of preference and to explain their priories. The transcripts from the ranking sessions were examined to identify factors that influenced their ranking. 6.3.1. Fidelity Configuration for Assessment 2 The fidelity configuration for Assessment 2 can be summed up as follows. Prototype fidelity can be described as being medium to low. Similar to Assessment 1, the simplified GUIs and information content offered by the prototypes reduced their fidelity in spite of using real computing devices as part of the test. Similar to the previous assessment, the environment fidelity can be described as medium to high. The lack of physical representations of the medicine to be administered as well as medication trays was the primary features that were missing from the simulation vis-à-vis the performance context. We regard task fidelity as being medium. At an overall level the simulation reflected the key tasks that are part of an administration round (e.g., moving between patients, informing about medication to be administered, signing), but details concerning the administration was deliberately omitted. This, for example, included physically handing over medicine to patients. Similar to the case for Assessment 1, we consider the functional fidelity of Assessment 2 to be relatively high. The sensor-based interaction techniques were implemented using high-fidelity sensor technology. As with the previous assessment, the patient actors had been given general instructions on how to respond during the simulation. 6.3.2. Summary of Identified Usability Factors in Assessment 2 Attention on computer devices vs. attention on patient. Many test participants expressed a general concern that cumbersome information navigation would require them to pay too 27

Fidelity in Simulation-Based Usability Assessments much attention to the computer devices, rather than attending the patient. They consequently all saw the benefit of automatic patient identification. The two location-based interaction techniques were ranked high. These design alternatives made use of the clinician’s natural mobility in the physical environment. The fact that these techniques allowed patient identification to occur in the background of the user’s attention can be viewed as an important reason for their high rating. In order to retrieve patient information via tokens (i.e., barcodes), the users had to explicitly scan them. The test participants who preferred location-based interaction to token-based interaction argued that barcode scanning took attention away from the patient and the care situation. Predictability and control. Earlier work on context-aware and ubiquitous computing has suggested that autonomous computer behavior often implicates loss of user control (Bellotti & Edwards, 2001; Barkhuus & Dey, 2003). The conducted usability tests revealed similar findings. Users that preferred token-based interaction to location-based interaction found that getting computer response as a result of an explicit and deliberate action gave them a feeling of greater control over the application. This made the users more certain that they were signing off on the correct patient medication chart. We found that the potential lack of control some users experienced when testing the locationbased solutions was related to the fact that the zones in the room were invisible. Thus, the physical test environment did not provide any visible cues to participants when to expect system response based on their physical movements. Despite the lack of control many users experienced with the location-based solution, many were willing to give up control as long as it made patient identification easier.

28

Fidelity in Simulation-Based Usability Assessments Integration with work situation. Most test subjects commented that when administering medicine in their everyday work, they were accustomed to informing the patient verbally what medicine he or she was given. Many of the test participants therefore saw an additional benefit of having the opportunity to visually show medical information to the patient via the shared screen of the bedside terminal. Accomplishing this via the small screen on the PDA was experienced as being far more cumbersome. Several test participants expressed that another benefit of using stationary patient terminals versus a portable device was that it allowed them to have both hands free. This was seen as important as they often perform tasks at point of care that require both hands free (e.g. hand over medicine, help patients in and out of their beds). Hence, most test subject found the fixed bedside terminals to be more integrated with the overall work situation, while the PDA imposed more of a physical constraint. One of the potential drawbacks of the implementation involving a stationary device, as pointed out by test participants, was related to privacy. When using a shared screen it is also possible for others (e.g., patients and visitors) in the room to see the information.

6.4. Summary of Results As the results presented above from suggest, the two assessments identified a number of usability factors external to those of the GUI of the evaluated design solutions. Assessment 1 identified issues concerning screen size and ergonomics of patient terminals, sharing (and hiding) of patient related information, and focus shifts between multiple devices during patient visits. Assessment 2 identified issues related to time spent on computer devices vs. time spent on patients, predictability and user control, and integration with physical and social aspects of the work situation.

29


7. Analysis While we regard the results above highly relevant in terms of informing future design of mobile ICT for hospitals, they also raise some interesting implications for how we evaluate the usability of such technology. In particular, we consider the issue of how the respective fidelity configurations of the two assessments contributed to the findings to be relevant. Essentially, the analysis provided below highlight the close relationship between fidelity configuration of the simulation-based usability assessments and the type of feedback we received from the participants.

7.1. Environment Fidelity The custom-designed laboratory that the usability evaluations were conducted in provided a test environment accurate with regard to the physical size and configuration of patient rooms that the test subjects knew from their everyday work. The significance of replicating key physical aspects, such as room proportions, patient beds, and in-bed patients, is particularly evident if we consider the physical and bodily aspects of usability that were identified through the experiments. An example of this is the findings concerning shared information displays. In both assessments, participants found it highly valuable to share a common screen display with in-bed patients. Participants commented this after they had been given concrete ideas of the physical configuration of different design solutions, i.e., the physical placement of the terminal in relation to the patient bed and in relation to the in-bed patient. In the setup shown in Figure. 4 (p. 23), the patient actor in the bed is not only needed for facilitating a patient-caregiver dialog during test scenario. Responses from participants also suggested that the patient actors’ physical position and posture also helped give an idea of how suitable the design solutions were for sharing information between caregiver and patient. In addition these physical factors gave an idea about the ergonomic suitability. In many cases the

30

Fidelity in Simulation-Based Usability Assessments participating hospital workers used the patient actor in the bed and the terminal as physical reference points in the room. Another simple example highlighting how certain physical aspect of the environment can help detect ergonomic requirements is shown in Figure 6. A recurring comment from the test participants was that during point-of-care situations they could strongly benefit from having ICT solutions that are physically ready at hand, and that can be brought forth or temporarily stored in the background depending on what the immediate care situation calls for. During the evaluations, test participants would occasionally put handheld terminals in the pocket of their work uniform when they, e.g., needed to physically examine patient actors. The handheld terminal would later be picked up to complete the simulated tasks. Thus, outfitting test participant with actual hospital uniforms helped indicate that hospital workers can benefit from mobile media having a physical size that easily fit their pockets.

FIGURE 6

In this case the work uniform the test subject wears helps identify a

relevant usability requirement – The handheld terminal has an acceptable size, fitting the test pocket of the uniform, making both hands free for the clinician. Although the examples above may seem trivial when considered in isolation, they serve to illustrate the intimate relationship between the physical environment and the size and form

31

Fidelity in Simulation-Based Usability Assessments factors of interactive devices used at the point of care. They illustrate how physical cues in the test environment can help them relate design solution to their everyday work. By being concrete with regard to certain physical features (room proportions, furnishing, in-bed patient, work uniforms) the test environment explicitly encouraged participant to think about physical and bodily aspects of their work in relation to the solutions. At the same time, none of the participants responded negatively to physical features (e.g., medical drugs and trays, blankets on beds, patient uniforms) that were missing in the simulations, but normally be a part of actual performance context. We have not been able to find results from conventional usability test of applications running PCs that identify similar physical and bodily usability factors. This suggests that participants need to be exposed to a sufficiently realistic physical work environment during testing in order for such usability issues to surface.

7.2. Prototype (Equipment) Fidelity Both assessments involved the use of working prototypes. The fidelity of the prototype GUIs were deliberately kept low, showing only simplified graphical representations of medical information. This was essentially to promote feedback describing how users experienced novel interaction techniques in relevant work situations, which was primary objective of the evaluations. The behavior of the participants during simulations and comments in post-test interviews suggested that they, despite simplified GUIs and medical content, perceived the simulation experience as sufficiently realistic. For example, test participants frequently evaluated the different candidate solutions against current (and often paper-based) practices, and made usability related comments as if the simulation experience had occurred in the actual performance context. Often, these comments would be related the form factors of design solutions, and how they accommodated the care situation.

32


7.3. Task Fidelity Task fidelity plays a central role in helping participants place the simulation in a context – In our case, relate the simulation experience and design solution to their everyday work. The tasks that the test participants were given were all related to particular clinical scenarios (administration of medical drugs, pre-surgical briefings), but fairly open in the sense that they did not strictly dictate how the participants performed the tasks. The main motivation for providing the test participants fairly generic tasks was that we wanted to do an early exploration of the design space for each of the cases, and that we did not have a precise understanding of relevant usability factors. As such, we considered it important to allow for central usability factors arise out of the way the test participants chose to solve the tasks, and draw on their work experience. We also recognized that providing more detailed tasks would possibly require higher GUI prototype fidelity and more accurate medical information content. Based on our experience, the tasks play an important role in providing participants credible reasons for executing behavior of relevance during usability assessment. For example, the tasks made sure that the participant would communicate with the simulated patients and move between different beds, and interact at least once with each candidate solution during the evaluation.

7.4. Functional Fidelity As previously noted, we considered the functional fidelity of the conducted assessment to be fairly high. By employing high-fidelity sensors and interactive devices in the tests, the participants where arguably given a relatively realistic experience of how the different design concepts could work in an actual point-of-care context with state-of-the art technology. In particular, this was relevant with regard to the findings concerning predictability and control.

33

Fidelity in Simulation-Based Usability Assessments We consider it unlikely that test participants would have reflected on these issues without being given a realistic impression of how the design concepts present themselves to users in practice. The patient actors also played a part in providing realistic responses to the tasks and actions executed by participants. Among, the patient actors most important functions was to help the test scenarios progress by responding to information from and questions asked by the participating clinicians. In addition, they served as the primary focus of attention for the clinicians during the simulations.

7.5. Simulation Acceptance Model We have applied simulation-based usability assessments to evaluate early prototypes of mobile ICT for hospitals. In order to provoke realistic behavior and design relevant reflections among hospital workers during simulated care scenarios, however, they need to accept the simulation experience as a credible replacement for real-world episodes. Simulation fidelity plays a key role in this context. Drawing on the analysis above, Figure 7 proposes a simulation acceptance model. The figure gives a conceptual view of the factors that influence the extent to which a simulation experience evokes commitment and engagement among participants. The overall simulation fidelity configuration affects how each participant perceives the realism of the simulation experience. Evaluators may adjust the various simulation fidelity dimensions according to the goal of the assessment. The required accuracy of each fidelity dimension, however, is ultimately dependant on the individual participant. It is essential that all fidelity dimensions exceeds or equals a lower threshold (f) for him or her. Failure to do so is likely to cause a negative user experience, and make it difficult for the participant to reflect on the usability of a prototype. However, it can be argued that high-fidelity components may partly compensate for low-fidelity components if

34

Fidelity in Simulation-Based Usability Assessments they interact closely (Schricker, Franceschini, & Johnson, 2001). If all fidelity dimensions meet the requirements of a specific test subject (∏), simulation acceptance is achieved. Simulation acceptance is one of the key factors affecting the engagement and commitment of participants during a trial. Other influencing factors include personal interests, attitudes towards technology and approach, rewards for participating, etc.

FIGURE 7

Simulation acceptance model.

8. Discussion In the current section we will briefly discuss some issues of relevance to simulation-based usability assessments of mobile ICT for hospitals.

8.1. Replicating a Sufficient Degree of Realism? Rather than incorporating highest possible fidelity across multiple dimensions, the analysis above suggest that a more feasible approach is to carefully select which aspects of the performance context to replicate accurately, and which aspects to simplify or remove. The motivation behind customizing the simulated use context this way is that it allows evaluators to gradually increase the complexity of the simulated use context as the product is developed. 35

Fidelity in Simulation-Based Usability Assessments A fundamental question that arises from such an approach is: How can you tell design-time if you are incorporating a suitable degree of realism into a simulation-based usability assessment, so that relevant user reflections are evoked? We have suggested various aspects of the research setting (fidelity components) that may affect the perceived realism of a simulation-based usability assessment, and that these have to be adjusted according to what issues that one wants to put focus on. Yet, we recognize that there is no definite way of telling a priori whether the gap between the simulated context and the performance context is successfully bridged or not. Having a simulator (i.e., a customized usability laboratory) allowing for realistic mobility and care situations to be acted out in a physically realistic environment is not a “valid” system for evaluating mobile usability for hospitals per se. Rather, validity of usability data is dependant on the focus of the evaluation and the context from which the data was derived. Below, we describe some applicable techniques and measures that can be taken to help ensure that a simulated use context can generate valid usability data. 8.1.1. Using Consultants from Health Care During preparations of simulation-based usability assessments we have used hospital workers as consultants. Nurses and physicians have assisted in identifying clinical situation that could benefit from mobile ICT. Moreover, they have participated in pilot tests to help verify that the structure of the simulated task scenarios is in accordance with real hospital practice. 8.1.2. Field Studies One of the main motivations for employing simulation-based usability assessment has been to overcome practical and ethical challenges of studying mobile usability in the field. This is not to say that simulations eliminate the need for field studies. On the contrary, field studies described, e.g., by, Bardram and Bossen (2005), Munkvold and Divitini (2006), and Sørby et al. (2006), give a rich picture of the dynamic nature of clinical work providing insights to 36

Fidelity in Simulation-Based Usability Assessments how information tools are used, and how hospital workers constantly face interruptions and changes tasks. Thus, field studies provide knowledge that are essential for understanding the underlying processes of clinical routines. This is critical for designing operationally realistic simulations. We also recognize the need for post-simulation field studies to help validate results. In this sense, we consider simulation-based usability assessments and filed studies complementary approaches. 8.1.3. Learning Process Learning to use simulation-based usability assessments as an effective tool in the design of mobile ICT for hospitals is a continuous process. Factors that typically affect fidelity configuration include the goal of the assessment (i.e., which design aspects are being addressed), cost-to-benefit of increased fidelity, and how far the evaluated concept is from becoming a finalized product. Balancing these criteria requires experience.

8.2. Limitations of the Current Study The current study is retrospective in nature. We have analyzed data from former usability evaluations of mobile ICT for hospitals, using simulation fidelity theory to identify how various aspects of the research setting contributed to the findings. This stands in contrast to developing a theoretical basis concurrently as studies are carried out. Weaknesses of retrospective studies include possible selection bias towards examples from trials that confirms, rather than rejects, a theory. We also recognize that the lack of systematic evaluations, in which different simulation fidelity configurations are applied to the same test case, makes it impossible to compare and estimate the impact different setups have on identified usability issues.

37


9. Summary and Concluding Remarks 9.1. Summary In the current work we have discussed the relevance of simulation fidelity theory in the design of assessments addressing the usability of mobile ICT for hospitals. In training simulations design, fidelity is considered a multi-dimensional concept. Various features of simulations are typically adjusted to meet specific training goals. By systematically adjusting fidelity of various features, simulations can form effective training tools helping participants to focus on relevant issues. We have argued that the principle of carefully adapting the simulated context is highly applicable for usability evaluations in order for them effectively inform design. This is particularly relevant when evaluating mobile ICT for complex use settings, such as hospitals. We supported our argument by drawing on result from two relevant usability assessments. The assessment indicated that relevant usability factors in mobile clinical care situations often are related to aspects of the external use context. We then how the overall fidelity configuration for the two assessments helped evoke relevant behavior and reflections among test participants. Based on the review we proposed a simulation acceptance model. The model describes key features affecting the extent to which simulation-based usability assessments evoke commitment and reflection among test participants.

9.2. Concluding remarks This paper has presented an alternative view to the fidelity concept compared to the common understanding of the term in HCI. Drawing on simulation research we have proposed that the concept can also be used to describe the accuracy of various features in simulated use context beyond the evaluated products. The need to evaluate usability of ICT in realistic context of use is acknowledged in HCI and in international standards. Applying simulation fidelity 38

Fidelity in Simulation-Based Usability Assessments theory in the design of usability evaluation targeting complex use contexts, gives a more nuanced view on the level of realism required in order for participant to accept simulated trials. Rather than aiming for high fidelity across multiple dimensions, a more feasible approach is do be selective about which dimensions of the use context to faithfully replicate. By providing a sufficient degree of realism in usability assessments, as opposed to testing in fully authentic settings, evaluators can systematically address how various elements of the use context affect usability. Evoking relevant user behavior and feedback during simulation-based usability assessments is essentially about setting the stage (i.e., the research setting) right, and making participants accept the illusion of the simulation. Setting the stage right is intimately dependent on the objective of the assessment. If the objective is changed, the stage needs to be adjusted accordingly. Simulation fidelity theory has helped us become aware of the strong relationship between the fidelity configuration of simulation-based usability assessments and the type of usability factors that are identified through such trials. We encourage further research on this topic.

References Alessi, S. M. (1988). Fidelity in the Design of Instructional Simulations. Journal of Computer-Based Instruction 15(2): 40-47. Bardram, J. E. & Bossen, C. (2005). Mobility Work: The Spatial Dimension of Collaboration at a Hospital. Computer Supported Cooperative Work 14(2): 131-160. Barkhuus, L. & Dey, A. (2003). Is context-aware computing taking control away from the user? Three levels of interactivity examined. Proceedings of Ubicomp 2003: 159-166. Beaubien, J. M. & Baker, D. P. (2004). The use of simulation for training teamwork skills in health care: how low can you go? Qual Saf Health Care 13(51-56). Bellotti, V. & Edwards, K. (2001). Intelligibility and accountability: Human considerations in context-aware systems. Human-Computer Interaction 16(2, 3 & 4): 193-212. Bohnenberger, T., Jameson, A., Kruger, A., & Butz, A. (2002). Location-Aware Shopping Assistance: Evaluation of a Decision-Theoretic Approach. Mobile HCI 2002, London, UK, Springer Verlag.

39

Fidelity in Simulation-Based Usability Assessments Dahl, Y. & Svanæs, D. (2008). A comparison of location and token-based interaction techniques for point-of-care access to medical information. Personal and Ubiquitous Computing 12(6): 459-478. Duncan, J. C. & Feterle, L. C. (2000). The use of personal computer-based aviation training devices to teach aircrew decision-making, teamwork, and resource management. Proceedings of the IEEE 2000 National Aerospace and Electronic Conference, Dayton, OH. Gagné, R. M. (1962). Simulators. Training Research and Education. Glaser, R. New York, Wiley. Gross, D. C., Pace, D., Harmoon, S., & Tucker, W. (1999). Why fidelity? Proceedings of the Spring 1999 Simulation Interoperability Workshop. Hays, R. T., J.W., J., Prince, C., & Salas, E. (1992). Flight simulator training effectiveness: a meta-analysis. Mil. Psychol. 4(2): 63-74. Hays, R. T. & Singer, M. J. (1989). Simulation fidelity in training system design: Bridging the gap between reality and training Springer-Verlag. Hughes, T. & Rolek, E. (2003). Fidelity and validity: issues of human behavioral representation requirements development. Proceedings of the 2003 Winter Simulation Conference, New Orleans, LA. ISO 9241-11 (1998). Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) -- Part 11: Guidance on Usability. ISO 13407 (1999). Human-centred design processes for interactive systems. Kaikkonen, A., Keklinen, A., Cankar, M., Kallio, T., & Kankainen, A. (2005). Usability testing of mobile applications: A comparison between laboratory and field testing. Journal of Usability Studies 1(1): 4-17. Kaiser, M. K. & Schroeder, J. A. (2003). Flights of fancy: the art and science of flight simulation. Principles and Practice of Aviation Psychology. Vidulich, M. A. and Tsang, P. S. Mahwah, NJ, Lawrence Erlbaum Associates: 435-471. Kaptelinin, V. (1996). Activity Theory: Implications for Human-Computer Interaction. Context and Consciousness: Activity Theory and Human-Computer Interaction. Nardi, B. A. Cambridge, MA, USA, Massachusetts Institute of Technology: 103-116. Kinkade, R. & Wheaton, G. (1972). Training device design. Human engineering guide to equipment design. Van Cott, H. and Kinkade, R. Washington, DC, Department of Defense: 668-699. Kjeldskov, J. & Skov, M. B. (2003). Creating Realistic Laboratory Settings: Comparative Studies of Three Think-Aloud Usability Evaluations of a Mobile System. Interact 2003, IOS Press, Amsterdam. Kjeldskov, J. & Skov, M. B. (2007). Studying Usability In Sitro: Simulating Real World Phenomena in Controlled Environments. International Journal of Human-Computer Interaction 22(1): 7-36. Kjeldskov, J., Skov, M. B., Als, B. S., & Høegh, R. T. (2004). Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. Mobile HCI 2004. Kozlowski, S. W. J. & DeShon, R. P. (2004). A psychological fidelity approach to simulation-based training: theory, research and principles. Scaled Worlds: Development, Validation and Applications. Schiflett, S. G., Elliott, L. R., Salas, E. and Coovert, M. D., Ashgate Publishing. Kuutti, K., Iacucci, G., & Iacucci, C. (2002). Acting to Know: Improving Creativity in the Design of Mobile Services by Using Performances. Proceedings of the 4th conference on Creativity & cognition, Loughborough, UK, ACM, New York, NY, USA.

40

Fidelity in Simulation-Based Usability Assessments Liu, D., Macchiarella, N. D., & Vincenzi, D. A. (2008). Simulation Fidelity. Human Factors in Simulation and Training Vincenzi, D. A., Wise, J. A., Mouloua, M. and Hancock, P. A., CRC Press. Maran, N. J. & Glavin, R. J. (2003). Low- to high-fidelity simulation – a continuum of medical education? Medical Education 37 (Suppl. 1): 22-28. Maynes, R. (2008). Human patient simulation in ambulatory care nursing. AAACN Viewpoint 30(1). McGrath, J. E. (1995). Methodology Matters: Doing Research in the Behavioral and Social Sciences. Readings in Human-Computer Interaction: An Interdisciplinary Approach. In Baecker, R. B., W. A. S. San Mateo, CA, Morgan Kaufman Publishers. Miller, R. (1953). Handbook on training and training equipment design. Munkvold, G. & Divitini, M. (2006). From storytelling to reporting - converted narratives. In proceedings of MCIS’06, the Mediterranean Conference on Information Systems, Venice, Italy. Nielsen, C. M., Overgaard, M., Pedersen, M. B., Stage, J., & Stenild, S. (2006). It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. NordiCHI 2006, Oslo, Norway, ACM, New York, NY, USA. Patrick, J. (2002). Training. Principles and Practice of Aviation Psychology. Tsang, P. S. and Vidulich, M. A., CRC Press. Pirhonen, A., Brewster, S., & Holguin, C. (2002). Gestural and Audio Metaphors as a Means of Control for Mobile Devices. Proceedings of the SIGCHI conference on Human factors in computing systems. Prophet, W. W. & Boyd, H. A. (1970). Device-Task Fidelity and Transfer of Training: Aircraft Cockpit Procedures Training. Tech. Report. Human Resources Research Organization, Alexandria, VA. Reddy, M., Dourish, P., & Pratt, W. (2006). Temporality in Medical Work: Time also matters. Computer-Supported Cooperative Work 15(1): 29-53. Rehmann, A., Mitman, R., & Reynolds, M. (1995). A handbook of flight simulation fidelity requirements for human factors research. Tech. Report No. DOT/FAA/CT-TN95/46. Wright-Patterson AFB, OH: Crew Systems Ergonomics Information Analysis Cente. . Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R. E., Hursey, J., & Toscos, T. (2007). Why It’s Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp. UbiComp 2007, LNCS 4717. Krumm, J. e. a., Springer-Verlag Berlin/Heidelberg: 336–353. Roza, M. (2000). Fidelity considerations for civil aviation distributed simulations. Proceedings of the AIAA Modeling and Simulation Technologies Conference and Exhibit, Denver, CO. Schricker, B., Franceschini, R., & Johnson, T. (2001). Fidelity evaluation framework. Proceedings of the IEEE 34th Annual Simulation Symposium, Seattle, WA. Svanæs, D. & Alsos, O. A. (2006). Interaction Techniques for Using Handhelds and PCs Together in a Clinical Setting. NordiCHI 2006, Oslo, Norway. Svanæs, D. & Seland, G. (2004). Putting the users center stage: role playing and low-fi prototyping enable end users to design mobile systems. Proceedings of the SIGCHI conference on Human factors in computing systems, Vienna, Austria, ACM. Sørby, I. D., Melby, L., & Nytrø, Ø. (2006). Characterising cooperation in the ward: framework for producing requirements to mobile electronic healthcare records. International Journal of Healthcare Technology and Management 7(6): 506-521. Thomas, M. J. W. (2003). Operational Fidelity in Simulation-Based Training: The Use of Data from Threat and Error

41

Fidelity in Simulation-Based Usability Assessments Management Analysis in Instructional Systems Design. In Proceedings of SimTecT2003: Simulation Conference. Thorndike, E. L. & Woodworth, R. S. (1901). The influence of improvement in one mental function upon the effciency of other functions. Psychological Review 9: 374-382. Virzi, R. A. (1989). What can you learn from a low-fidelity prototype? In Proceedings of the Human Factors Society 33rd Annual Meeting., Santa Monica, CA. Virzi, R. A., Sokolov, J. L., & Karis, D. (1996). Usability problem identification using both low- and high-fidelity prototypes. Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, Vancouver, British Columbia, Canada, ACM, Neew York, NY, USA. Zhang, B. (1993). How to consider simulation fidelity and validity for an engineering simulator. Am. Inst. of Aeronaut. Astronaut.: 298-305.

42