TURF: A Unified Framework for Defining, Evaluating, Measuring, and ...

TURF: A Unified Framework for Defining, Evaluating, Measuring, and Designing EHR Usability Jiajie Zhang & Muhammad Walji National Center for Cognitive Informatics & Decision Making in Healthcare

What is Usability? • Usability refers to how – useful, – usable, – satisfying • a system is for the intended users to accomplish goals in the work domain by performing certain sequences of tasks Zhang J, Walji MF. TURF: A Unified Framework of EHR Usability. Manuscript under review.

2

TURF Framework for EHR Usability Functions Useful Intrinsic Complexity

Users

Extrinsic Difficulty

Representations

Electronic Health Record

Tasks

Satisfying

Usable System Usability

3

How to Measure Usefulness? • Function Saturation • Overhead

4

Function Saturation

=

#Domain Functions in Designer Model #Domain Functions in All Three Models

37 = 46% 80

A Small EDR System

5

(From Chen, 2008)

Overhead in Designer Model

=

Domain Functions

CPOE 2

# Overhead Functions #Domain Functions + #Overhead Functions

Overhead Functions

64%

CPOE 1

36%

58%

0%

20%

42%

40%

60%

80%

100% 6

How to Measure Usableness? • Learnability – – – –

# trials to reach a certain performance level # items that need to be memorized # sequences of steps that need to be memorized Etc.

• Efficiency – – – –

Time on task Task steps Task Success Mental effort

• Error Prevention and Recovery – Error occurrence rate – Error recovery rate 7

Time on Task (seconds) CPOE ePrescribing Clinical Summary Smoking Status Problem List Growth Chart BMI Vital signs Demographics 0

100

200

300

400

500

600 8

# of Task Steps CPOE ePrescribing Clinical Summary Smoking Status Problem List Growth Chart BMI Vital signs Demographics 0

100

200

300

400

500

600

700

9

Mental Effort Physical

Mental

CPOE ePrescribing Clinical Summary Smoking Status Problem List Growth Chart BMI Vital signs Demographics

33.3% 25% 33.8% 52% 36.2 % 38.1 % 37.5% 31.6% 37.9%

0%

20%

40%

60%

80%

100% 10

How to Measure Satisfaction? • User’s subjective impression of how likable the system is • Likert scale type of evaluations – System Usability Scale (SUS) – AHRQ Survey Instruments for EHR usability (in development) – Other measures 11

How to Improve Usability? TURF in Action 12

TURF in Action: Redesigning EHR  OpenVista: Open Source EHR  NIST Use Case §170.302(e)  Maintain active medication allergy list  Add allergy; Modify allergy; Review allergy

Time (seconds)

# Steps 250 200

199

187

21

21

Record

150

Modify 97

91

100 50

75

Retrieve

79

82

15 14

31 10

50

81

Total

41

0 OpenVista: Steps Harrington, Wood, Breuer, Pinzon, Howell, Pednekar, Zhu, & Zhang (2011)

Prototype: Steps

OpenVista: Timing

Prototype: Timing 13

TURF in Action: Redesigning EHR

Harrington, Wood, Breuer, Pinzon, Howell, Pednekar, Zhu, & Zhang (2011)

14

TURF in Action: Redesigning EHR  81% reduction in Overhead Functions  89% increase in Domain Functions OpenVista 120 100

Prototype

99

80 53

60 40 20

19

28

0 Overhead Functions Functions Overhead Harrington, Wood, Breuer, Pinzon, Howell, Pednekar, Zhu, & Zhang (2011)

DomainFunctions Functions Essential 15

Usability at SHARPC • Rapid Usability Assessment Protocol • Comprehensive TURF Evaluation • Redesign • Training • Short Course • AMIA 10x10 Course • Certificate, MS, & PhD 16

Summary • Usability is defined • Usability is measurable • Usability is doable

17

EHR Usability Symposium October 21, 2011 Washington Hilton Pre-AMIA Event