Wireless Usability Report 2001 - Bitly

HASTINGS RESEARCHINC.

Internet Development and Infrastructure

Wireless Usability 2001-2002: A Glass Half-Full Wireless devices and services have—through many wrong turns— started to become usable in the U.S. and Canada. The key to effective implementation is in wisely choosing devices, services, and feature sets.

Executive Briefing — p. 2 Table of Contents — p. 7 Body of Report — p. 9

By Nicholas Carroll, Sheldon Brahms, Mardee McGraw, Deborah Rodgers © 2001-2002 Hastings Research, Inc. All rights reserved. www.hastingsresearch.com

License for this copy of WUR2001-2002 Report: free for individual users. Not for resale.

Executive Briefing Synopsis This report is based on a study designed and conducted by Hastings Research to determine the present state of usability for wireless devices, protocols, and connectivity in the U.S. The study consisted of extensive user testing, with a total of 25 subjects (17 of them IT professionals, and 8 non-technical users), 10 wireless services, 6 portals, and 23 devices, including a variety of cellular phones, Blackberries, Palms and Handsprings, and handheld PCs (HP Jornada and Compaq IPAQ). Methodology We conducted the study on a qualitative basis rather than quantitative. From long experience, we prefer good qualitative studies to bad quantitative studies. (And from equally long experience, we know that most quantitative studies can be misleading, unless you know exactly what questions to ask.) Subject Training The test was for usability of wireless access to the Web — plus email. To make things simple, we showed each subject the basic keys and navigation, which key or icon to get onto the Web, and they were on their own from that point. This allowed each subject to start from the same point in the learning curve. ______________________ Devices Used We chose the following devices for the study, out of almost three dozen choices at our disposal. Cell Phone: Sprint TP 2100 Blackberry: 957/857 Palm OS: Palm VIIx Handheld PC: HP Jornada Cellular Phones One thing stood clear about web-enabled cell phones: they connected. While over half the users grumbled about “2400 bps downloading,” the connections were at least reliable. Past that, opinions varied widely. Readability varies from phone to phone depending on the brightness of the screen and backlighting. Joystick features for navigation make them easy to use although the size and touch sensitivity of the joystick do make a difference in how they were rated. Would users equip their company with these? Some would, saying “They’re excellent 2

for quick messages and light web surfing.” Most would not, feeling that the small display and difficult typing placed cell phones at a deadend for web access. Blackberries When it came to email, Blackberries just worked. Sending and receiving email were almost flawless. Web access was not as easy. The Blackberries came with different services preinstalled: one using the GoAmerica portal, and another with Yahoo!. Our testers didn’t like using the Yahoo portal because of the default search engine, Ask Jeeves. They would have rather been able to use Google. Would users equip their company with these? All but one user replied that they would happily equip their company with Blackberries for email. They were more cautious about the benefits of web access, feeling that GoAmerica was not quite perfected. Palms The Palm VIIxs came preinstalled with palm.net service. Connecting through palm.net was adequate, however accessing the Web was not. Apparently Palm has placed their strategic partnerships before the needs of their users. Users were able to send email with ease, in fact the Palms almost matched the Blackberries in ease of use on email. But they found WWW access a disaster. Would users equip their company with these? Users responded with “No,” or “Never with Palm.net.” (We carried out further Palm testing with AvantGo and EudoraWeb, observations included in the body of the report.) Handheld PCs Handheld PCs, both the Compaq IPAQs and the HP Jornadas we used in the test, had slow and unreliable connections. Users downgraded them for waiting time, regardless of the usability of the actual interface. However, users liked them for the novelty of the color screen, and familiar Windows Internet Explorer interface. Would users equip their company with these? Excited by the color display and graphics rendering, most said “Yes!” Then they would begin hedging, saying “... maybe, if the connectivity improves.” ______________________ The Best Best web access: Palm OS devices using AvantGo or EudoraWeb, or Blackberries using GoAmerica. Best cellular phone Web access service: Yahoo! and Google Best email service: Blackberries. In our view Blackberries have real potential, despite their relatively high cost. They are the best available device for email, which remains 3

the killer app of the Internet. Best Handheld PC: Inconclusive. When it comes to wireless, we are not sure that they currently fit in anywhere. The screens are far too narrow to display most web sites. Like Palms and Blackberries, they are largely restricted to text-only web sites. In connectivity they were consistently the worst devices we tested. The fact that Windows CE will run on them does not mean they give true Windows usability. As best we can tell, most handheld PC owners use them as “PIM-plus” devices; a personal information manager with a little bit more capability than a Palm or Blackberry, and the ability to access Microsoft files in a pinch. ______________________ Conclusions & Strategies Preliminary Conclusion Contrary to previous reports by other research companies, we found that wireless devices themselves were far from perfected. However, wireless access has indeed arrived in modest but often useful ways. And on some devices, certain features have been nearly perfected. Blackberries have become increasingly popular in spite of their high price because they provide excellent service with that traditional killer app: email. Even cell phones have been successfully enabled; address lookup and driving directions are extremely useful to road warriors. The fact that users become adroit with such devices merely proves that mobile telephony is a killer app. It does not demonstrate that users are willing to learn their way around the Web using devices with illogical controls and poor ergonomics. Just as many VCRs are used purely for playback of rented movies—recording being too complicated to be worth the owner’s time—we expect that most wireless devices currently available will be used for the absolute minimum necessary tasks.

Strategies 1. Equip and train your employees to use wireless devices with currently available services Choosing Devices - look at many before testing, test several before buying. We tested 11 cell phones for the study. Not a single one was fully satisfactory (reasons why we excluded most of them are listed in the Appendix). However, if the Sprint TP 2100s had the larger screen, good backlighting, and the high quality font hinting of the Motorola Nextel L85, they would have been thoroughly satisfactory. So when it comes to cell phones, testing 30 different phones is probably a realistic minimum. Figure on a week of initial research by two or three varied employees. This 4

would involve taking our checklist, visiting multiple computer stores and cell phone suppliers, and simply testing the ergonomics, keypad logic, and readability of several different types of phones.

2. Rework your intranet to support your employees, suppliers, or B2B customers. Obviously the employees that would benefit most from being able to connect directly to the company intranet would be the field staff. They could look up availability of product, check for technical updates, or company news. If they were selling software they could download the latest version for demonstrations. They could access other more mundane but useful information like company phone lists, email addresses, company health benefits and other employee information. Enabling your remote staff to connect and access public files cuts down on administrative support. It also cuts down on wasted time, especially when you are dealing with a 3 hour time difference from one coast to the other. Suppliers & B2B customers: although in theory this is a good idea, most manufacturers still don’t want buyers to know exactly what they have and how much. It interferes with their pricing strategies. Most manufacturers would rather keep their databases off-line. Suppliers of raw materials like oil, gas, metals and minerals are different, being more subject to a market economy. Their prices constantly fluctuate based on how much is available and what the demand is. Providing this information via wireless might be an added convenience for their buyers.

3. Rework your Internet access (web site) to support niche markets or the general public. If your market is financials, and your customers are day traders, you may want to rework your web site to enable wireless access. You have to judge your market carefully—and do the numbers. If your market is pet food, you probably won’t want to rework your site for wireless access. Yahoo! and Google have done a good job of making their sites usable to even the most display-challenged of wireless devices, cellular phones. However, these are both broadmarket plays, that will live or die by popular appeal. And the services that a portal like Yahoo! provides are the types of services that consumers want: news, restaurant listings, weather reports, driving directions, and yellow pages. Of those, only the last three are likely to be of value to business users.

Cellular Phone Checklist These are a few of the questions you should ask when looking at buying cell phones: Devices 5

1. Are the controls intuitive for both voice and WWW use? 2. Can they be used by people of varying hand size and fingernail length? 3. Can they be one-handed? 4. Is the screen legible under both good and poor lighting? 5. Do they have a minimum of 6 lines of text visible? 6. Is the screen legible to older users? 7. Is the speaker volume loud enough? 8. Are rapid-chargers available for road warriors? Service Checklist 1. Is there adequate service to employee’s offices, travel routes, and homes? 2. Check both voice and web access coverage areas. Testing Your Service Area Once you have settled on some potential devices, have your employees test wide, test far. Wireless service providers themselves have numerous employees roaming the countryside looking for coverage gaps. Unfortunately, while they often try to fill those gaps, they rarely bother to tell customers about them. Coverage is caveat emptor, and the wise buyer will test coverage broadly before signing up their entire company. Conclusion Wireless access has indeed arrived — in modest but often useful ways. On some devices, certain features such as email, address lookup, directions, and download of plain text documents have been nearly perfected. (Contrary to recent product announcements from various PDA manufacturers, we do not expect “all-in-one” devices offering simultaneous phone and web connectivity to be usable any sooner than mid-2000 — if then.) Meanwhile, the WAP protocol/markup set has lost its hold on wireless. Wireless is quickly becoming a wide-open playing field, and HTML the standard for providing content. So this is a reasonable time to start moving into wireless, either as a user of devices, or a provider of content. It is also a time to spend money cautiously. Devices should be tested thoroughly before purchase. Content should be delivered in simple HTML. In this way a company can get value from wireless now, while waiting to see which (if any) standards emerge over the next year or two. ______________________ Note: this report is about the present usability of wireless services and devices in the U.S. For a long-range view of 3G, i-Mode, SMS — and other technologies currently erupting in Asia and Europe — please see our forthcoming report Potential for Asian and European Wireless Technologies in the U.S. Market, available at www.hastingsresearch.com in January 2002.

6

Table of Contents Executive Summary ..... 2 Table of Contents ..... 7 List of charts, illustrations, questionnaires ..... 8 Introduction ..... 9 Test Methodology, Subjects, Devices, and Raw Results ..... 10 Synopsis ..... 10 Methodology ..... 10 Subject Training ..... 11 Subjects ..... 11 Network World subjects ..... 11 Our Second Group of Subjects ..... 12 Our Researchers’ Previous Personal Experience with Wireless Devices ..... 12 Tests Conducted ..... 13 Devices Used ..... 13 Descriptions & Photos ..... 14 Cell Phone - Sprint (Qualcomm) Touchpoint 2100 Blackberry 957/857 Palm VIIx Handheld PC - HP Jornada Raw Test Results ..... 16 Feedback and Preliminary Conclusion ..... 18 Cellular Phones ..... 18 Blackberries ..... 19 Palms ..... 21 Our Own Study Group’s Opinions on Palm VIIxs with AvantGo ..... 21 Handheld PCs ..... 22 Preliminary Conclusion ..... 24 Analysis: What Works, What Doesn’t, and Why ..... 26 The Wireless Landscape: there have been changes ..... 26 Wireless Reception Quality ..... 27 What Doesn’t Work ..... 27 One Device For All Seasons ..... 27 What Does Work ..... 28 Functions ..... 28 Best Devices and Services ..... 28 Strategies For Implementing Wireless ..... 31 General Implementation Notes ..... 39 WML is a legacy markup language; wireless now supports HTML ..... 31 Bloat ..... 32 Strategies ..... 32 1. Equip and train your employees to use wireless devices with currently available services ..... 32 7

Choosing Devices - look at many before testing, test several before buying ..... 32 Cellular Phone Checklist ..... 33 Configuring Devices - less is usually more ..... 33 Testing Your Service Area ..... 34 The Need For Writing Your Own Device Documentation ..... 34 2. Rework your intranet to support your employees, suppliers, or B2B customers ..... 35 3. Rebuild your Internet access (Web site) to support the general public ..... 37 Conclusion ..... 38 Glossary of Wireless Terms ..... 39 Why and How This Report Was Written ..... 41 Background on Hastings Research ..... 41 Biographies of Authors ..... 42 Bibliography ..... 43 Appendix ..... 44 List of all devices, and why they were chosen or not for the study ..... 44 Questionnaires used ..... 45 Notes On the Questionnaires ..... 48 Feature Chart for the Sprint TP 2100/2200 Cell Phone ..... 49 Ordering Information ..... 53

List of charts and illustrations Charts Ease of Use by Device and Task ..... 16 Ease of Use by Task ..... 17 Suitability To Task ..... 28 List of devices tested, and which were used in the study ..... 44 Menu tree for functions on a Sprint TP2100/2200 ..... 49 Illustrations Sprint Qualcomm TouchPoint 2100 cellular phone, Blackberry 957, Palm VIIx, HP Jornada handheld PC ..... 15 Kyocera 6035 ..... 34

8

Introduction Much discussion has been published regarding wireless access over the last few years, and its implementation in both the U.S. and other countries. Much—perhaps most—of this writing has a distinctly negative slant, focusing on WAP deficiencies, why it won’t succeed, and how long it will be before it is ready for a mass audience—along with grand prophecies about the coming of i-Mode and 3G wireless services. As an R&D company, we are interested in the facts, and began the study with as few preconceptions as possible. However, the “D” part of our business is development: making software systems work. In that light, we were surprised just how far wireless has come in the U.S., and have written this report with “a glass half-full” view: what wireless access is capable of doing now and in the near future. Thus this report is not only about current wireless usability; it is also about what businesses, partners, and customers can get out of wireless in today’s marketplace, and how hard it is to implement.

9

Test Methodology, Subjects, Devices, and Raw Results Synopsis Hastings Research conducted the core of this wireless usability study at IDG’s Computerworld offices in San Mateo, California on June 14-15, 2001, and more extended studies with our own subjects in the interim, with a total of 25 subjects, 10 services, 6 portals, and 23 devices, including a variety of cellular phones, Blackberries, Palms and Handsprings, and handheld PCs (HP Jornada and Compaq IPAQ). The purpose of the study was to determine whether wireless devices and services are presently useful to businesses, for communicating with their employees, business partners, and the general public. (For disclosure of our limited relationship with IDG, please see page 41.)

Methodology We conducted this study on a qualitative basis rather than quantitative. From long experience, we prefer good qualitative studies to bad quantitative studies. And from equally long experience, we know that most quantitative studies are bad, unless you know exactly what questions to ask. With the flux of the wireless market, there was little likelihood of knowing exactly what questions to ask. So, in the language of researchers, this was an ethnographic study. From Ethnographic Research Strategy, by Sonny Kirkley, Computer Interaction Dept., U. of Indiana. “As personal and social factors enter into the decision to use or not use new technologies, research methodologies must penetrate the everyday use of technologies. We must gain entry into everyday life to observe and interpret technology as it is lived. Ethnographic methodologies are particularly pertinent to this goal. Using in-depth interviewing, focus group research, oral narratives, participant-observation, and journaling, ethnographers have contributed hugely to our understanding of on-the-ground experience—despite the common criticism of sampling limitations.” In short, qualitative studies mean: Watch carefully. Question closely. Think hard. And above all, prepare to be surprised. Because if there are no surprises, why are you doing a study anyway? (Quantitative studies are good for confirming what you already suspect, and so are important in the academic world of peer review. Qualitative studies are a lot better for learning something new, and thus are more useful in the business world.)

10

Subject Training Because of the number of devices being tested, manufacturers were understandably reluctant to supply us with 20-some of each. Superficially this posed a quandary — without letting subjects use the devices until they gained familiarity, how could we conduct a valid test? However, experience with previous studies has taught us that unless subjects are formally trained on-site, hours of practice will vary from 20+ a week to less than one hour a week. In short, without a massive, formal, on-site training program, we were bound to have test subjects of varying skill. Since the test was for usability of wireless access to the Web — plus email — we simply showed each subject the basic keys and navigation, which key or icon to get onto the Web, and they were on their own from that point. This allowed each subject to start from the same point in the learning curve.

Subjects Number The core test was made on June 14-15, 2001 with seventeen subjects in various capacities in IT departments. As a preliminary and follow up, we tested the devices with an additional 8 subjects recruited by Hastings Research. Network World subjects All subjects were San Francisco Bay area residents employed in information technology jobs, with 3 self-employed, and 14 in varying positions at companies sized from under 100 employees up to multinationals. All were men (the only woman responding was one of the 7 no-shows), ranging in age from early 20s to late 50s. Because these people work in the high-tech industry, some highly placed in public companies, most of them were unwilling to be quoted by name, and so we have simply referred to them by job description and number of employees in their company. Job descriptions/titles/number of employees: 1. 2. 3. 4. 5. 6. 7. 8. 9.

Senior Network Administrator (250+) Network Administrator (100+) CEO (less than 100) System Analyst (50,000+) IT consultant (self-employed) Senior Network Engineer (5,000+) Network Architect, (5,000+) CTO (less than 100) IT consultant (self-employed) 11

10. IT consultant (self-employed) 11. Senior Software Engineer (20,000+) 12. Network Administrator (less than 100) 13. IT Manager (100+) 14. Web Administrator (100+) 15. Network Manager (2,500+) 16. Network Manager (5,000+) 17. Network Engineer (20,000+) It was a good test group. Though it did not precisely mirror the typical enduser,— probably a road warrior, business partner, or consumer—there was a definite bonus to testing with IT personnel: while only five of the subjects were blazing fast with all four devices, all of the subjects were able to work all the devices. We could not have expected the same from many other groups. Our Second Group of Subjects Before and after the two days of testing at the IDG offices in San Mateo, we tested all the devices with 8 more subjects, ages ranging from 15 to 50, professions varying from student to COO. This was not because we doubted the IT professionals’ opinions—but because we felt that due to poor connectivity and service providers, the Palms and handheld PCs had not gotten a fair shake. Our Researchers Previous Personal Experience with Wireless Devices Nicholas Carroll - before the test, he disliked PDAs in general, and used cell phones reluctantly. Since the other Hastings’ assignees to the study were already familiar with devices such as Palms, Blackberries, and hand-held PCs, he grudgingly took the assignment of web-enabled cell phones. He now carries a web-enabled cell phone everywhere, and routinely uses it for address lookup, directions, and weather reports (but still uses a pocket notebook for scheduling). Sheldon Brahms - has used all manner of devices, including Jornadas, IPAQs, Palms, and Handsprings. He normally carries a wireless Handspring (also equipped for digital photography), two cell phones, and occasionally a Compaq IPAQ handheld computer. Mardee McGraw - is a regular user of cell phones, adept with pager messaging, and a Blackberry cultist. Deborah Rodgers - is familiar with cell phones, PDAs, and handheld PCs, with extensive experience designing maps for Palm OS devices.

12

Tests Conducted After looking at a variety of possibilities — including writing our own WML test pages — we settled on fairly simple criteria: 1) tests should represent real-life usage of wireless devices, and 2) they should be more or less achievable by the user group. In that light, we settled on the following tests, using commonly accessible services: 1. Search using a search engine, preferably Google. 2. Read an HTML web page and identify hyperlinks 3. Locate a phone number and address on Yahoo! 4. Read news 5. Send email 6. Use an online calculator to calculate a life insurance premium Devices Used Contrary to previous reports by other research companies, we found that wireless devices themselves were far, far from being ready for prime time. The problem was not just wireless services and sites; it was the devices themselves. (We are still puzzled by cell phones that are turned on with a “No” button.) Bad ergonomics, illogical controls, and poor displays were commonplace. The fact that users become passably proficient with cell phones merely proves that mobile telephony is a killer app—enough so that users will put up with poor interface in order to communicate by voice. It does not demonstrate that users are willing to surf the Web using devices with those same illogical controls and poor ergonomics. Just as many VCRs are used purely for playback of rented movies—recording being too complicated to be worth the owner’s time—we expect that most wireless devices currently available will be used for the absolute minimum of necessary tasks. We chose the following devices for the study, out of almost three dozen choices at our disposal. Cell Phone: Sprint TP 2100 Blackberry: 957/857 Palm OS: Palm VIIx Handheld PC: HP Jornada Network World supplied us with a wide selection of wireless-enabled devices—more than we actually wanted (partial list in the Appendix). After informally testing them with 9 people, we settled on those four devices on the following basis: 1. The Sprint TP 2100 was the simplest and most self-revealing of the phones. 2. The Blackberry is a group of its own, and we chose the larger 957 (and 857 “enterprise” version) over the older, smaller 950s. 3. Palms are no longer a group of their own, since the OS has been licensed to Handspring and other companies, but they are still the most familiar device in the PDA class, and the Palm VIIx is the top of the line. 13

4. The HP Jornada was chosen in lieu of the Compaq IPAQ for no particular reason; both had approximately the same ease (or difficulty) of use, but the IPAQ was much more difficult to set up for initial use. Descriptions & Photos Cell Phone - Sprint (Qualcomm) Touchpoint 2100 This was the simplest and most ergonomically pleasing of the cell phones we were offered. On the upper left is an “OK” button, which equals “Enter.” On the upper right is an “End” button, which is sort of like “Escape.” Below the End button is a “Clr” button, which fairly logically changes function from a “Backspace” button if the user is in the process of typing something, to a “Back” button if they are surfing. In the center is a depressed “joystick,” allowing the user to travel up, down, left or right. Once this was explained, subjects were able to surf immediately. Blackberry 957/857 Though Blackberries are widely known as email and PIM devices, the Blackberries using GoAmerica and Yahoo! wireless service also have web access capability. The Blackberry reseller network is so chaotic—service providers include Motient, Cingular, Mobitex, Datatec, GoAmerica, Yahoo!, and even AOL—that it was difficult to figure out who was providing what portion of the service on any given device. Regardless, we found that Blackberries had very good email reception in most Northern California and Boston-Washington metropolitan areas. (Web access was spottier.) A coverage map can be found at: http://www.blackberry.net/support/coverage/northamerica/index.shtml Palm VIIx Currently the top of the Palm line, the VIIx (which the manufacturer refers to as the “Palm 7”) is the most compact and elegant of current Palm-OS devices. Unlike other Palms or Handsprings, no clip-on modems or “sleds” are needed. Flip up the antenna, and you are in business. Handheld PC - HP Jornada The HP Jornada is a PDA size personal computer running the Windows CE operating system. Its chief competitors are the Compaq IPAQ and Sony CLIE. It must be slipped into a modem sled for wireless access (as pictured). (photos on next page)

14

Cell Phone - Sprint (Qualcomm) Touchpoint 2100

RIM Blackberry 857/957

Palm VIIx

HP Jornada

15

Raw Test Results As the saying goes in ethnography, “Scorecards are what you give subjects to make them feel comfortable, while you observe their true behavior and record their unrehearsed comments.” So we wrote down the numbers, and present them below. The higher the number, the greater the perceived usability. But the meat of the results is the next section: “Feedback and Preliminary Conclusion.” Ease of Use by Device and Task Question Cellular

Palm

Blackberry

Handheld

Average

Devices' Overall Average

Search with Google Read a simple web page Find a news story Read a news story Find an address with Yahoo! Send an email Use a web site calculator

3.5 3.1 4.4 3.7 4.0 4.2 n/a

3.8


1.2 1.3 1.5 1.0 1.0 4.5 2.5

1.9


4.2 4.4 3.5 3.1 2.5 4.8 2.0

3.5


4.2 4.1 3.8 3.5 3.3 3.3 2.8

3.6

16

Ease of Use Broken Down By Activity

Send email

Search Google

Find news

Read Web page

Read news

Find address

Use calculator

Average Score For This Activity

Cell phone Palm Blackberry Jornada

Score 4.2 4.5 4.8 3.3


3.5 1.2 4.2 4.2

3.3


4.4 1.5 3.5 3.8

3.3


3.1 1.3 4.4 4.1

3.2


3.7 1.0 3.1 3.5

2.8


4.0 1.0 2.5 3.3

2.7


1.0 2.5 2.0 2.8

2.1

17

4.2

Easiest

Most Difficult

Feedback and Preliminary Conclusion This section has a lot of user comments, many contradictory. It was not as simple as finding that users were either cheering or bashing wireless. Nevertheless some commonsense patterns emerged from the users. (Less patient readers may just want to skip to the end of this section, “Preliminary Conclusion.”) Cellular Phones One thing stood clear about web-enabled cell phones: they connected. While over half the users grumbled about “2400 baud downloading,” the connections were at least reliable. Past that, opinions varied widely. User commentary follows. User 3, a CEO, calmly went through the task list with no difficulty, but said, “I’m being kind with these 3s and 4s. Just because I can do it doesn’t mean I like it.” With no further comment, he headed for the next device. On the exit survey, he said he would not equip his company with cell phones, because they were “too hard to use.” User 4, the system analyst, had a similar take: “It’s easy to search Google once I learned to navigate the menu, but actually reading a web page is ugly at best. Downloading the news is surprisingly quick response time, but again the small screen limits the usefulness.” His opinion on equipping his company with them was, “No way. It’s too cumbersome to type, and the screen is terrible.” User 5, an IT consultant, said “It’s indifferent; better then most such devices, but a bit of a learning curve. Viewing web pages is typical for a cell phone. In this case the links are clearly defined, but reading content is hard. I find it easy to navigate to the news, but again reading is hard. Finding directions was easy on Yahoo!, much nicer than other WAP portals. Sending email was also easy, but the phone offers every choice except normal email.” Would he equip a client with these? “No. The screen was way too small, and it was too difficult to tap into normal email channels.” User 8, a CTO, was one of the few who didn’t even care for the TP2100 button/joystick configuration: “It’s highly non-intuitive. As far as navigating the menus, there are far too many choices. Add in the fact that it needs more screen space, and this isn’t worth the while.” Would you equip your company with these? “NO. It’s useless, too complex, and of too little value.” User 10, an IT consultant, voiced what other users merely grimaced at: “The main problem is the multiple punching of the number keys to bring up the right letter. It tries my patience. This would only be useful for very short messaging.” Would you equip your company with these? “Only if a client had the need for short paging-type messages. It’s worthless for the web.” User 13, a gung-ho IT manager, flew right through the tests, saying “Very nice, and

18

very easy one-hand operation. The controls are very intuitive. Finding directions was easy, and sending email was very fast.” Would you equip your company with these? “Yes, we need new phones, and these are very intuitive.” User 14, a Web Administrator, likewise felt the phone was easy to use, but was not happy with screen legibility, “Search was very easy since Google is a prominent option on the phone. It was easy enough to distinguish between content and links while reading a web page, but the small screen and constant scrolling made reading difficult. Sending email on this is more user-friendly than surfing.” Would you equip your company with these? “Yes. The functionality is key. Our staff primarily needs phone use and email access. They don’t need extensive web access when remote.” User 17, a Network Engineer, said “Figuring out the navigation was hard, but searching was easy after that. Reading the page was a little rough but not too bad, and it was easy to scroll. Getting to the news wasn’t hard, though trying to get back and forth between different news sites is not intuitive at all. I love the Yahoo! address lookup and directions, it’s the best I’ve seen on a wireless device. Sending email was intuitive and easy.” Would you equip your company with these? “Yes. They’re excellent for quick messages and light web surfing.” Readability on the TP2100s took such a beating on the first morning of testing that we began asking users to read the same web page on a Motorola Nextel L85 cell phone. Users immediately raised their readability ratings — which is not reflected in the raw data chart in the previous section. Had it been, the cell phones would have scored an astonishing 4.2 in usability. Blackberries When it came to email, Blackberries just worked. Sending and receiving were almost flawless. Web access was another matter. We had two Blackberries at the test, one using the GoAmerica portal, and another with Yahoo!. The users made their displeasure with Yahoo! clear. They were particularly displeased with the default search engine, Ask Jeeves. In fact some of the IT professionals had never heard of Ask Jeeves, and almost all said “Where’s Google?” User 1, a Sr. Network Administrator, used a GoAmerica Blackberry, and said “Finding a web site through Google was easy. Reading the page was easy; it was fast to scroll, and easy to search. Finding and reading news was easy on simple web sites; framed web pages were difficult to follow. Email was a no-brainer.” Would you equip your company with these? “Yes. Real-time two way communication with Exchange. Since our company is Microsoft based, this is a plus!” User 3 had little to say about the Blackberry’s web capability, but wrote “Great email!!” Would you equip your company with these? “Yes, the email app is great!” 19

User 4 used a Blackberry connecting to the Yahoo! portal, and wrote, “It was not readily apparent when a search had started. Clicking through to the page was easy and it was easy to read. Finding news, again it was not clear whether the search had started. Finding an address took way too long, and I gave up. The calculator was easy to launch, but I gave up on getting the results because it took too long.” On the other hand, he found the reliability of the email unsurprising, since he already supervises over 500 Blackberry 957s in use by his multinational company. He said that in major metropolitan markets “They work anywhere you can get cell phone coverage. We’ve had a bit of trouble on the I-5 [Interstate highway 5] corridor, but that’s about it.” He equipped his company with Blackberries “... for employees that travel a lot and are primarily interested in email. Companies like AvantGo are paving the way for the Blackberries’ ‘always on, always connected’ feature.” User 5 had a GoAmerica Blackberry, and wrote, “Search was easy, however it was a bit slow. I lost the connection several times. Likewise the web page was slow to load, and the formatting was a bit odd, but it was readable, and the links were easy to find. The Chronicle news was impossible to get to, but Yahoo! news was easy to find and read. I couldn’t get to Yahoo! address lookup, though. I gave up on the life insurance calculator because it took too long to come up. Email is definitely the killer app on this device.” Would you equip your company with these? “Yes. I’m a devoted Palm user, and I like the Blackberry’s ‘always on’ capability. We most likely wouldn’t use it for anything but email, unless we built custom apps for the device. It’s nice to see that it will work even when my cell phone doesn’t.” (His personal cell phone couldn’t connect from the IDG offices, but the Blackberry did.) User 13, an IT Manager, said “It was easy to enter a URL and proceed through links, though I didn’t like the scroll wheel, it was too slow. It was easy to read web pages, with nice clear fonts, and relatively fast hyperlinking. The news was easy to find, and the large screen kept the whole lead paragraph in view without scrolling. The email was great; it was very easy to read, and drill down through the menus. All in all, this is very intuitive.” Would you equip your company with these? “No, it’s one more item to carry.” User 17, a Network Engineer using the GoAmerica Blackberry, said: “It’s easy to read a web page, and distinguish the links, but I wish there were a scroll bar on the side so I could keep track of my position on the page. Also, there should be a “Find” function. It was easy to find and read the news, but again, I’d like a scroll bar. Also, it would be nice if the scrolling speed increased with the speed I spin the scroll wheel at [like a ballistic mouse]. I’d like to be able to “lock” the orange shift button, like Shift Lock, especially for typing numbers.” Would you equip your company with these? “No. The email was not intuitive.”

20

Palms The Palm VIIxs came into the test with a noose around their neck: palm.net Connectivity through palm.net appeared adequate, except at lunch hour. Users were able to begin downloads, but quickly became disgusted or even angered by the rigid channels they were forced into. “The World Wide Web” was simply not available. Apparently Palm had placed their strategic partnerships before the needs of the users. When it came time to test the email capability, one user sent Hastings Research an email saying “Palm.net must die.” (We found it in our inbox that evening.) We subsequently obtained another Palm VIIx and installed AvantGo software on it. Our comments follow the user comments on the palm.net-equipped devices. Regrettably, we did not have a chance to test this outside the San Francisco Bay area. As the to the palm.net-equipped devices used in the IDG portion of the study: User 2, a Network Administrator, commented that “Aside from being slow, it’s not clear where the browser is. Am I in the browser now?” He went on to rate every task except email with a “1” for usability, taking the time to circle the word “Impossible” at the “Very difficult / Impossible” end of the scale. Would you equip your company with these? “No. The browser didn’t work.” User 4 said, “I’m not familiar with the Ask Jeeves interface, and found it impossible to get to any web site whatsoever. The email was no problem at all.” User 6 said, “It took forever to get to Google, and once there, I couldn’t figure out how to enter text. The email was the only thing that worked.” Would you equip your company with these? “No. There’s no browser there.” It would be pointless to continue with user comments on Palm.net-equipped Palm VIIxs. In this case, the raw test results do tell the whole story. Users were able to send email with ease (in fact the Palms almost matched the Blackberries in ease of use on email) but they found WWW access a disaster. Users responses to the question “Would you equip your company with these?” came in two varieties: “No,” or “Never with Palm.net.” Our Own Study Group’s Opinions on Palm VIIxs with AvantGo: Palm.net performed so badly in the usability tests at Computerworld, and we had heard so many positive comments on Palms equipped with AvantGo, that we obtained another Palm VIIx, and installed AvantGo. The results were disappointing. All of our users concluded that AvantGo was head and shoulders above palm.net in both usability and access to the Web. However, AvantGo offering 1500 “channels” from “partners” is still a far cry from being able to access the entire web. 21

In fact the Kyocera 6035 with EudoraWeb — if clunkier than AvantGo — was somewhat more usable for surfing. Go to Google, search, and you could click through. Or type in a URL, hit Enter, and you’re there. It appears that wireless web access providers are paying far more attention to business development than users. They are missing the concept of the World Wide Web. The first wireless access provider to supply good interface, broad connectivity and true web access, may well dominate the market.

Handheld PCs Handheld PCs, both the Compaq IPAQs, and the HP Jornadas we used in the test, had such slow and unreliable connections that users stubbornly would downgrade them for waiting time, regardless of usability of the actual interface. Some users were so annoyed that they wrote “SLOW” in huge block letters across the questionnaire page. User 4 said, “I reached Google, but it was not readily apparent that the search had launched. Once I reached the web page, it was easy to read. Again when searching for news, I almost quit because it looked like it was done. I gave up on finding directions, it was far too slow. Email was easy. I found the calculator with no trouble, but gave up on getting the results, it was way too slow.” [This user accurately described a general wireless problem of sticky caching: you never really know whether you’re at a preinstalled or cached page, or viewing new content. The Jornadas reached the apex of this problem, presenting a Devil’s choice; should the user turn caching off, as much as possible, so they at least know they are getting fresh content? If so, they will be dealing with long download times.] User 5 commented that “It’s like browsing at home, but very, very slow, comparable to a 2400 baud modem. Pages were easy to read, but it’s not as nice as the AvantGo browser for the Palm. Links were similar to IE on the desktop. News was easy to find, but painfully slow. The IE browser really only works at 28.8 speeds and higher. I tried getting an address from Yahoo! and found it after three tries, but two submits came up with errors. The calculator was a very, very long wait, but at least usable.” The comments from User 10 were startling: “It was the quickest device of the bunch! It’s very similar to the real web. Reading a web page was as close to the real web as I’ve seen on these devices. In searching for news, apart from the speed, I was again highly impressed by the similarity to the real web. Sending email was harder; while it was easy to compose, it was hard to know if the email had been sent, and impossible to view it after sending.” He went on to rate the Jornada as the best of the devices, and the one he would be most likely to recommend to a client. It would be easy to classify this user as naïve, except that his comments on other de22

vices were consistent with the other network professionals, and his comments on Palms and Blackberries showed an exceptional knowledge of wireless service providers. Likewise, our researcher was sitting there confirming that he was getting fast downloads, sitting at the same table in the same building, where other users had been disgusted with the download speed. We call this startling because of the wild variability it shows in Jornada downloading. If the cell phones were slow, at least they were consistent throughout the greater San Francisco Bay area. The Jornada was not consistent even sitting in the same spot. (And the same was true of Compaq’s IPAQ, in our own testing.) At the same time as users stubbornly downgraded the Jornadas for slow downloads, they stubbornly would upgrade them for the novelty of the color screen, and familiar Windows Internet Explorer interface. User 12 had nothing but good to say about the Jornada, “It has a nice visual screen. Slow, but good graphics and presentation. Sending email was no problem. Overall, it has very easy web browsing.” He rated the Jornada with 4’s and 5’s on every task. Would you equip your company with these? “Yes. It gave good web access plus email.” Like user 10, number 12 was one of the few to get reasonably fast downloads over the two days of testing. This is one reason why qualitative testing is growing in popularity; without a trained observer taking notes, we would have been left with nothing but raw data. By the time User 13 was trying the Jornada, it was lunch hour in San Mateo, and downloading had slowed to a crawl: “Smaller text is needed to minimize scrolling. I don’t need graphics! I couldn’t download the news at all. I was able to reach Yahoo! address lookup, but it took way too long. The email was nice, but I gave up on the calculator, it was taking too long for the page to come up!” In fact so many of the users seemed to feel that they were holding a real personal computer, that we asked several of the ones who had experienced good connections if they would consider giving up their laptops for a handheld. Apparently their subconscious minds did not agree with their superficial enthusiasm, because the general reaction was of utter incomprehension, with users saying, “Could you repeat that?” or “What did you say?” So in raw usability statistics the Jornada scored 3.6, better by a tenth than the 3.5 scored by the Blackberries. Yet when we repeated “Would you equip your company with these, the users began to hedge, saying they would have to do further testing. The 3.6 score seemed to express “So close. Yet so far?”

23

Preliminary Conclusion The feedback varied for several reasons: different organizations have different needs; different users have different likes; and connectivity varied wildly by time of day for most of the devices. While some of the test subjects liked gadgets—a network engineer for a major university happily saying “I like devices!” as he blazed through the cell phone tests onehanded—most of the users were fairly hard-headed about what their organization or clients would buy. (The exception was the HP Jornadas, where several users were captivated by the color screen.) Though the cell phones enjoyed the highest ratings for usability, technical managers on the whole showed little or no interest in equipping their companies with them, due to the difficulty of use. Also, in most organizations field people come equipped with cell phones of their own. Cell phones have almost become ubiquitous. Field people are normally reimbursed for the part of their cell bill that was business related, but they are not usually reimbursed for the cost of their cell phone or provided with a certain type of cell phone from the IT department. It could just be that IT managers are used to field people having their own cell phones, so buying them a cell phone is something they don’t even consider. Even though a webenabled phone is different than a regular voice phone, it’s not exotic enough or high priced enough or the type of device you would normally use to access the web. Most field people have a cell phone for voice communication and an laptop for everything else. The reason users rejected cell phones as an email device—even though they scored the task as “easy”—was obvious: users didn’t like the slowness of using a numeric keypad to generate letters, or the tiny screen. The reason they were pleased with email on all the other devices—even ones with unreliable connections—was equally obvious if you were sitting there observing them: sending email doesn’t require much downloading. The users would compose, send, and move on to the next task. If they were satisfied with how that worked, then they were indifferent to whether the device took five seconds or five minutes to actually complete sending the email. Once beyond brief emails, and on to surfing the web, users became more demanding, and the prime demand was “the whole web.” In this, all the devices failed, for most of the users. A few users who were concentrating on straight-text web pages were pleased with the Blackberry using GoAmerica service. An even rarer few who got good connections with the HP Jornada, and who were will24

ing to work on a 150-pixel-wide screen, were happy with the whole experience. On the whole, users felt that the whole web had not arrived in their pocket. We agree.

===========================

In our view, manufacturers and service suppliers have seriously mis-marketed the current crop of web-enabled devices, where business is concerned. The average techie consumer may buy a web-enabled device based on “the web in your pocket!” hype. This hype is unlikely to make it past both IT and finance in a business. The more reliable Palm and Blackberry service providers, in particular, could be making a legitimate sales pitch for using these devices to access straight-text in-house documentation.

25

Analysis: What Works, What Doesnt, and Why The Wireless Landscape: there have been changes Because of the tiny screen sizes WAP and WML was designed to serve, as little as four lines, it started life based on the “card deck” analogy (possibly borrowed from Apple’s Hypercard stack of some years ago). Rather than having users scroll downwards on a page, they would shuffle downwards through a stack of “cards” (single-screen views). This was logical enough; when WML was conceived, few cellular phones had scrolling capability. Foreseeably enough, growing screen sizes also created an implementation nightmare: should a company design “cards” for 4-line screens, 5-line screens, 6-line screens? Or perhaps 8-line screens? Companies trying to deliver content to cell phones soon found themselves addressing dozens of different devices. To state the problem from a personal computer design perspective, consider the problems caused by trying to serve users with monitor widths varying from 640 pixels or less, to 1024 pixels or more. To this day, many web sites are fixed at 600 pixel width, testimony to the many organizations that gave up in frustration. Trying to anticipate and design for screen depth as well as width compounds this problem logarithmically. Four lines, five lines, six lines, seven lines, eight lines, etc.— multiplied by numerous screen widths—quickly becomes a nightmare. (At least at the design, financial, and management end; presumably to the WML coder, it is job security.) Happily, the problem may be over. Every device we tested, including all the cellular phones, had scrolling capability. Further, every device we tested was capable of rendering HTML. From a usability point of view, it is relatively unimportant whether the HTML was delivered directly to the device via HTTP and TCP/IP, or whether it was converted to WML and sent through WAP, or any other set of protocols. In the case of the phones, this was limited to straight text. Tables were rendered as sequential cells; while this was readable, it was hard to follow. It was also annoying. Many web sites have the same menu bar or sidebar on every page; this is usually beneficial in a web site, but with the limited real estate of a cellular phone, it means the user has to scroll though the menu all over again with each new web page they access. These observations were so contrary to the hype of WML, and the idea that smallscreen devices “must” use WML, that we went outside Hastings and contacted five colleagues involved in standards or interface design, all with the same question: “Now that new wireless devices all have scrolling capability, and all of them render HTML, why would you use WML instead of HTML?”

26

Two said that they couldn’t think of any good reason to use WML. The other three flatly said that in their opinion, WML is dead. Big Details Wireless Reception Quality Wireless reception in the San Francisco Bay area and the Boston-Washington corridor varied from good to appalling. Generally cell phones and Blackberries just worked (though both suffered occasional coverage gaps). Palms usually connected. Handheld PCs more often came under the category of, “let’s walk around to the other side of the building and see if we can connect there.” As a consequence, some of the testing at IDG’s San Mateo offices was conducted on the sidewalk out front rather than in the testing area. The tests were for usability, not nationwide connectivity, so our data is sketchy. Nevertheless, devices that do not connect well in Silicon Valley are questionable for national use. What Doesnt Work One Device For All Seasons Users repeatedly said they wanted “the one device that does it all”—if not a replacement for their phone and desktop computer, at least a passable facsimile for travel. In one of the most device-capable user’s comments about an HP Jornada, “How am I supposed to communicate with this thing, unless it has an earpiece and cell phone built in?” The Kyocera 6035 and Sprint TP 3000 theoretically achieved most of this, incorporating cell phone, personal organizer, and web access in a single device. Unfortunately both carried a fatal flaw: users could not read their personal information (or surf the Web) and talk on the phone at the same time. So it was impossible for them to find an address or meeting time, and tell someone on the phone, or discuss a Web site in realtime. This need has been recognized since the earliest days of computing. The NLS (oNLineSystem) for collaborative computing, showcased in 1968, was supposed to allow complete collaborative work to be done online. Finding email much slower than speech — and far too slow when brainstorming — NLS users promptly started communicating with their hands on the keyboard and a telephone cradled on their shoulder. (More than 40 years later, users of Ray Ozzie’s P2P “Groove” program do the same thing.) For speed and clarity, nothing equals visual information delivery plus voice. Now, as then, the “one device” doesn’t exist. Organizations that need this capacity right now should simply settle on a good voice device and a good visual device, and expect employees to use both at once. (Speakerphones and earpieces help, of course. While 1960s telephones were designed to be cradled on the shoulder, cell phones are not.) 27

There are some cell phones now coming to market — Nokia comes to mind — that claim to do it all. Likewise Handspring has announced the Treo, and RIM has announced plans to incorporate cell phones in their Blackberries. We have not had an opportunity to test them. But contrary to such product announcements, we do not expect “all-in-one” devices offering simultaneous phone and web connectivity to be usable any sooner than mid-2002 — if then. The reason? Connectivity. Cell phones are reliable because they have had 20 years of network development. Blackberries have reliable email because they transmit over ancient, well-tested pager networks. The dubious connectivity of all other devices in this test suggest that the chances of having a reliable all-in-one device any sooner than mid-2002 are slim to none. What Does Work Functions All tested devices gave access to simple HTML 3.2 pages. Among other things, this makes wireless devices potentially useful right now for field support, along with other possibilities listed in the chart below. In overt usability, the handheld PCs would be the best choice for web access plus email. But connectivity remained a problem. The unshaded cells show the features that had solid connectivity, with darker cells showing progressively worse connections. By the time you reach the right side of the table, usability begins to look a lot less important than “will it work at all?”

Suitability To Task T as k Addres s lookup F inding directions W eather reports Very s hort email B rief email L ong email S tock prices R eading documents W as ting time

Cell phones adequate adequate adequate adequate inadequate inadequate adequate poor poor

B lackberries with GoAmerica good good good very good very good inadequate good adequate adequate

P alm w/AvantGo or E udoraW eb good good good good good inadequate good adequate adequate

H andheld P Cs good good good good good inadequate good adequate good

Best Devices and Services Most of the Network World user group as well as our own user group showed a consistent preference for certain devices and services. Unlike our own researchers and user group, the Network World group did not have access to all the devices and services during the test. However many of them had previous experience with these other devices, and expressed corroborating opinions on their own initiative. Best whole-Web access While palm.net scored miserably with users, Palm OS devices using EudoraWeb or AvantGo received much better ratings from our own Hastings user group. If GoAmerica 28

improves Blackberry web connectivity, it will be a toss-up for users who like the Palm interface, using Graffiti and icons, and those who prefer typing on a keyboard, with drop-down menus. Best cellular phone usability in a Web access service Yahoo! and Google. Yahoo! in particular has made an extensive and relatively successful effort to provide a full range of their services on cell phones. Case in point: one of our researchers used his cell phone to get directions to the BART (subway) station south of San Francisco. Not knowing the area, he found the directions confusing. However one of the options was “Call BART.” He selected this, the phone dialed BART, and an operator gave him directions to the subway station. Tiny screens make “drilling down” a nuisance at best. In this case Yahoo! navigation intelligently minimized the nuisance, by allowing the user to fast-forward to the task a cell phone does best: voice communication. Best email usability and service Blackberries. In our view Blackberries have real potential, despite their relatively high price. They are the best available device for email, which to many remains the primary value of the Internet. The proof of good design is often that every user has the same complaint. (With poorly designed equipment, complaints are all over the map.) In testing the Blackberry 857/ 957, the consistent usability problem was in sending email: users could not figure how to get from the email’s “Subject” line down to the message body area. The answer was too simple; just turn the scroll wheel down one click, and the cursor drops into the message-writing area. Users laughed outright when they were shown how to do this. The only other problem regularly mentioned with the Blackberry itself was the scroll wheel being the sole up-and-down control; several users felt there should be a “Page Down” button added to the device, as well as a scroll bar to give some hint of how long the page was. We concur; there is a big difference between reviewing or reading a brief email, and wading into a web page of unknown length, where one might want to simply drop to the very bottom and search for a “Contact Us” link. That Blackberries “just work” is not surprising. They originate in Canada, a country that has been providing simple, usable, does-one-thing-very-well software for some years. Historically, this has not played well in the U.S. market, since American consumers and businesses are notorious for buying the biggest feature set they can get for their money—even when the extra features are useless or worse. It is possible that this American weakness may have reached its limit with handheld devices, which simply can’t handle bloat as a laptop can. Where do handheld PCs fit in? When it comes to wireless, we are not sure that they fit in anywhere. The screens are far too narrow to display most web sites — like Palms and Blackberries, they are largely restricted to text-only web sites. Despite that fact, 29

they try to display graphics that simply won’t fit in a 150+-pixel-wide screen. In connectivity they were consistently the worst devices we tested. The fact that Windows CE will run on them does not mean they give true Windows usability. As best we can tell, most handheld PC owners use them as “PIM-plus” devices; a personal information manager with a little bit more capability than a Palm or Blackberry, and the ability to access Microsoft files in a pinch. In short, a cell phone is a voice device trying to expand its processing capacity, memory, and visual display enough to provide good Internet access. A handheld PC is a laptop desperately trying to stuff itself into a box with inadequate processing capacity, memory, and display area. This leaves Palm-OS devices and Blackberries as the leading candidates for Internet access right now. While the Palms originated as PIM devices, and Blackberries as email devices, they were both designed from the ground up for the visual presentation of information in a hand-sized device.

30

Strategies For Implementing Wireless General Implementation Notes WML is a legacy markup language; wireless now supports HTML. Every device we tested was capable of reading HTML pages, down to the lowliest cell phone. This is not to say that they thrived on 3.2. Remember HTML before tables existed? This is what cell phones love. They are quite happy reading dry, virtually unformatted HTML pages. The same goes for Blackberries and Palm OS devices. They are not happy with tables, much less anything more complicated, but handle simple HTML nicely. This is not to say that the devices are literally rendering HTML onto their screens; many are still using WML and associated transmission protocols. But the playing field has changed in the last couple of years. Users are demanding the whole World Wide Web, and service providers are starting to give it to them. Whether those HTML pages are being converted to WML and transmitted in WBXML, or simply transmitted as is, via http and TCP/IP, the key point is that content providers no longer have to incur the expense of converting to WML. (Note that i-Mode ignores WML altogether, and uses cHTML—compact HTML, a subset of HTML.) Of course the need for simple HTML would not apply to handheld PCs such as the HP Jornada or Compaq IPAQ? On the contrary, it does. The screen real estate on handhelds is so limited that when delivering anything but the simplest HTML—floating to browser width—a horizontal scroll bar appears at the bottom of the screen. (Horizontal scroll bars—forcing users to scroll to the right in order to view an entire screen—are a usability offense so grave that they have practically disappeared from the Web. It is startling to see this problem re-emerge with the handheld PCs.) In our opinion, WML is a legacy protocol, and a short-lived one at that. The handheld PCs never needed it, the other devices don’t need it any longer—and with steadily increasing screen resolutions and sizes, there is absolutely no reason to believe this trend will reverse. So while Europe may be stuck with WML for some years, the U.S. should probably just skip over it. There are benefits to being the “second mover.”

31

Bloat Feature bloat is rampant in wireless devices. It is a massive impediment to usability. Unlike a PC or a Mac, the user can’t simply delete unwanted programs, or drag unwanted icons into some obscure window, never to be seen again. With wireless devices the user is usually stuck with what they get, and they get plenty. Manufacturers are stuffing in every possible function, in an effort to make their devices appropriate for every group from 15-year-olds to programmers and from salesmen to CEOs. While one expects bloat on a handheld running Windows CE, the number of useless features that can be packed into a cell phone is astonishing, and to the user, mindnumbing. The Sprint/Qualcomm TP 2100 and 2200 cell phones came equipped with six species of email service, including Sprint “Short Mail,” Juno mail, Yahoo mail—everything except real email. Once over their dismay about that, the survey subjects were pleased with the option of pre-written messages such as “Will return your call tomorrow” or “Meeting postponed until ________ ,” but were baffled by message options such as “Is that your final answer?” and “Wassup?” The size of the feature set was mindboggling, and test subjects continually said exactly that. To make the point loud and clear, we have included a partial feature tree for the Sprint TP2100/2200s in the Appendix. The entire tree would probably be four feet wide and six feet high, if printed out as a single chart.

Strategies Wireless access - or wireless information delivery — comes in three increasingly expensive categories. 1. Equip and train your employees to use wireless devices with currently available services Contrary to rumor, wireless access has indeed arrived—in modest but often useful ways. On some devices, certain features have been nearly perfected. Blackberry devices have spread because they provide excellent service with that traditional killer app, email. Even cell phones have been successfully enabled; address lookup and driving directions are extremely useful to road warriors. However, none of the dozens of devices we tested were ready-to-go off the shelf. Regardless of whether you configure devices in-house (with attendant labor costs) or try to order them properly configured from suppliers (with attendant haggling costs), you should figure on doubling - or tripling—the purchase price of the devices, to arrive at a real cost estimate. Choosing Devices - look at many before testing, test several before buying We tested 11 cell phones for the study. Not a single one was fully satisfactory (reasons why we excluded most of them are listed in the Appendix). 32

However, if the Sprint TP 2100s had possessed the larger screen, good backlighting, and the high quality font hinting of the Motorola Nextel L85, they would have been thoroughly satisfactory. So for a follow-up test, we asked Sprint PCS to provide a pair of TP2200s, which have longer screens. Problem solved? Not quite. Qualcomm apparently built the 2200 on the same chassis as the 2100 (at least the overall size of the phone is similar). Though the keypad was overtly identical, they had reduced its size to make room for the bigger screen. In the process, they shrunk the buttons, eliminating the fine ergonomics and easy one-handing that so many users had praised. Nor did the 2200’s screen approach the Motorola Nextel’s. So when it comes to cell phones, testing 30 different phones is probably a realistic minimum. Figure on a week of initial research by two or three varied employees. This would involve taking our checklist below, visiting multiple computer stores and cell phone suppliers, and simply testing the ergonomics, keypad logic, and readability.

Cellular Phone Checklist After reviewing all of the questionnaires, and doing further testing, we came up with this checklist of questions you should have on hand when looking at phones for eventual purchase: Devices: Are the controls intuitive for both voice and WWW use? Can they be used by people of varying hand size and fingernail length? Can they be one-handed? Is the screen legible under both good and poor lighting? (Even good fonts are not legible with poor backlighting.) Do they have a minimum of 6 lines of text visible? (8 to 10 lines opens whole new areas of usability.) Is the screen legible to older users? Is the speaker volume loud enough? Are rapid-chargers available for road warriors? Service Checklist: Is there adequate service to employee’s offices, travel routes, and homes? (Check both voice and web access coverage areas.)

Configuring Devices - less is usually more Of course, “de-configuring” devices is not all that easy. Larger organizations may be able to negotiate a simple feature set with the manufacturer or service provider. Smaller organizations are limited to the option of intensive shopping.

33

Testing Your Service Area Once you have settled on some potential devices, have your employees test wide, test far. Wireless service providers themselves have numerous employees roaming the countryside looking for coverage gaps. Unfortunately, while they often try to fill those gaps, they rarely bother to tell customers about them. Coverage is caveat emptor, and the wise buyer will test coverage broadly before signing up their entire company. Though we tested all of these device classes throughout Northern California, and some of them along some of the Boston-Washington corridor, we do not even begin to express an opinion on most of the service providers. All the services had some coverage gaps, even in major metropolitan areas. (We do list services that were notably good or bad everywhere.) The Need For Writing Your Own Device Documentation No one really expects good documentation to come with modern mass-production computing devices (other than perhaps Apples). Just as the typical PC comes with a hodgepodge of manuals from the hardware manufacturer and Microsoft—neither one accurately describing the model of computer you have just bought—wireless devices come with a hodge-podge of information from the manufacturer and the wireless service provider. (If you’re lucky, that is; many devices came with only the manufacturer’s generic manual, and no web access information at all.) It is tempting to think that shoddy documentation “comes with the territory.” However, whether you are equipping employees with wireless devices for Web access, or asking them to log in to your wireless-friendly intranet, they are quite likely sitting at home, or on the road. They cannot just call IT to come down and show them where the coffee cup holder disappeared to. Case in point: the Kyocera 6035 was one of the best devices we tested, in usability, reception, and ergonomics. Aside from good design, Kyocera thought about users enough to include three extra styluses with the phone. Clearly, a company that thinks ahead. Was this reflected in their user manual? Not at all. For example, the manual gave the following procedure for a “hard reset” (complete memory wipe): 1. Turn phone off 2. Open flip 3. Remove battery cover 4. While holding down the “Backlight” button, press the “Reset” button 5. A screen will appear asking if you wish to “Hard Reset.” 6. If so, tap the “Yes” icon.

34

The actual procedure is: 1. Turn phone off 2. Open flip 3. Remove battery cover, and find the tiny recessed “Reset” button to the left of the battery. 4. Randomly push the reset button with the stylus while also randomly pushing the Backlight button. Repeat in different patterns a dozen times or so, sometimes pushing one button first, other times the other, and sometimes both at once. 5. Turn on phone and check to see if “Recent Calls” memory has been wiped. 6. If not, repeat #4 as necessary. Further reflecting the need for realistic documentation, the Kyocera suffered from “perspiring ear” crashes, not mentioned in the manual. In the hot California summer, perspiration from the user’s ear tends to coat the screen. Naturally enough, the user then tries to clean the screen. However, this causes the device to freeze completely. Solution? Remove the battery for a few seconds, replace it, and reactivate the phone. (Removing batteries becomes something of a regular ritual as cell phones get more complicated; it is the quick-and-dirty fix for numerous freezes on many brands of cell phones, not just Kyoceras.) 2. Rework your intranet to support your employees, suppliers, or B2B customers The survey included one last exit question, “Shifting gears completely: would you be likely to rework your intranet or web site to serve these devices?” Seven users responded “Yes,” seven responded “No,” and three could not decide. Their comments are included in this section (intranets) and the next section, on reworking public web sites for wireless. Again, comments were more useful than numbers could have been. All of the test subjects were seasoned professionals, and the “cool factor” was rarely part of their decision. Rather they had specific reasons grounded in their user base or the nature of their organization. User 3 said, “They would be great for our field team, but the only one I’d want to support would be the Blackberry.” User 4, although he had already deployed 500 Blackberries for email use, said, “No, implementing wireless web access would be too cumbersome for a company of our size, with more than 50,000 employees.” User 5 said “Yes,” in part because he liked fishing: “Simply to free myself from a laptop or wired connection. If I can troubleshoot, monitor, or talk with employees while fishing on a lake, I would be in IT bliss.” Asked which he would be most likely to support, he said either the Palms or Blackberries, and “maybe” the wireless phones. His take on the handheld PC was “No way. Too slow!”

35

User 6 said no, feeling it was too early to move: “We want as many people as possible to be able to view our web site. However, I’m not sure it would be worth the investment to rework it, since the access technology seems to be immature.” User 8 said, “No. My employees and clients would never actually use it with these results, so it would be a waste. In the future I might consider the Palms or pocket PCs if they came with an appropriate interface.” User 15: “Yes. But mostly for our drivers and dispatchers. I would be willing to support the Blackberry.” User 17 said, “Yes. This would be wonderful to have in the field for troubleshooting, to access documentation, and for email. I’d support either the Palm or handheld PCs as soon as they can provide reliable connectivity, and had an RJ 45 for ethernet connection.” In short, IT professionals were saying that the devices and service would have to fit a known need. Where IT ambitions are limited and specific, this is often surprisingly achievable— particularly for the fortunate few organizations that have simple intranets. Of the devices we tested, all the better ones, including cellular phones, were quite capable of reading clean, simple HTML files. Serving content in WML (wireless markup language) was simply not necessary. In some cases this was because the device was capable of reading HTML directly. In other cases, intermediary services transmitting to the wireless devices were doing the conversion from HTML to WML, or whatever form of screen rendering their endusers require. Thus the actual decision to implement wireless information delivery is an internal one, involving heavy organizational and markets number-crunching, and goes beyond the scope of this report. Unless your organization has already bought into devices that only work with WML — think “HTML.”. At that point, supporting your employees outside the office is no longer a matter of converting your intranet content to WML, but simply to clean HTML—the traditional “Text only” alternative (now making a slow reappearance under the pressure of Federal 508 disability access regulations). While wireless device manufacturers have tried to eke the most out of the limited memory in devices costing as little as $100, the rational organization delivery to wireless devices will keep their pages as small and clean as possible. This means no javascript, no Cold Fusion, no Flash—and ideally putting every executable other than links on the server side.

36

3. Rebuild your Internet access (Web site) to support the general public The short answer is in two words: probably not. Yahoo! and Google have done a credible job of making their sites usable to even the most display-challenged of wireless devices, cellular phones. However, these are both broad-market plays, that will live or die by popular appeal. None of the test subjects worked with such broad consumer markets. ROI was more of an issue to this group than any hypothetical “first mover” advantage. Likewise there is the development curve to consider. Web sites tend to be rebuilt when they won’t scale anymore, and the design stage of a new architecture is a logical place to introduce wireless support. User 1 said, “Since our web site is in the works, developing wireless support would be a plus in reaching customers on all types of devices.” User 10 was also trying to reach the public, but the company web site was already in place. He said, “Not today. Until the speed and functionality can be improved, the need for these devices is limited to mobile sales force or onsite customer support.” User 12 said, “Not yet. Not until there’s a greater wireless need.” User 13 was surprised how all the devices would access any text-based web site, and said “Yes. I would ‘text enable’ our site for all to browse easily.” There are exceptions, of course. As endusers, the survey subjects made it clear to us that they would regularly use any tolerable wireless device, even a cellular phone, to access certain types of information, such as short emails, news, financial reports, travel information such as addresses, directions, and weather; and certain opportunistic types of information such as restaurant reviews. Basically, these are all hard niches: information which is specific, time-sensitive, and deliverable in simple text. As an ROI decision, rebuilding a web site—or creating an alternate web site for wireless users—is an extension of number 2. If your market is financials, and your customers are day traders, the answer flows from number 2: maybe. You have to judge your market carefully—and do the numbers.

37

Conclusion A small range of wireless functions has already become useful in the U.S. and Canada, especially for those in the field. It will be at least six months to two years before the U.S. is moving in a clear direction. Yet, regardless of who hypes what new standard, two trends stand independent of any one group’s control. First, HTML will be around for awhile. In some part this is through suppliers’ fear of waplocking. Many content providers are unwilling to write in WML or any other semiproprietary format. Yet there is a bigger, market-force reason why HTML is still around. In order to gain new customers, wireless service providers are increasingly being forced to provide wider and wider access to the Internet. Since the vast majority of the world’s Internet sites have no intention of providing content in WML — if they even know what WML is — the service providers who want to capture market share are forced to transcode HTML to whatever markup language or screen rendering protocol they are delivering to their end-users. And second: big, fold-out high-resolution color screens are not coming to handheld devices anytime soon. LCD production costs guarantee that. Wireless in the U.S. is going to be plain text for awhile—regardless of the excitement Americans feel over seeing i-Mode in action in Japan. In short, “text” will be king for awhile yet. This is not all bad. While text cannot substitute for a wiring schematic, for most situations it remains the finest information compression technology ever invented. So spend intelligently on devices. But go cautiously into delivering content by any hyped new standards. And in either case, whether as user or content provider: stick with key functions and core content.

— END —

38

Glossary of Wireless Terms 3G is an ITU specification for the third generation (analog cellular was the first generation, digital PCS the second) of mobile communications technology. 3G promises increased bandwidth, up to 384 Kbps when a device is stationery or moving at pedestrian speed, 128 Kbps in a car, and 2 Mbps in fixed applications, plus the ability to work over wireless air interfaces such as GSM, TDMA, and CDMA. CDMA - Code-Division Multiple Access. It is a digital cellular technology that uses spread-spectrum techniques. Unlike competing systems,(such as GSM) that use timedivision multiplexing (TDM), CDMA does not assign a specific frequency to each user. Instead, every channel uses the full available spectrum. Individual conversations are encoded with a pseudo-random digital sequence. CDMA is a military technology and was first used during World War II by the English allies to foil German attempts at jamming transmissions. The allies decided to transmit over several frequencies, instead of one, making it difficult for the Germans to pick up the complete signal. Qualcomm Inc. created communications chips for CDMA technology. This information was classified during the war. Once the information became public, Qualcomm claimed patents on the technology and was first to use it for commercial purposes. cellular - Refers to communications systems, especially the Advance Mobile Phone Service (AMPS), that divide a geographic region into sections, called cells. The purpose of this division is to make the most use out of a limited number of transmission frequencies. Each connection, or conversation, requires its own dedicated frequency, and the total number of available frequencies is about 1,000. To support more than 1,000 simultaneous conversations, cellular systems allocate a set number of frequencies for each cell. Two cells can use the same frequency for different conversations so long as the cells are not adjacent to each other. For digital communications, several competing cellular systems exist, including GSM and CDMA. Compact HTML, or cHTML—the markup language used by i-Mode, a subset of HTML. i-Mode - a packet-based service originally developed by NTT DoCoMo in Japan. NonWAP compliant with support for cHTML (compact HTML). Capable of color and multimedia data delivery. packet - a piece of a message transmitted over a packet-switching network. See under packet switching. One of the key features of a packet is that it contains the destination address in addition to the data. In IP networks, packets are often called datagrams. packet switching - refers to protocols in which messages are divided into packets before they are sent. Each packet is then transmitted individually and can even follow different routes to its destination. Once all the packets forming a message arrive at the 39

destination, they are recompiled into the original message. Most modern Wide Area Network (WAN) protocols, including TCP/IP, X.25, and Frame Relay, are based on packet-switching technologies. In contrast, normal telephone service is based on a circuit-switching technology, in which a dedicated line is allocated for transmission between two parties. Circuit-switching is ideal when data must be transmitted quickly and must arrive in the same order in which it’s sent. This is the case with most realtime data, such as live audio and video. Packet switching is more efficient and robust for data that can withstand some delays in transmission, such as e-mail messages and Web pages. SDMA - space division multiple access : A variation of TDMA and CDMA that potentially will be used in high-bandwidth, third-generation wireless products. SMS - Short Message Service (SMS) is the transmission of short text messages to and from a mobile phone, fax machine and/or IP address. Messages must be no longer than 160 alpha-numeric characters and contain no images or graphics. Originally this referred to the broadly used European service, which was driven to popularity by the high European rates for airtime minutes. The phrase is now being knocked off by various U.S. proprietary messaging systems. TDMA - Short for Time Division Multiple Access, a technology for delivering digital wireless service using time-division multiplexing (TDM). TDMA is used by the GSM digital cellular system WAP - Wireless Access Protocol, one of many systems for transmitting content to wireless devices. Created for cell phones by phone.com, and promulgated by the WAP Forum, backed primarily by Motorola, Nokia, and Ericsson. Often thought synonymous with WML, WAP actually includes a variety of standards and protocols for markup, compression, and transmission (such as WTP, WTP, WCMP, WTLS) with the intent of serving content to WAP-enabled devices in the same manner that HTML, http, and TCP/IP serve web content from servers to HTML browsers. waplocking - locking users into a proprietary wireless protocol. The term originated in criticism of European manufacturers’ practices, but has been applied to other standards like i-Mode. WML - wireless markup language. Derived from the W3C’s XML data exchange format, WML is more compact yet far more complex than HTML. Like XML—or CSS, or any data formatting language that allows the same templates to be used for multiple documents—it potentially needs less bandwidth, since formatting commands are transmitted just once, and then the content is filled in. But unlike HTML, WML can be hard to write—and it needs good documentation so future employees can learn the setup.

40

Why and How This Report Was Written IDG’s Network World approached us in Spring of this year and asked if we would be interested in carrying out a broad study of wireless device usability, if they provided use of the devices and their facilities. Since the only well-known wireless study of any size was released almost a year ago by the Nielsen-Norman Group, tested only two devices (both cellular phones), and was conducted in England, we agreed. We conducted the bulk of the testing at the Computerworld offices in San Mateo on June 14-15, 2001, and more extended testing with our own subjects in the interim, with a total of 25 subjects, 10 wireless services, 6 portals, and 23 devices, including a variety of cellular phones, Blackberries, Palms, Handsprings, and handheld PCs (HP Jornada and Compaq IPAQ). IDG provided loaner devices for testing (supplied by various manufacturers), and the use of their San Mateo offices on June 14-15, 2001. The study was otherwise designed and executed by Hastings Research at our expense. Any conclusions IDG may have drawn from the testing are independent of Hastings Research, and are not incorporated in this report.

Background on Hastings Research Hastings Research (http://www.hastingsresearch.com) carries out basic computer research, and also develops high quality interactive software systems, interfaces, and web sites, both commercial and non-commercial. All of our commercial web site builds are making a profit for our clients. We have been in business since 1994, and our principals have 20+ years apiece of computer, interaction, and business experience. Based in Northern California, we have successfully delivered jobs to clients in all parts of the U.S. and Canada, ranging from startups to Fortune 500 companies, commercial to non-profit, hard goods to services, B2C to B2B. Our work ranges from the exotic (research and development for Doug Engelbart’s Open Hyperdocument System) to the very commercial (building web sites that sell banking services, books, or inflatable boats). We are not funded by or financially linked in any way to any product or service in this report. 41

Biographies of Authors Nicholas Carroll is a web analyst and information architect. His primary work— aside from analyzing patterns of web use—is setting strategic goals and leading projects for both .com and .org websites in the U.S. and Canada, including traffic sourcing plans, UI, navigation, usability, and site search. His background is in programming, project management, marketing, and media. He has had numerous business books published, as well as dozens of columns on high technology and business, in newspapers ranging from The Chicago Tribune to The Toronto Globe and Mail. His marketing papers Mousetraps on the Web were widely praised for the application of traditional direct marketing strategies to the new technologies of the Internet. Sheldon Brahms is a widely-known interaction designer skilled in human-computer interaction, interface, and graphics. With one foot in research and the other in business and production, he has been able to do ground-breaking work and pioneer many of the areas that are now taken for granted, such as Multimedia/New Media, computer based training, electronic publishing & pre-press, computer-based animation & editing, and learning technologies, both independently and while working in the Advanced Technology Group at Apple Computer. He has been a practicing psychologist, and was founder of Nighthawk Productions. Mardee McGraw is a financial and markets researcher with a background in smallto medium-capitalization stock markets ranging from high-tech to mining, as well as having been operations manager of a high-volume botanicals distributorship. She tracks technology, both software and hardware, and trends in ecommerce business models, including B2B, B2C, and supply chains. She is a graduate of U.C. Berkeley. Deborah Rodgers is a cartographer by training, and a highly experienced designer of atlases, mobile maps, and interactive software. She designed the map interfaces for the Digital Travel Guide by Lonely Planet publishers, including the CitySync feature.

42

Bibliography WAP Forum http://www.wapforum.org Start Coding: An Introduction to WML, by David Sims, 7/28/2000 O’Reilly Wireless DevCenter http://www.oreillynet.com/pub/a/312 The Invisible Substrate of Information Science, Marcia J. Bates, UCLA Dept. of Information Sciences http://www.gseis.ucla.edu/faculty/bates/substrate.html Retrieval Structure Manipulations, Muskingum College http://muskingum.edu/~cal/database/Encoding6.html WAP Usability: Déjá Vu: 1994 All Over Again, Marc Ramsey and Jakob Nielsen Yahoo! Cataloging the Web, Anne Callery http://www.library.ucsb.edu/untangle/callery.html Ericsson WAP Developer’s Zone http://www.ericsson.com/WAP Ethnographic Research Strategy, by Sonny Kirkley Computer Interaction Dept., U. of Indiana. Augment Differentiators, John Rothermel, ESL/TRW http://www.bootstrap.org/augment-133247.htm Wireless Developer Network http://www.wirelessdevnet.com/channels/wap/training/wml.html Why WAP May Never Get off the Ground. [Binary Critic.] By Ted Lewis, DaimlerChrysler Research & Technology, September 07, 2000 WML Takes XML on the Road http://www.networkcomputing.com/1206/1206ws1.html WAP: Already a Thing of the Past?, Intranet Journal http://www.intranetjournal.com/articles/200009/un_09_20_00a.html

43

Appendix List of all devices, and why they chosen or not for the study Genus B rand B lackberry Blackberry

Cell P hones

D evice N ame

Model

W eb Acces s

950

Blackberry

957

B lackberry

957

Blackberry

857

B lackberry

857

Motorola

T imeport

Motorola

L85s

Qualcomm

T P 2100

Cell phone Kyocera w/ P alm OS

OCP

LG

P alm-OS D evices

H andheld P Cs

It was ergonomically friendly, with big buttons and a good feel, was not a clamshell, and had the simplest controls, everything accessible through OK, No, CLR , and the inverted " joystick" pad. Features were already being tested with the cellulars and the Palms, plus we calculated the " cool factor" would prove distracting.

6035

S ame as above

Handspring

Visor Edge

Handspring

Go America

Palm

Visor Platinum? Vx

P alm

VIIx

P almnet

IPAQ

HP

Jornada

Palmnet

3649

Poor web connectivity, adequate navigation. Better web connectivity, v. poor navigation Nice ergonomics.

T he controls were workable, but there were Very readable screen, good fonting. two more buttons to learn than the T P2100s had; this would have slowed down the testing unacceptably. Was used only for readability testing. S ervice was discontinued shortly after we got it, and restored after the T P2100s had been chosen Judged the MP3 capability would prove a distraction from the tests -- which proved true when people wanted to play with it at the demo table later. Also, pretesting proved the buttons were too small for some users. Came without charger. Good color screen; disappointed we didn't have a working one.

T P 3000

Compaq

Had web access. T he clamshell design made it inherently more complicated than the T P2100s.

Uproar

S CP 5000

Had web access. No web access

Yahoo

P8767

S anyo

Comments

No web access Go America

Nokia

S amsung

R eas on us ed or not us ed Obsolete in U.S . market

Users' unfamiliarity with Handsprings could have slowed down the test. By contrast, any Handspring user knows that a Palm is almost identical to a Handspring. ditto S upplanted by the newer Palms with built-in modem and antenna S tate of the market device for wireless Palm devices, with built-in antenna and modem. Very cumbersome assembly and setup Far easier to get up and running than the IPAQs

44

Far from the best screen, but the other factors outweighed this.

Very good ergonomics, acoustics, and navigation. S tronger reception than any other cellular tested. Workable, though not as good as the Kyocera.

Questionnaires used DEMOGRAPHICS Which of the following best describes your role in your organization ? (Check one) ____ Chief Information Officer ____ Sr. Vice President / Vice President MIS/IT/IS/DP ____ General Corporate Management (CEO, COO, CFO, President, VP Operations) ____ MIS/IT/IS/DP Manager or Director ____ LAN, WAN, or Network Manager or Director ____ Telecommunications Manager or Director ____ Internet/Intranet/E-commerce Manager or Director ____ Network Architect, Designer, or Database Administrator ____ Network Operator, Technician ____ Trainer, Help Desk, or Tech Support ____ Software programmer ____ IT consultant ____ Other

How many people does your company employ, including all remote offices, departments, and divisions? ____ Over 20,000 ____ 10,000 - 20,000 ____ 5,000 - 9,999 ____ 2,500 - 4,999 ____ 1,000 - 2,499 ____ 500 - 999 ____ 250 - 499 ____ 100 - 249 ____ 99 or less

DEVICE TASKS 1. Search Google (www.google.com) for “hastings research”. (No quote marks needed.) How difficult was this task? (Circle one.) Very Difficult Difficult Easy 1 2 3 4 5 Comment: 45

2. Click through to Hastings Research and read the page. How difficult was it to read the page? Very Difficult Difficult Easy 1 2 3 4 5 Could you distinguish links from plain text?

Yes

No

Comment:

3. Go to www.sfgate.com or www.yahoo.com, find the news, and read the first part of a top news story. How easy was it to find the news? Very Difficult Difficult Easy 1 2 3 4 5 How difficult was it to read the page? Very Difficult Difficult Easy 1 2 3 4 5 Comment:

4. Go to www.yahoo.com and find the address for ______________ in _____________, California. How difficult was it to locate the address? Very Difficult Difficult Easy 1 2 3 4 5 Comment:

5. Compose and send a short email. How difficult was it to send email? Very Difficult Difficult Easy 1 2 3 4 5 Comment: 46

6. Go to www.insurancecalculators.com/needscalc.htm and calculate your life insurance premium. How difficult was it to send perform the calculation? Very Difficult Difficult Easy 1 2 3 4 5 Comment:

EXIT SURVEY Which of these would you buy for your employees as wireless access devices? 1. Web-enabled phone Why or why not?

2. Blackberries Why or why not?

3. Palms Why or why not?

4. Handheld PCs Why or why not?

Now - changing gears completely: would you be likely to rework your company intranet or web site to support these devices? (Circle one.) Yes No

If so, which ones would you be likely to support?

47

Notes On the Questionnaires Question 1: we asked users to search for Hastings Research because it comes up high on any search engine for an exact name search. We were not interested in testing the users’ drill-down tolerance on the first test - merely whether they could identify the hyperlinks among the other text, and judge which one they wanted, from the limited information some of the devices provide. (E.g., cell phones don’t give a full Google listing, or even a full site name.) Question 2: here we used the www.hastingsresearch.com site because it is a middle ground. It uses tables — notably the left sidebar-style menu — but is otherwise clean HTML 3.2 with minimal graphics. Question 3, finding news: We gave users the option of www.sfgate.com (The San Francisco Chronicle) and www.yahoo.com because some devices were simply unable to access the Chronicle site. Question 4: This shows blanks for name of business and town. This is because. wireless devices have exceptionally “sticky” memories, presumably in an effort to work around slow download speeds. Thus we had to give each user a new business name for testing address lookup, or the device would simply pull the information out of cache. Clearing memory between each user took too long, within the time constraints of the test. (Hilariously, some of these highly technical users would pop the batteries out of the devices the moment they sat down at a new test station, saying “Going to clear the cache.” At our amused expressions, they would say, “Not that easy, eh?”)

48

Feature Chart for the Sprint TP 2100/2200 Cell Phone The task: to send an email. However, #2, the logical selection, is devoted to voice mail and other things. In this phone you select #7, “Wireless Web” to reach email. On to the next page...

Sprint TP2200 Internet Enabled Cell Phone Main Menus 1) Call Logs

Sub Menus 1) 2) 3) 4)

Outgoing Log Incoming Log Missed Call Log Erase Logs

2) Messages

1) Voice Mail 2) Page/Text 3) Browser Msgs 1) Find Name 2) Find Speed Dial 3) Add New Entry 4) Add New Folder 5) Edit Folder 6) Sort Name 7) Services 8) My Phone #

3) Phone Book

4) Roaming

1) Set Mode 2) Call Guard

5) Scheduler

SMTWTFS

6) Settings

7) Wireless Web

8) Security

1) 2) 3) 4) 5) 6) 7) 8) 9)

1) Exit

My Wireless Web Messaging Games Web Portals Entertainment Finance Sports News&Weather @Sprint PCS

9) Calculator

10) Games

1) 2) 3) 4) 5) 6) 7)

1) Blackjack 2) Football 3) Dice 1) 2) 3) 4) 5)

11) Voice Services

12) Keyguard

1) Headline 2) Sounds 3) Display 4) Alarm Mode 5) Data/Fax (IM) 5) Car Use More...

(Hold Clear Key To Disable)

49

Record Memo Memo List Record Ringers Commands Voice Dial List

= equals + add - subtract * multiply / divide . decimal pt. Exit

In a hierarchical filesystem, you are now at root:/Wireless Web/ The logical choice for email would once again be “Messaging.” However, the actual path goes through #1, “My Wireless Web.

Sprint TP2200 Internet Enabled Cell Phone 7a) Wireless Web Menus

Sub Menus

1) My Wireless Web

1) Messaging 2) Acct Usage 3) Finance 4) Weather 5) Sports 6) Travel 7) Games 8) Business Tools 9) Bookmarks More...

2) Messaging

3) Games

4) Web Portals

5) Entertainment

1) AOL 2) Yahoo! 3) MSN Mobile 4) Google 5) Go2Online

1) Sign In 2) Mail & Addresses 3) AOL IM 4) Finance 5) Local 6) News & Sports 7) Shopping 8) Travel 9) Entertainment Search & Tools

6) Finance

7) Sports

8) News & Weather

9) @Sprint PCS

More...

1) Acct Usage 2) Acct Invoice 3) Short 4) Sprint PCS Mail 5) My Music 6) Tools 7) Help

50

You are now at root:/Wireless Web/My Wireless Web/, and the email choices are revealed in #1, “Messaging,” including AOL IM, Short Mail, Sprint PCS Mail, etc. Drilling down into messaging on the next page....

Sprint TP2200 Internet Enabled Cell Phone 7b) Wireless Web Menus 1) Messaging

Sub Menus

1) AOL IM 2) Short Mail 3) Sprint PCS Mail 4) Wireless Chat 5) Yahoo! Mail 6) Juno Mobile

2) Acct Usage

3) Finance

4) Weather

1) The Weather Channel 2) Go2Weather

1) FIdelity 2) Ameritrade 3) Schwab 4) Bloomberg 5) CBS Market Watch 6) Countrywide 7) E*Trade 8) Forbes.com 9) HooverÕ s Online More...

5) Sports

1) ESPN.com 2) Book4golf 3) Go2scores 4) PGA.com 5) Yahoo! Fantasy Sports 6) Buzztime Sports Trivia

6) Travel

7) Games

8) Business Tools

9) Bookmarks

More...

1) Acct Usage 2) Acct Invoice 3) Short Mail 4) Sprint PCS Mail 5) My Music 6) Tools 7) Help

1) 2) 3) 4) 5) 6) 7) 8) 9)

10 Best Countrywide OAG Mobile Bloomberg CNET CNN Forbes.com HooverÕ s Online NY Times More...

1) 2) 3) 4) 5) 6) 7) 8) 9)

Sabre/Travelocity Expedia.com MapQuest 10 Best Fodors.com MyPNA (Traffic) GetThere.com Qtaxi-limo OAG Mobile More...

51

You are now at root:/Wireless Web/My Wireless Web/Messaging/. We chose to use #2, “Short Mail,” for the tests. While you can simply chose #1, “Type Msg,” this is timeconsuming on a cell phone, and we have shown the time-saving menu offering “Pre Set Messages.”

Sprint TP2200 Internet Enabled Cell Phone 7c) Wireless Web Menus 1) Messaging

2) Acct Usage

1) AOL IM 2) Short Mail 3) Sprint PCS Mail 4) Wireless Chat 5) Yahoo! Mail 6) Juno Mobile

Sub Menus 1) Type Msg 2) Pre-Set Msg 3) Icons

1) To: xxx-xxx-xxxx 2) Msg (Edit) 3) Add Pre-Set Msg 4) Add Icon 5) Cancel 1) IÕm running late 2) CanÕt talk now 3) I received your message 4) Where are you 5) Can you talk 6) DonÕt forget 7) Call me at 8) Tag youÕre it 9) Is that you more...

3) Finance

4) Weather

5) Sports

1) YouÕve got mail 2) I love you 3) Wassup 4) LetÕs party 5) Hugs & kisses 6) IÕll be there 7) Meet me at 8) IÕm going to 9) You go girl more...

6) Travel

7) Games

8) Business Tools

1) 2) 3) 4) 5)

9) Bookmarks

More...

Lock This Msg Block This Sender Return to Inbox Reply Sprint PCS Home

This concludes a brief tour of a tiny part of the TP2100/2200 feature set. If you look again at the first menu for this phone, and project outwards through all the submenus, you will have some sense of the impossibility of mastering the entire feature set.

###