The bloodhound project: automating discovery of web usability issues using the InfoScentπ simulator

ABSTRACT

According to usability experts, the top user issue for Web sites is difficult navigation. We have been developing auto-mated usability tools for several years, and here we describe a prototype service called InfoScent™ Bloodhound Simula-tor, a push-button navigation analysis system, which auto-matically analyzes the information cues on a Web site to produce a usability report. We further build upon previous algorithms to create a method called Information Scent Absorption Rate, which measures the navigability of a site by computing the probability of users reaching the desired destinations on the site. Lastly, we present a user study involving 244 subjects over 1385 user sessions that show how Bloodhound correlates with real users surfing for in-formation on four Web sites. The hope is that, by using a simulation of user surfing behavior, we can reduce the need for human labor during usability testing, thus dramatically lower testing costs, and ultimately improving user experience. The Bloodhound Project is unique in that we apply a concrete HCI theory directly to a real-world prob-lem. The lack of empirically validated HCI theoretical model has plagued the development of our field, and this is a step toward that direction.

References

1. Anderson, J. R., Pirolli, P. L. (1984) Spread of Activation. Journal of Experimental Psychology: Learning, Memory and Cognition, 10: pp. 791--798.
2. Bates, M.J. (1989). The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5): pp. 407--431.
3. Belkin, N. J. (1980). Anomalous states of knowledge as the basis for information retrieval. Canadian Journal of Information Science, 5, May 1980 (pp. 133--143).
4. Blackmon, M. H., Polson, P.G., Kitajima, M., & Lewis, C. (2002). Cognitive walkthrough for the web. CHI 2002 Conference on Human Factors in Computing Systems. ACM Press, pp. 463--470.
5. Chakrabarti, S., B. Dom, P. Raghavan, S. Rajagopalan, D. Gibson, and J. Kleinberg. (1998) Automatic resource compilation by analyzing hyperlink structure and associated text. In Proc. Of the 7th International World Wide Web Conference (WWW7) (pp. 65--74), Brisbane, Australia.
6. Chi, E., P. Pirolli, and J. Pitkow. (2000) The scent of a site: A system for analyzing and predicting information scent, usage, and usability of a Web site. In Proceedings of the Conference on Human Factors in Computing Systems, CHI 2000, pp.161--168. Hague, Netherlands.
7. Chi, E.H., Pirolli, P., Chen, K., and Pitkow, J. (2001). Using information scent to model user information needs and actions on the Web. Proc. of the ACM Conference on Human Factors in Computing Systems, CHI 2001 (pp. 490--497), Seattle, WA.
8. Devore, Jay. (1987) Probability & Statistics for Engineering and the Sciences. 2nd Ed. Brooks/Cole Publishing: Monteray, CA.
9. Heer, J., Chi, E.H. Separating the Swarm: Categorization Methods for User Access Sessions on the Web. In Proc. of ACM CHI 2002 Conference on Human Factors in Computing Systems, pp. 243--250. ACM Press, April 2002. Minneapolis, MN.
10. Huberman, B.A., Pirolli, P., Pitkow, J.E., and Lukose, R.M. (1998). Strong regularities in World Wide Web surfing. Science, April 3, 1998, vol. 280, num. 5360 (pp. 95--97).
11. Ivory, M. Y., R. R. Sinha, and M. A. Hearst, (2001) "Empirically Validated Web Page Design Metrics," Proc. Human Factors in Computing Systems (CHI2001), pp. 53--60.
12. Kleinberg, J. M. (1998) Authoritative sources in a hyperlinked environment. In Proc. Of the 9th Annual ACM-SIAM Symposium on Discrete Algorithms, (pp. 668--677), San Francisco, CA.
13. Koman, Richard. (1998) The Scent of Information. WebReview, May, 1998. http://www.webreview.com/1998/05_15/strategists/05_15_98_1.shtml
14. Nielsen, J. http://www.useit.com/alertbox/20010819.html, August 19, 2001.
15. Pirolli, P. and S.K. Card. (1999) Information foraging. Psychological Review. 106: p. 643--675.
16. Pirolli, Peter. (2000) A Web Site User Model Should at Least Model Something About Users. Internetwork-ing 3:1, Mar 2000. ITG Publications. http://www.internettg.org/newsletter/mar00/critique_max.html
17. Reeder, R. W., Pirolli, P. and Card, S. K. (2001). WebEyeMapper and WebLogger: Tools for Analyzing Eye Tracking Data Collected in Web-use Studies. Pro-ceedings of CHI 2001, Seattle.
18. Silva, I., B. Ribeiro-Neto, P. Calado, E. Moura, N. Ziviani. (2000) Link-based and Content-based Evidential Information in a Belief Network Model. In Proc. of the 21st ACM SIGIR Conference on Research and Development in Information Retrieval (pp.96--103). Athens, Greece.
19. Spool, J.M., Scanlon, T., Snyder, C., and Schroeder, W. (1998). Measuring Website usability. Proceedings of the Conference on Human Factors in Computing Systems, CHI '98 (pp. 390), Los Angeles, CA.
20. Tilt, Chris. (2000) Response to Pirolli's Critique of MAX model. Internetworking 3:1, Mar 2000. ITG Publications. http://www.internettg.org/newsletter/mar00/response_critique_max.html
21. Turtle, H., Croft, W. (1991) Evaluation of an inference network-based retrieval model. ACM Transactions on Information Systems, 9(3):187--222.
22. WebCriteria SiteProfile. (2002) http://www.webcriteria.com

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

      • Language resources

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

    • HCI theory, concepts and models

    • Interaction paradigms

      • Web-based interaction

  • Interaction design

    • Interaction design theory, concepts and paradigms

• Information systems

  • Information retrieval

    • Evaluation of retrieval results

• Software and its engineering

  • Software notations and tools

    • Development frameworks and environments

      • Integrated and visual development environments