Towards task-based personal information management evaluations

ABSTRACT

Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and refind information. A feature of PIM research is that many systems have been designed to assist users manage and refind information, but very few have been evaluated. This has been noted by several scholars and explained by the difficulties involved in performing PIM evaluations. The difficulties include that people re-find information from within unique personal collections; researchers know little about the tasks that cause people to re-find information; and numerous privacy issues concerning personal information. In this paper we aim to facilitate PIM evaluations by addressing each of these difficulties. In the first part, we present a diary study of information re-finding tasks. The study examines the kind of tasks that require users to refind information and produces a taxonomy of refinding tasks for email messages and web pages. In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using two different methods of task creation.

REFERENCES

• 1: R. Boardman, Improving tool support for personal information management, Ph.D. thesis, Imperial College London, 2004.
• 2: P. Borlund, The iir evaluation model: A framework for evaluation of interactive information retrieval systems, Information Research 8 (2003), no. 3, paper no. 152.
• 3: Katriina Byström , Kalervo Järvelin, Task complexity affects information seeking and use, Information Processing and Management: an International Journal, v.31 n.2, p.191-213, March/April 1995 [doi>10.1016/0306-4573(94)00041-Z]
• 4: R. G. Capra and M. A. Perez-Quinones, Re-finding found things: An exploratory study of how users re-find information, Tech. report, Virginia Tech, 2003.
• 5: Robert G. Capra, III , Manuel A. Pérez-Quiñones, Using Web Search Engines to Find and Refind Information, Computer, v.38 n.10, p.36-42, October 2005 [doi>10.1109/MC.2005.355]
• 6: R. G. Capra and M. A. Perez-Quinones, Factors and evaluation of refinding behaviors., SIGIR 2006 Workshop on Personal Information Management, August 10-11, 2006, Seattle, Washington, 2006.
• 7: Edward Cutrell , Daniel Robbins , Susan Dumais , Raman Sarin, Fast, flexible filtering with phlat, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124812]
• 8: Mary Czerwinski , Eric Horvitz , Susan Wilhite, A diary study of task switching and interruptions, Proceedings of the SIGCHI conference on Human factors in computing systems, p.175-182, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985715]
• 9: Susan Dumais , Edward Cutrell , JJ Cadiz , Gavin Jancke , Raman Sarin , Daniel C. Robbins, Stuff I've seen: a system for personal information retrieval and re-use, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860451]
• 10: D. Elsweiler and I. Ruthven, Memory and email re-finding, In preparation for ACM TOIS CFP special issue on Keeping, Re-finding, and Sharing Personal Information (2007).
• 11: D. Elsweiler, I. Ruthven, and C. Jones, Dealing with fragmented recollection of context in information management, Context-Based Information Retrieval (CIR-05) Workshop in CONTEXT-05, 2005.
• 12: David Elsweiler , Ian Ruthven , Christopher Jones, Towards memory supporting personal information management tools, Journal of the American Society for Information Science and Technology, v.58 n.7, p.924-946, May 2007 [doi>10.1002/asi.v58:7]
• 13: D. Harman, What we have learned, and not learned, from trec, Proc. ECIR 2000, 2000.
• 14: Peter Ingwersen, Information retrieval interaction, Taylor Graham Publishing, London, UK, 1992
• 15: D. Kelly, B. Bederson, M. Czerwinski, J. Gemmell, W. Pratt, and M. Skeels (eds.), Pim workshop report: Measurement and design, 2005.
• 16: D. Kelly and J. Teevan, (to appear in) personal information management, ch. Understanding what works: Evaluating personal information management tools, Seattle: University of Washington Press., 2007.
• 17: B. Kwasnik, How a personal document's intended use or purpose affects its classification in an office, ACM SIGIR Forum, v.23 n.SI, p.207-210, June 1989
• 18: M. W. Lansdale, The psychology of personal information management., Appl Ergon 19 (1988), no. 1, 55--66.
• 19: Leysia Palen , Marilyn Salzman, Voice-mail diary studies for naturalistic data capture under mobile conditions, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587092]
• 20: M. Ringel, E. Cutrell, S. Dumais, and E. Horvitz, Milestones in time: The value of landmarks in retrieving information from personal stores., Proc. INTERACT 2003, 2003.
• 21: George Robertson , Mary Czerwinski , Kevin Larson , Daniel C. Robbins , David Thiel , Maarten van Dantzich, Data mountain: using spatial memory for document management, Proceedings of the 11th annual ACM symposium on User interface software and technology, p.153-162, November 01-04, 1998, San Francisco, California, United States [doi>10.1145/288392.288596]
• 22: K. Rodden, How do people organise their photographs, BCS IRSG 21st Annual Colloquium on Information Retrieval Research, Glasgow, Scotland, 1999.
• 23: D. C. Rubin and A. E. Wenzel, One hundred years of forgetting: A quantitative description of retention, Psychological Bulletin 103 (1996), 734--760.
• 24: Abigail J. Sellen , Richard H.R. Harper, The Myth of the Paperless Office, MIT Press, Cambridge, MA, 2003
• 25: Pertti Vakkari, Task complexity, problem structure and information actions: integrating studies on information seeking and retrieval, Information Processing and Management: an International Journal, v.35 n.6, p.819-837, 35 [doi>10.1016/S0306-4573(99)00028-X]
• 26: P. Vakkari, A theory of task-based information retrieval, Journal of Documentation 57 (2001), no. 1, 44--60.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation, Human Factors, Management, Measurement

Keywords:
personal information management, user evaluation