Retrieval of context-aware applications on mobile devices: how to evaluate?

ABSTRACT

This paper discusses the issue of evaluation of context-aware retrieval applications. We begin by describing MoBe, a specific architecture that allows automatic download and execution of context-aware applications on mobile devices. In MoBe, the most relevant applications are selected by matching context and application descriptors. Since several alternatives for descriptors implementation exist, it is important to compare their effectiveness. To this aim, we develop a TREC-like benchmark, in which the collection is made up by a set of application descriptors and the topics are made up by context descriptors. We then use the benchmark to evaluate the effectiveness of different descriptors components, and of structured and unstructured (i.e., free text) data, finding results that are somehow useful for future MoBe development. We also discuss the issue of the evaluation methodology for highly interactive and novel applications like context-aware retrieval systems, and MoBe in particular.

REFERENCES

• 1: Gregory D. Abowd , Anind K. Dey , Peter J. Brown , Nigel Davies , Mark Smith , Pete Steggles, Towards a Better Understanding of Context and Context-Awareness, Proceedings of the 1st international symposium on Handheld and Ubiquitous Computing, p.304-307, September 27-29, 1999, Karlsruhe, Germany
• 2: Matthias Baldauf , Schahram Dustdar , Florian Rosenberg, A survey on context-aware systems, International Journal of Ad Hoc and Ubiquitous Computing, v.2 n.4, p.263-277, June 2007 [doi>10.1504/IJAHUC.2007.014070]
• 3: M. Bazire and P. Brezillon. Understanding context before using it. In CONTEXT 2005, pages 29--40, 2005.
• 4: Andrei Broder, Computational advertising, Proceedings of the nineteenth annual ACM-SIAM symposium on Discrete algorithms, p.992-992, January 20-22, 2008, San Francisco, California
• 5: P. J. Brown, J. D. Bovey, and X. Chen. Context-aware applications: from the laboratory to the marketplace. IEEE Personal Communications, 4(5):58--64, 1997.
• 6: P. J. Brown , G. J. F. Jones, Context-aware Retrieval: Exploring a New Environment for Information Retrieval and Information Filtering, Personal and Ubiquitous Computing, v.5 n.4, p.253-263, December 2001 [doi>10.1007/s007790170004]
• 7: Guanling Chen , David Kotz, A Survey of Context-Aware Mobile Computing Research, Dartmouth College, Hanover, NH, 2000
• 8: P. Coppola, V. Della Mea, L. Di Gaspero, S. Mizzaro, I. Scagnetto, A. Selva, L. Vassena, and P. Zandegiacomo Riziò. Information filtering and retrieving of context-aware applications within the MoBe framework. In International Workshop on Context-Based Information Retrieval (CIR-05), CEUR Workshop Proceedings, 2005.
• 9: A. K. Dey. Context-aware computing: The CyberDesk project. In AAAI 1998 Spring Symposium on Intelligent Environments, pages 51--54, Palo Alto, 1998. AAAI Press.
• 10: D. J. Harper and D. G. Hendry. Evaluation light. In Proceedings of the second MIRA workshop, pages 53--56, 1997. Technical Report TR-1997-2, Department of Computing Science, University of Glasgow, Glasgow. http://www.dcs.gla.ac.uk/mira/workshops/padua_procs/.
• 11: G. Hripcsak and A. S. Rothschild. Agreement, the f-measure, and reliability in information retrieval. Journal of American Medical Informatics Association, 12(3):296--298, 2005.
• 12: G. J. F. Jones and P. J. Brown. Context-aware retrieval for ubiquitous computing environments. In Mobile HCI Workshop on Mobile and Ubiquitous Information Access, pages 227--243, 2003.
• 13: M. Korkea-aho. Context-aware applications survey. Technical Report Internetworking Seminar (Tik-110.551), Helsinki University of Technology, 2000.
• 14: Bradley James Rhodes , Pattie Maes, Just-in-time information retrieval, Massachusetts Institute of Technology, 2000
• 15: B. Schilit and M. Theimer. Disseminating active map information to mobile hosts. IEEE Network, 8(5):22--32, 1994.
• 16: Timothy Sohn , Kevin A. Li , William G. Griswold , James D. Hollan, A diary study of mobile information needs, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357125]
• 17: Ellen M. Voorhees, The Philosophy of Information Retrieval Evaluation, Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, p.355-370, September 03-04, 2001

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Keywords:
benchmark, context-aware retrieval, experimental evaluation, mobile search