Search strategies in multimodal image retrieval

ABSTRACT

This paper reports on a study on search strategies in multimodal image retrieval. We analyzed the queries and search tactics employed by image journalism professionals and non-professionals in a user test. Transaction log data show that searchers are able to combine up to four query modes into a query. Most queries combined at least two of the modes (text, color, sketch, quality, and category). Task type was shown to affect the choice of which modes to employ. Known item and data search tasks led to queries combining text, color and category modes. Visually cued tasks resulted in searches combining several content-based and textual modes. Conceptual tasks led to a large number of queries by text and category only. User background also significantly affected the types of queries constructed. Professionals used the color mode more whereas non-professionals drew more sketches. Non-professionals were more likely to switch query modes whereas professionals edited the content of their queries. We described the search processes with Markov models and maximal repeating patterns. Common patterns and probable transitions dealt with querying, inspecting result images and saving them into the workspace or iterating queries of the same type. The results indicate a need to support multimodal image query formulation.

REFERENCES

• 1: Christel, M. 2007. Examining User Interactions with Video Retrieval Systems. Proceedings of SPIE Vol. 6506 Multimedia Content Access: Algorithms and Systems.
• 2: Christel, M. and Conescu, R. 2006. Mining Novice User Activity with TRECVID Interactive Retrieval Tasks. Proceedings of CIVR 2006, 21--30.
• 3: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008 [doi>10.1145/1348246.1348248]
• 4: Enser, P. G. B. 1993. Query analysis in a visual information retrieval context. Journal of Document and Text Management. 1, 1, 25--52.
• 5: Fidel, R. 1997. Image Retrieval task: Implications for the Design and Evaluation of Image Databases. The New Review of Hypermedia and Multimedia. 3, 181--199.
• 6: Goodrum, A. A., Bejune, M. M. and Siochi, A. C. 2003. A State Transition Analysis of Image Search Patterns on the Web. Proceedings of CIVR 2003, LNCS 2728, 281--290.
• 7: Hung, T-Y. 2005. Search Moves and Tactics for Image Retrieval in the Field of Journalism: A Pilot Study. Journal of Educational Media and Library Sciences. 42, 3, 329--346.
• 8: Jaimes, A. 2006. Human Factors in Automatic Image Retrieval System Design and Evaluation. Proceedings of IS&T/SPIE Internet Imaging 2006.
• 9: Jörgensen, C. 1999. Access to Pictorial Material: A Review of Current Research and Future Prospects. Computers and the Humanities. 33, 4, 293--318.
• 10: Corinne Jörgensen , Peter Jörgensen, Image querying by image professionals: Research Articles, Journal of the American Society for Information Science and Technology, v.56 n.12, p.1346-1359, October 2005 [doi>10.1002/asi.v56:12]
• 11: Marjo Markkula , Eero Sormunen, End-User Searching Challenges Indexing Practices inthe Digital Newspaper Photo Archive, Information Retrieval, v.1 n.4, p.259-285, January 2000 [doi>10.1023/A:1009995816485]
• 12: Marziliano, P., Dufaux, F., Winkler, S. and Ebrahimi, T. 2002. A no-reference perceptual blur metric. Proceedings of International Conference on Image Processing, vol. 3, 57--60.
• 13: Sharon McDonald , John Tait, Search strategies in content-based image retrieval, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860452]
• 14: Antonio C. Siochi , Roger W. Ehrich, Computer analysis of user interfaces based on repetition in transcripts of user sessions, ACM Transactions on Information Systems (TOIS), v.9 n.4, p.309-335, Oct. 1991 [doi>10.1145/119311.119312]
• 15: Stina Westman , Pirkko Oittinen, Image retrieval by end-users and intermediaries in a journalistic work context, Proceedings of the 1st international conference on Information interaction in context, October 18-20, 2006, Copenhagen, Denmark [doi>10.1145/1164820.1164843]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

General Terms:
Experimentation, Human Factors

Keywords:
content-based, image queries, query modes, search patterns, text-based, transaction logging, user test