Automated gathering of Web information: An in-depth examination of agents interacting with search engines

ABSTRACT

The Web has become a worldwide repository of information which individuals, companies, and organizations utilize to solve or address various information problems. Many of these Web users utilize automated agents to gather this information for them. Some assume that this approach represents a more sophisticated method of searching. However, there is little research investigating how Web agents search for online information. In this research, we first provide a classification for information agent using stages of information gathering, gathering approaches, and agent architecture. We then examine an implementation of one of the resulting classifications in detail, investigating how agents search for information on Web search engines, including the session, query, term, duration and frequency of interactions. For this temporal study, we analyzed three data sets of queries and page views from agents interacting with the Excite and AltaVista search engines from 1997 to 2002, examining approximately 900,000 queries submitted by over 3,000 agents. Findings include: (1) agent sessions are extremely interactive, with sometimes hundreds of interactions per second (2) agent queries are comparable to human searchers, with little use of query operators, (3) Web agents are searching for a relatively limited variety of information, wherein only 18% of the terms used are unique, and (4) the duration of agent-Web search engine interaction typically spans several hours. We discuss the implications for Web information agents and search engines.

REFERENCES

• 1: Arvind Arasu , Junghoo Cho , Hector Garcia-Molina , Andreas Paepcke , Sriram Raghavan, Searching the Web, ACM Transactions on Internet Technology (TOIT), v.1 n.1, p.2-43, Aug. 2001 [doi>10.1145/383034.383035]
• 2: Yariv Aridor , David Carmel , Yoelle S. Maarek , Aya Soffer , Ronny Lempel, Knowledge encapsulation for focused search from pervasive devices, ACM Transactions on Information Systems (TOIS), v.20 n.1, p.25-46, January 2002 [doi>10.1145/503104.503106]
• 3: Greg Barish , Craig A. Knoblock , Yi-Shin Chen , Steven Minton , Andrew Philpot , Cyrus Shahabi, The TheaterLoc Virtual Application, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.980-987, July 30-August 03, 2000
• 4: Michael W. Berry , Murray Browne, Understanding search engines: mathematical modeling and text retrieval, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1999
• 5: Kurt D. Bollacker , Steve Lawrence , C. Lee Giles, CiteSeer: an autonous Web agent for automatic retrieval and identification of interesting publications, Proceedings of the second international conference on Autonomous agents, p.116-123, May 10-13, 1998, Minneapolis, Minnesota, United States [doi>10.1145/280765.280786]
• 6: Brandman, O., Cho, J., Garcia-Molina, H. and Shivakumar, N. 2000. Crawler-friendly Web servers. In Proceedings of the Workshop on Performance and Architecture of Web Servers (PAWS). Santa Clara, California.
• 7: Andrei Z. Broder , Marc Najork , Janet L. Wiener, Efficient URL caching for world wide web crawling, Proceedings of the 12th international conference on World Wide Web, May 20-24, 2003, Budapest, Hungary [doi>10.1145/775152.775247]
• 8: Brody, R. 2000. Illusions of plenty: The role of search engines in the structure and suppression of knowledge. In Proceedings of the IEEE International Symposium on Technology and Society. Rome, Italy, 157--161.
• 9: Budzik, J. and Hammond, K. 1999. Watson: Anticipating and Contextualizing Information Needs. In Proceedings of the 60nd Annual Meeting of the American Society for Information Science. 727--740.
• 10: Cappelli, P. 2001. Making the most of online recruiting. Harvard Bus. Rev. 79,3, 139--146.
• 11: Carmine Cesarano , Antonio d'Acierno , Antonio Picariello, An intelligent search agent system for semantic information retrieval on the internet, Proceedings of the 5th ACM international workshop on Web information and data management, November 07-08, 2003, New Orleans, Louisiana, USA [doi>10.1145/956699.956725]
• 12: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.11-16, p.1623-1640, May 17, 1999
• 13: Chien Chin Chen , Meng Chang Chen , Yeali Sun, PVA: a self-adaptive personal view agent system, Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, p.257-262, August 26-29, 2001, San Francisco, California [doi>10.1145/502512.502548]
• 14: Jianjun Chen , David J. DeWitt , Feng Tian , Yuan Wang, NiagaraCQ: a scalable continuous query system for Internet databases, Proceedings of the 2000 ACM SIGMOD international conference on Management of data, p.379-390, May 15-18, 2000, Dallas, Texas, United States
• 15: Liren Chen , Katia Sycara, WebMate: a personal agent for browsing and searching, Proceedings of the second international conference on Autonomous agents, p.132-139, May 10-13, 1998, Minneapolis, Minnesota, United States [doi>10.1145/280765.280789]
• 16: Zhixiang Chen , Xiannong Meng , Richard H. Fowler , Binhai Zhu, FEATURES: real-time adaptive feature and document learning for Web search, Journal of the American Society for Information Science and Technology, v.52 n.8, p.655-665, June 2001 [doi>10.1002/asi.1115.abs]
• 17: Cyber Atlas. 1999. U.S. top 50 internet properties, Dec. 1999, at home/work combined. 1 (July 2000).
• 18: Cyber Atlas. 2001. U.S. top 50 internet properties, (May 2001) at home/work combined. (July 2000).
• 19: Cyber Atlas. 2002. (Nov. 2002) internet usage stats. (Jan. 2002).
• 20: Michelangelo Diligenti , Frans Coetzee , Steve Lawrence , C. Lee Giles , Marco Gori, Focused Crawling Using Context Graphs, Proceedings of the 26th International Conference on Very Large Data Bases, p.527-534, September 10-14, 2000
• 21: Robert B. Doorenbos , Oren Etzioni , Daniel S. Weld, A scalable comparison-shopping agent for the World-Wide Web, Proceedings of the first international conference on Autonomous agents, p.39-48, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267666]
• 22: Dumais, S. T. 2002. Web experiments and test collections. The 11th International World Wide Web Conference. 2003 (April).
• 23: Jenny Edwards , Kevin McCurley , John Tomlin, An adaptive model for optimizing performance of an incremental web crawler, Proceedings of the 10th international conference on World Wide Web, p.106-113, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371960]
• 24: Etzioni, O. 1996a. Moving Up the information food chain: Deploying softbots on the World Wide Web. In Proceedings of the 13th National Conference on Artificial Intelligence and the 8th Innovative Applications of Artificial Intelligence Conference. 1322--1326.
• 25: Larry Fitzpatrick , Mei Dent, Automatic feedback using past queries: social searching?, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.306-313, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 26: Gary W. Flake , Eric J. Glover , Steve Lawrence , C. Lee Giles, Extracting query modifications from nonlinear SVMs, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511488]
• 27: Rayid Ghani , Rosie Jones , Dunja Mladenic, Online Learning for Web Query Generation: Finding Documents Matching a Minority Concept on the Web, Proceedings of the First Asia-Pacific Conference on Web Intelligence: Research and Development, p.508-513, October 23-26, 2001
• 28: Natalie S. Glance, Community search assistant, Proceedings of the 6th international conference on Intelligent user interfaces, p.91-96, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.360293]
• 29: Eric J. Glover , Gary W. Flake , Steve Lawrence , Andries Kruger , David M. Pennock , William P. Birmingham , C. Lee Giles, Improving Category Specific Web Search by Learning Query Modifications, Proceedings of the 2001 Symposium on Applications and the Internet (SAINT 2001), p.23, January 08-12, 2001
• 30: Nathaniel Good , J. Ben Schafer , Joseph A. Konstan , Al Borchers , Badrul Sarwar , Jon Herlocker , John Riedl, Combining collaborative filtering with personal agents for better recommendations, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.439-446, July 18-22, 1999, Orlando, Florida, United States
• 31: James Hendler, Agents and the Semantic Web, IEEE Intelligent Systems, v.16 n.2, p.30-37, March 2001 [doi>10.1109/5254.920597]
• 32: Julio César Hernández , José María Sierra , Arturo Ribagorda , Benjamín Ramos, Search Engines as a Security Threat, Computer, v.34 n.10, p.25-30, October 2001 [doi>10.1109/2.955094]
• 33: Christoph Höscher , Gerhard Strube, Web search behavior of Internet experts and newbies, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.33 n.1-6, p.337-346, June 2000
• 34: Michael N. Huhns , Munindar P. Singh, Personal Assistants, IEEE Internet Computing, v.2 n.5, p.90-92, September 1998 [doi>10.1109/4236.722236]
• 35: Gareth Hurley , David C. Wilson, DubLet: An Online CBR System for Rental Property Recommendation, Proceedings of the 4th International Conference on Case-Based Reasoning: Case-Based Reasoning Research and Development, p.660-674, August 02, 2001
• 36: Lucas Introna , Helen Nissenbaum, Defining the Web: The Politics of Search Engines, Computer, v.33 n.1, p.54-62, January 2000 [doi>10.1109/2.816269]
• 37: Bernard J. Jansen , Udo Pooch, A review of web searching studies and a framework for future research, Journal of the American Society for Information Science and Technology, v.52 n.3, p.235-246, Feb. 1, 2001 [doi>10.1002/1097-4571(2000)9999:9999<::AID-ASI1607>3.3.CO;2-6]
• 38: Jansen, B. J. and Spink, A. 2003. An analysis of Web information seeking and use: Documents retrieved versus documents viewed. In Proceedings of the 4th International Conference on Internet Computing. Las Vegas, NV, 65--69.
• 39: Jansen, B. J. and Spink, A. 2005. An analysis of Web searching by European Alltheweb.com users. Inform. Process. Manag. 42, 1, 248--263.
• 40: Bernard J. Jansen , Amanda Spink , Judy Bateman , Tefko Saracevic, Real life information retrieval: a study of user queries on the Web, ACM SIGIR Forum, v.32 n.1, p.5-17, Spring 1998 [doi>10.1145/281250.281253]
• 41: Jansen, B. J., Spink, A. and Pederson, J. 2003a. Monsters at the gates: When Softbots visit web search engines. In Proceedings of the 4th International Conference on Internet Computing. Las Vegas, NV, 620--626.
• 42: Jansen, B. J., Spink, A. and Pederson, J. 2003b. Web searching agents: What are they doing out there? In Proceedings of the 2003 IEEE International Conference on Systems, Man and Cybernetics. Washington, DC, 10--16.
• 43: Bernard J. Jansen , Amanda Spink , Jan Pedersen, A temporal comparison of AltaVista Web searching: Research Articles, Journal of the American Society for Information Science and Technology, v.56 n.6, p.559-570, April 2005 [doi>10.1002/asi.v56:6]
• 44: Bernard J. Jansen , Amanda Spink , Tefko Saracevic, Real life, real users, and real needs: a study and analysis of user queries on the web, Information Processing and Management: an International Journal, v.36 n.2, p.207-227, Jan.1.2000 [doi>10.1016/S0306-4573(99)00056-4]
• 45: Joachims, T., Freitag, D. and Mitchell, T. 1997. WebWatcher: A tour guide for the World Wide Web. In Proceedings of the 15th International Joint Conference on Artificial Intelligence (IJCAI 97). 770--775.
• 46: Jones, W. 2004. Finders, keepers? The present and future perfect in support of personal information management. First Monday. 9, 3.
• 47: Anupam Joshi, On proxy agents, mobility, and web access, Mobile Networks and Applications, v.5 n.4, p.233-241, Dec. 2000 [doi>10.1023/A:1019120915034]
• 48: Knoblock, C. A., Minton, S., Ambite, J. L., Ashish, N., Muslea, I., Philpot, A. G. and Tejada, S. 2001a. The Ariadne approach to Web-based information integration. Int. J. Coopera. Inform. Syst. (IJCIS). 10, 12, 145--169.
• 49: Koster, M. 1998. The Web robots FAQ. www.robotstxt.org/wc/faq.html 15 (March 2002).
• 50: Lawrence, S. 2001. Online or invisible? Nature. 411,6837, 521.
• 51: Steve Lawrence , C. Lee Giles , Kurt Bollacker, Digital Libraries and Autonomous Citation Indexing, Computer, v.32 n.6, p.67-71, June 1999 [doi>10.1109/2.769447]
• 52: Lee, G., Lee, J.-H., Rho, H., Park, Y.-T., Choi, J. and Seo, J. 1998. Interactive NLI agent for multiagent Web search model. In Proceedings of the International Workshop on Intelligent Agents on the Internet and Web, in 4th World Congress on Expert Systems. Mexico City, Mexico, 67--74.
• 53: Henry Lieberman , Christopher Fry , Louis Weitzman, Exploring the Web with reconnaissance agents, Communications of the ACM, v.44 n.8, p.69-75, Aug. 2001 [doi>10.1145/381641.381661]
• 54: Shou-de Lin , Craig A. Knoblock, Exploiting a Search Engine to Develop More Flexible Web Agents, Proceedings of the IEEE/WIC International Conference on Web Intelligence, p.54, October 13-17, 2003
• 55: Hongen Lu , Leon Sterling, Interoperability and Semi-Structured Data in an Open Web-Based Agent Information System, Proceedings of the First International Conference on Web Information Systems Engineering (WISE'00)-Volume 1, p.80, June 19-20, 2000
• 56: Samuel Madden , Mehul Shah , Joseph M. Hellerstein , Vijayshankar Raman, Continuously adaptive continuous queries over streams, Proceedings of the 2002 ACM SIGMOD international conference on Management of data, June 03-06, 2002, Madison, Wisconsin [doi>10.1145/564691.564698]
• 57: Innes Martin , Joemon M. Jose, A personalised information retrieval tool, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860532]
• 58: Filippo Menczer , Gautam Pant , Padmini Srinivasan , Miguel E. Ruiz, Evaluating topic-driven web crawlers, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.241-249, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383995]
• 59: Retrieving and Semantically Integrating Heterogeneous Data from the Web, IEEE Intelligent Systems, v.19 n.3, p.72-79, May 2004 [doi>10.1109/MIS.2004.16]
• 60: The W3C's Semantic Web Activity: An Update, IEEE Intelligent Systems, v.19 n.3, p.95-96, C3, May 2004 [doi>10.1109/MIS.2004.19]
• 61: Dunja Mladenic, Text-Learning and Related Intelligent Agents: A Survey, IEEE Intelligent Systems, v.14 n.4, p.44-54, July 1999 [doi>10.1109/5254.784084]
• 62: Munarriz, R. A. 1997. How did it double? www.tool.com/ddouble/1997/ddouble 970812 html/. 10 November,
• 63: Gautam Pant , Filippo Menczer, MySpiders: Evolve Your Own Intelligent Web Crawlers, Autonomous Agents and Multi-Agent Systems, v.5 n.2, p.221-229, June 2002 [doi>10.1023/A:1014853428272]
• 64: James Pitkow , Hinrich Schütze , Todd Cass , Rob Cooley , Don Turnbull , Andy Edmonds , Eytan Adar , Thomas Breuel, Personalized search, Communications of the ACM, v.45 n.9, September 2002 [doi>10.1145/567498.567526]
• 65: B. J. Rhodes , P. Maes, Just-in-time information retrieval agents, IBM Systems Journal, v.39 n.3-4, p.685-704, July 2000
• 66: Neil C. Rowe, Marie-4: A High-Recall, Self-Improving Web Crawler That Finds Images Using Captions, IEEE Intelligent Systems, v.17 n.4, p.8-14, July 2002 [doi>10.1109/MIS.2002.1024745]
• 67: Neil C. Rowe, Marie-4: A High-Recall, Self-Improving Web Crawler That Finds Images Using Captions, IEEE Intelligent Systems, v.17 n.4, p.8-14, July 2002 [doi>10.1109/MIS.2002.1024745]
• 68: Searchtools.Com. 2001. Source Code for Web Robot Spiders.
• 69: Selberg, E. and Etzioni, O. 1995. Multi-service search and comparison using the metacrawler. In Proceedings of the 4th International World-Wide Web Conference. Boston, MA.
• 70: Design and Implementation of a High-Performance Distributed Web Crawler, Proceedings of the 18th International Conference on Data Engineering, p.357, February 26-March 01, 2002
• 71: Craig Silverstein , Hannes Marais , Monika Henzinger , Michael Moricz, Analysis of a very large web search engine query log, ACM SIGIR Forum, v.33 n.1, p.6-12, Fall 1999 [doi>10.1145/331403.331405]
• 72: Gabriel Somlo , Adele E. Howe, Adaptive Lightweight Text Filtering, Proceedings of the 4th International Conference on Advances in Intelligent Data Analysis, p.319-329, September 13-15, 2001
• 73: Gabriel L. Somlo , Adele E. Howe, Using web helper agent profiles in query generation, Proceedings of the second international joint conference on Autonomous agents and multiagent systems, July 14-18, 2003, Melbourne, Australia [doi>10.1145/860575.860706]
• 74: Spink, A. and Jansen, B. J. 2004. Web Search: Public Searching of the Web. Kluwer, New York, NY.
• 75: Amanda Spink , Bernard J. Jansen , Dietmar Wolfram , Tefko Saracevic, From E-Sex to E-Commerce: Web Search Changes, Computer, v.35 n.3, p.107-109, March 2002 [doi>10.1109/2.989940]
• 76: Sullivan, D. 2002. Search Engine Math. www.searchenginewatch.com/showPage.html 11 April,
• 77: Sullivan, D. 2003. Search Utilities. www.searchenginewatch.com 16 (March 2002).
• 78: J. Talim , Z. Liu , Ph. Nain , E. G. Coffman, Jr., Controlling the robots of Web search engines, Proceedings of the 2001 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.236-244, June 2001, Cambridge, Massachusetts, United States
• 79: Christoph G. Thomas , Gerhard Fischer, Using agents to personalize the Web, Proceedings of the 2nd international conference on Intelligent user interfaces, p.53-60, January 06-09, 1997, Orlando, Florida, United States [doi>10.1145/238218.238287]
• 80: Angi Voss , Thomas Kreifelts, SOAP: social agents providing people with useful information, Proceedings of the international ACM SIGGROUP conference on Supporting group work: the integration challenge, p.291-298, November 16-19, 1997, Phoenix, Arizona, United States [doi>10.1145/266838.267334]
• 81: J. L. Wolf , M. S. Squillante , P. S. Yu , J. Sethuraman , L. Ozsen, Optimal crawling strategies for web search engines, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511465]
• 82: Xiaohui, Z., Huayong, W., Guiran, C. and Hong, Z. 2001. An autonomous system-based distribution system for Web search. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. Tucson, AZ, 435--440.
• 83: G. Michael Youngblood, Web hunting: design of a simple intelligent Web search agent, Crossroads, v.5 n.4, p.26-32, Summer 1999 [doi>10.1145/331648.331661]
• 84: Edmund S. Yu , Ping C. Koo , Elizabeth D. Liddy, Evolving intelligent text-based agents, Proceedings of the fourth international conference on Autonomous agents, p.388-395, June 03-07, 2000, Barcelona, Spain [doi>10.1145/336595.337550]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

General Terms:
Algorithms, Performance

Keywords:
Search engines, Web searching, agent searching