Personalized spiders for web search and analysis

ABSTRACT

Searching for useful information on the World Wide Web has become incr easingly difficult. While Internet search engines have been helping people to search on the web, low recall rate and outdated indexes have become more and more problematic as the web grows. In addition, search tools usually present to the user only a list of search results, failing to provide further personalized analysis which could help users identify useful information and comprehend these results. To alleviate these problems, we propose a client-based architecture that incorporates noun phrasing and self-organizing map techniques. Two systems, namely CI Spider and Meta Spider, have been built based on this architecture. User evaluation studies have been conducted and the findings suggest that the proposed architecture can effectively facilitate web search and analysis.

REFERENCES

• 1: C. Mic Bowman , Peter B. Danzig , Udi Manber , Michael F. Schwartz, Scalable Internet resource discovery: research problems and approaches, Communications of the ACM, v.37 n.8, p.98-ff., Aug. 1994 [doi>10.1145/179606.179704]
• 2: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Proceeding of the eighth international conference on World Wide Web, p.1623-1640, May 1999, Toronto, Canada
• 3: Chen, H., Schufels, C. and Orwig, R. Internet Categorization and Search: A Self-Organizing Approach, Journal of Visual Communication and Image Representation, 7(1): 88-102 (1996).
• 4: Mark H. Chignell , Jacek Gwizdka , Richard C. Bodner, Discriminating meta-search: a framework for evaluation, Information Processing and Management: an International Journal, v.35 n.3, p.337-362, May 1999 [doi>10.1016/S0306-4573(98)00065-X]
• 5: Junghoo Cho , Hector Garcia-Molina , Lawrence Page, Efficient crawling through URL ordering, Proceedings of the seventh international conference on World Wide Web 7, p.161-172, April 1998, Brisbane, Australia
• 6: Edward A. Fox , Deborah Hix , Lucy T. Nowell , Dennis J. Brueni , Durgesh Rao , William C. Wake , Lenwood S. Heath, Users, user interfaces, and objects: Envision, a digital library, Journal of the American Society for Information Science, v.44 n.8, p.480-491, Sept. 1993 [doi>10.1002/(SICI)1097-4571(199309)44:8<480::AID-ASI7>3.0.CO;2-B]
• 7: Marti A. Hearst, TileBars: visualization of term distribution information in full text information access, Proceedings of the SIGCHI conference on Human factors in computing systems, p.59-66, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223912]
• 8: Marti A. Hearst , Jan O. Pedersen, Reexamining the cluster hypothesis: scatter/gather on retrieval results, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.76-84, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243216]
• 9: Inktomi WebMap, http://www.inktomi.com/webmap/
• 10: Teuvo Kohonen, Self-organizing maps, Springer-Verlag New York, Inc., Secaucus, NJ, 1997
• 11: Kohonen, T. Exploration of Very Large Databases by Self- Organizing Maps, in Proceedings of the IEEE International Conference on Neural Networks, 1:1-6 (1997).
• 12: Lawrence, S. and Giles, C. L. Accessibility of Information on the Web, Nature, 400 (1999), 107-109.
• 13: Lin, C., Chen, H. and Nunamaker J. Verifying the Proximity and Size Hypothesis for Self-Organizing Maps. Journal of Management Information Systems, 16(3) (1999- 2000), 61-73.
• 14: Xia Lin , Dagobert Soergel , Gary Marchionini, A self-organizing semantic map for information retrieval, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.262-269, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122887]
• 15: Pattie Maes, Agents that reduce work and information overload, Communications of the ACM, v.37 n.7, p.30-40, July 1994 [doi>10.1145/176789.176792]
• 16: McBryan, O. GENVL and WWW: Tools for Taming the Web, in Proceedings of the 1st International World Wide Web Conference (Geneva, Switzerland, Mar 1994).
• 17: Edie Rasmussen, Clustering algorithms, Information retrieval: data structures and algorithms, Prentice-Hall, Inc., Upper Saddle River, NJ, 1992
• 18: Gerard Salton, Another look at automatic text-retrieval systems, Communications of the ACM, v.29 n.7, p.648-656, July 1986 [doi>10.1145/6138.6149]
• 19: Ben Shneiderman , David Feldman , Anne Rose , Xavier Ferré Grau, Visualizing digital library search results with categorical and hierarchical axes, Proceedings of the fifth ACM conference on Digital libraries, p.57-66, June 02-07, 2000, San Antonio, Texas, United States [doi>10.1145/336597.336637]
• 20: Selberg, E. and Etzioni, O. The MetaCrawler architecture for resource aggregation on the Web. IEEE Expert (1997).
• 21: Kristin M. Tolle , Hsinchun Chen, Comparing noun phrasing techniques for use with medical digital library tools, Journal of the American Society for Information Science, v.51 n.4, p.352-370, March 1, 2000 [doi>10.1002/(SICI)1097-4571(2000)51:4<352::AID-ASI5>3.3.CO;2-#]
• 22: Aravindan Veerasamy , Nicholas J. Belkin, Evaluation of a tool for visualization of information retrieval results, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.85-92, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243218]
• 23: Oren Zamir , Oren Etzioni, Grouper: a dynamic clustering interface to Web search results, Proceeding of the eighth international conference on World Wide Web, p.1361-1374, May 1999, Toronto, Canada

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Web-based interaction

General Terms:
Design, Experimentation

Keywords:
information retrieval, internet searching and browsing, internet spider, noun-phrasing, personalization, self-organizing map