Exploiting the deep web with DynaBot: matching, probing, and ranking

ABSTRACT

We present the design of Dynabot, a guided Deep Web discovery system. Dynabot's modular architecture supports focused crawling of the Deep Web with an emphasis on matching, probing, and ranking discovered sources using two key components: service class descriptions and source-biased analysis. We describe the overall architecture of Dynabot and discuss how these components support effective exploitation of the massive Deep Web data available.

REFERENCES

• 1: James Caverlee , Ling Liu , Daniel Rocco, Discovering and ranking web services with BASIL: a personalized approach with biased focus, Proceedings of the 2nd international conference on Service oriented computing, November 15-19, 2004, New York, NY, USA [doi>10.1145/1035167.1035190]
• 2: P. Lyman and H. R. Varian. How much information. www.sims.berkeley.edu/how-much-info-2003, 2003.
• 3: D. Rocco, J. Caverlee, L. Liu, and T. Critchlow. Focused Crawling of the Deep Web using Service Class Descriptions, U. of West Georgia, 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

General Terms:
Algorithms, Design

Keywords:
crawling, deep web, probing, service class