Collecting hidden weeb pages for data extraction

ABSTRACT

As the Web grows, more and more data has become available under dynamic forms of publication, such as a legacy database accessed by an HTML form (the so called Hidden Web). In situations such as this, integration of this data relies more and more on the fast generation of page fetching agents. As a result, there is an increasing need for tools that can help the user to generate such agents. In this paper, we describe an approach to automatically generating agents to collect hidden Web pages that uses a pre-existing data repository for identifying the contents of these pages and takes the advantage of some regularities that can be found among Web sites. To demonstrate the effectiveness of our approach, we discuss the results of a number of experiments carried out with sites from different domains. We also dicuss how such regularities among sites can be formalized.

REFERENCES

• 1: M. K. Bergman. The deep Web: Surfacing hidden value. White Paper, Bright Planet, 2000.
• 2: Hasan Davulcu , Juliana Freire , Michael Kifer , I. V. Ramakrishnan, A layered architecture for querying dynamic Web content, Proceedings of the 1999 ACM SIGMOD international conference on Management of data, p.491-502, May 31-June 03, 1999, Philadelphia, Pennsylvania, United States
• 3: Robert B. Doorenbos , Oren Etzioni , Daniel S. Weld, A scalable comparison-shopping agent for the World-Wide Web, Proceedings of the first international conference on Autonomous agents, p.39-48, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267666]
• 4: D. W. Embley , D. M. Campbell , Y. S. Jiang , S. W. Liddle , D. W. Lonsdale , Y.---K. Ng , R. D. Smith, Conceptual-model-based data extraction from multiple-record Web pages, Data & Knowledge Engineering, v.31 n.3, p.227-251, Nov. 1999 [doi>10.1016/S0169-023X(99)00027-0]
• 5: Daniela Florescu , Alon Levy , Alberto Mendelzon, Database techniques for the World-Wide Web: a survey, ACM SIGMOD Record, v.27 n.3, p.59-74, Sept. 1, 1998 [doi>10.1145/290593.290605]
• 6: Paulo B. Golgher , Altigran S. da Silva , Alberto H. F. Laender , Berthier Ribeiro-Neto, Bootstrapping for example-based data extraction, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502648]
• 7: Paulo Braz Golgher , Alberto H. F. Laender , Altigran Soares da Silva , Berthier A. Ribeiro-Neto, An Example-Based Environment for Wrapper Generation, Proceedings of the Workshops on Conceptual Modeling Approaches for E-Business and The World Wide Web and Conceptual Modeling: Conceptual Modeling for E-Business and the Web, p.152-164, October 09-12, 2000
• 8: Bruce Krulwich, Automating the Internet: Agents as User Surrogates, IEEE Internet Computing, v.1 n.4, p.34-38, July 1997 [doi>10.1109/4236.612213]
• 9: Alberto H. F. Laender , Altigran S. da Silva , Paolo B. Golgher , Berthier Ribeiro-Neto , Irna M. R. Evangelista-Filha , Karine V. Magalhães, The Debye Environment for Web Data Management, IEEE Internet Computing, v.6 n.4, p.60-69, July 2002 [doi>10.1109/MIC.2002.1020327]
• 10: Alberto H. F. Laender , Berthier Ribeiro-Neto , Altigran S. da Silva, DEByE - Date extraction by example, Data & Knowledge Engineering, v.40 n.2, p.121-154, February 2002 [doi>10.1016/S0169-023X(01)00047-7]
• 11: S. Lawrence and C. Giles. Searching the World-Wide Web. Science, 280(4):98--100, 1998.
• 12: Sriram Raghavan , Hector Garcia-Molina, Crawling the Hidden Web, Proceedings of the 27th International Conference on Very Large Data Bases, p.129-138, September 11-14, 2001

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Web-based interaction
      H.5.4 Hypertext/Hypermedia
          Subjects: Navigation

General Terms:
Algorithms, Experimentation

Keywords:
collecting agents, hidden web, navigation patterns