Extracting unstructured data from template generated web documents

ABSTRACT

We propose a novel approach that identifies web page templates and extracts the unstructured data. Extracting only the body of the page and eliminating the template increases the retrieval precision for the queries that generate irrelevant results. We believe that by reducing the number of irrelevant results; the users are encouraged to go back to a given site to search. Our experimental results on several different web sites and on the whole cnnfn collection demonstrate the feasibility of our approach.

REFERENCES

• 1: Valter Crescenzi , Giansalvatore Mecca , Paolo Merialdo, RoadRunner: automatic data extraction from data-intensive web sites, Proceedings of the 2002 ACM SIGMOD international conference on Management of data, June 03-06, 2002, Madison, Wisconsin [doi>10.1145/564691.564778]
• 2: Arvind Arasu , Hector Garcia-Molina , Stanford University, Extracting structured data from Web pages, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872799]
• 3: Helena Ahonen-Myka, Discovery of Frequent Word Sequences in Text, Proceedings of the ESF Exploratory Workshop on Pattern Detection and Discovery, p.180-189, September 16-19, 2002
• 4: Florian Beil , Martin Ester , Xiaowei Xu, Frequent term-based text clustering, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775110]
• 5: Ziv Bar-Yossef , Sridhar Rajagopalan, Template detection via data mining and its applications, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511522]
• 6: Xiaoli Li , Tong-Heng Phang , Minqing Hu , Bing Liu, Using micro information units for internet search, Proceedings of the eleventh international conference on Information and knowledge management, November 04-09, 2002, McLean, Virginia, USA [doi>10.1145/584792.584885]
• 7: David A. Grossman , Ophir Frieder, Information Retrieval: Algorithms and Heuristics, Kluwer Academic Publishers, Norwell, MA, 1998
• 8: Abdur Chowdhury , Ophir Frieder , David Grossman , Mary Catherine McCabe, Collection statistics for fast duplicate document detection, ACM Transactions on Information Systems (TOIS), v.20 n.2, p.171-191, April 2002 [doi>10.1145/506309.506311]
• 9: Keinosuke Fukunaga, Introduction to statistical pattern recognition (2nd ed.), Academic Press Professional, Inc., San Diego, CA, 1990
• 10: Thomas J. Froehlich, Relevance reconsidered—towards an agenda for the 21st century: introduction to special topic issue on relevance research, Journal of the American Society for Information Science, v.45 n.3, p.124-134, April 1994 [doi>10.1002/(SICI)1097-4571(199404)45:3<124::AID-ASI2>3.0.CO;2-8]
• 11: L. Su,, H. Chen,, and X. Dong, "Evaluation of Web-based search engines from the end-user's perspective: a pilot study", Proc. of the Conf. for the American Soc. for Inf. Science, 1998.
• 12: S.E. Robertson, S. Walker and M. Beaulieu. "Okapi at TREC-7: automatic ad hoc, filtering, and interactive", Proceedings of the Seventh Text REtrieval Conference (TREC-7), 1999.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

General Terms:
Design, Experimentation

Keywords:
automatic template removal, information retrieval, retrieval accuracy, text extraction