Removing manually generated boilerplate from electronic texts: experiments with project Gutenberg e-books

ABSTRACT

Collaborative work on unstructured or semi-structured documents, such as in literature corpora or source code, often involves agreed upon templates containing metadata. These templates are not consistent across users and over time. Rule-based parsing of these templates is expensive to maintain and tends to fail as new documents are added. Statistical techniques based on frequent occurrences have the potential to identify automatically a large fraction of the templates, thus reducing the burden on the programmers. We investigate the case of the Project Gutenberg™ corpus, where most documents are in ASCII format with preambles and epilogues that are often copied and pasted or manually typed. We show that a statistical approach can solve most cases though some documents require knowledge of English. We also survey various technical solutions that make our approach applicable to large data sets.

REFERENCES

• 1: T. Atkins. Newgut program. online: http://rumkin.com/reference/gutenberg/newgut, 2004. last checked 18-01-2007.
• 2: Darren C. Atkinson , William G. Griswold, Effective pattern matching of source code using abstract syntax patterns, Software—Practice & Experience, v.36 n.4, p.413-447, April 2006 [doi>10.1002/spe.v36:4]
• 3: R. S. Burkey. GutenMark download page. online: http://www.sandroid.org/GutenMark/download.html, 2005. last checked 18-01-2007.
• 4: Liang Chen , Shaozhi Ye , Xing Li, Template detection for large scale search engines, Proceedings of the 2006 ACM symposium on Applied computing, April 23-27, 2006, Dijon, France [doi>10.1145/1141277.1141534]
• 5: Graham Cormode , S. Muthukrishnan, What's hot and what's not: tracking most frequent items dynamically, ACM Transactions on Database Systems (TODS), v.30 n.1, p.249-278, March 2005 [doi>10.1145/1061318.1061325]
• 6: Sandip Debnath , Prasenjit Mitra , C. Lee Giles, Automatic extraction of informative blocks from webpages, Proceedings of the 2005 ACM symposium on Applied computing, March 13-17, 2005, Santa Fe, New Mexico [doi>10.1145/1066677.1067065]
• 7: Erik D. Demaine , Alejandro López-Ortiz , J. Ian Munro, Frequency Estimation of Internet Packet Streams with Limited Space, Proceedings of the 10th Annual European Symposium on Algorithms, p.348-360, September 17-21, 2002
• 8: David Gibson , Kunal Punera , Andrew Tomkins, The volume and evolution of web page templates, Special interest tracks and posters of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1062745.1062763]
• 9: J. Grunenfelder. Weasel reader: Free reading. online: http://gutenpalm.sourceforge.net/, 2006. last checked 18-01-2007.
• 10: D. Ideda and Y. Yamada. Gathering text files generated from templates. In IIWeb Workshop, VLDB-2004, 2004.
• 11: Richard M. Karp , Scott Shenker , Christos H. Papadimitriou, A simple algorithm for finding frequent elements in streams and bags, ACM Transactions on Database Systems (TODS), v.28 n.1, p.51-55, March 2003 [doi>10.1145/762471.762473]
• 12: O. Kaser and D. Lemire. Removing manually generated boilerplate from electronic texts: Experiments with project gutenberg e-books. Technical Report TR-07-001, Dept. of CSAS, UNBSJ, 2007. available from http://arxiv.org/abs/0707. 1913.
• 13: J. Misra and D. Gries. Finding repeated elements. Sci. Comput. Program., 2(2):143--152, 1982.
• 14: Project Gutenberg Literary Archive Foundation. Project Gutenberg. http://www.gutenberg.org/, 2007. checked 2007-05-30.
• 15: Saul Schleimer , Daniel S. Wilkerson , Alex Aiken, Winnowing: local algorithms for document fingerprinting, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872770]
• 16: R. Segal, J. Crawford, J. Kephart, and B. Leiba. SpamGuru: An enterprise anti-spam filtering system. In Proceedings of the First Conference on E-mail and Anti-Spam, 2004.
• 17: Daria Sorokina , Johannes Gehrke , Simeon Warner , Paul Ginsparg, Plagiarism Detection in arXiv, Proceedings of the Sixth International Conference on Data Mining, p.1070-1075, December 18-22, 2006 [doi>10.1109/ICDM.2006.126]
• 18: Wikipedia. Birthday paradox --- Wikipedia, the free encyclopedia, 2007. {Online; accessed 18-01-2007}.