Detecting similar documents using salient terms

ABSTRACT

We describe a system for rapidly determining document similarity among a set of documents obtained from an information retrieval (IR) system. We obtain a ranked list of the most important terms in each document using a rapid phrase recognizer system. We store these in a database and compute document similarity using a simple database query. If the number of terms found to not be contained in both documents is less than some predetermined threshold compared to the total number of terms in the document, these documents are determined to be very similar. We compare this to the shingles approach.

REFERENCES

• 1: Brown, Eric W. and Prager, John M., US Patent 05913208.
• 2: Andrei Z. Broder , Steven C. Glassman , Mark S. Manasse , Geoffrey Zweig, Syntactic clustering of the Web, Selected papers from the sixth international conference on World Wide Web, p.1157-1166, September 1997, Santa Clara, California, United States
• 3: Rabin, M. O., "Fingerprinting by random polynomials, " Center for Research in Computing Technology, Harvard University, Report TR-15-81, 1981.
• 4: Bloomfield, Louis, University of Virginia, interviewed on NPR's All Things Considered, May 9, 2001. See www.plagiarism.phys.virginia.edu.
• 5: James W. Cooper , Roy J. Byrd, Lexical navigation: visually prompted query expansion and refinement, Proceedings of the second ACM international conference on Digital libraries, p.237-246, July 23-26, 1997, Philadelphia, Pennsylvania, United States [doi>10.1145/263690.263828]
• 6: James W. Cooper , Roy J. Byrd, OBIWAN - A Visual Interface for Prompted Query Refinement, Proceedings of the Thirty-First Annual Hawaii International Conference on System Sciences-Volume 2, p.277, January 06-09, 1998 [doi>10.1109/HICSS.1998.651710]
• 7: Ravin, Y. and Wacholder, N. 1996, "Extracting Names from Natural-Language Text," IBM Research Report 20338.
• 8: Justeson, J. S. and S. Katz "Technical terminology: some linguistic properties and an algorithm for identification in text." Natural Language Engineering, 1, 9--27, 1995.
• 9: Byrd, R.J. and Ravin, Y. Identifying and Extracting Relations in Text. Proceedings of NLDB 99, Klagenfurt, Austria.
• 10: Mnis-Textwise Labs, www.textwise.com <http://www.textwise.com>. DR-LINK was developed at Syracuse University and is marketed by Textwise.
• 11: Evans, D. K., Klavans, J. and Wacholder, N., "Document Processing with LinkIT," Proc. Of the RIAO Conference, Paris, France, 2000.
• 12: InXight, Inc. www.inxight.com
• 13: Neff, Mary S. and Cooper, James W. "Document Summarization for Active Markup," in Proceedings of the 32nd Hawaii International Conference on System Sciences, Wailea, HI, January, 1999.
• 14: James W. Cooper , John M. Prager, Anti-Serendipity: Finding Useless Documents and Similar Documents, Proceedings of the 33rd Hawaii International Conference on System Sciences-Volume 3, p.3008, January 04-07, 2000
• 15: Cooper, J. W. "The Technology of Lexical Navigation," Workshop on Browsing Technology, First Joint Conference on Digital Libraries, Roanoke, VA, 2001.
• 16: Cooper, J.W., Cesar, C., So, Edward, and Mack R. L., "Construction of an OO Framework for Text Mining," OOPSLA, Tampa Bay, 2001.
• 17: Gemini plug-in for Adobe Acrobat Reader, Iceni Technology, Ltd, Norwich, England, www.iceni.com <http://www.iceni.com>.
• 18: T. Selkar , W. Burlesson, Context-aware design and interaction in computer systems, IBM Systems Journal, v.39 n.3-4, p.880-891, July 2000
• 19: Cooper, J W, "Loading Your Databases," JavaPro, May, 2000.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Clustering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering; Selection process

General Terms:
Algorithms, Design, Documentation, Measurement, Performance

Keywords:
databases, document similarity, duplicate documents, shingles, text mining