Complete statistical indexing of text by overlapping word fragments

ABSTRACT

By using overlapping word fragments to index text, we can combine the best features of the keyword and the full text approaches to document retrieval so as to facilitate searches on any content word. The characteristics of a retrieval system based on word fragment indexing can be precisely predicted from a multinomial model of text. Controlled experiments with two different text collections indicate that such a system can be highly effective under quite general conditions.