Another look at automatic text-retrieval systems

ABSTRACT

Evidence from available studies comparing manual and automatic text-retrieval systems does not support the conclusion that intellectual content analysis produces better results than comparable automatic systems.

REFERENCES

• 1: David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
• 2: Cleverdon. C.W. A computer evaluation of searching by controlled language and natural language in an experimental NASA data base. Rep. ESA l/432, European Space Agency, Frascati. Italy, July 1977. A description of a large-scale test of the NASA search system using various manual and automatic text-analysis methods.
• 3: Cyril Cleverdon, Optimizing convenient online access to bibliographic databases, Information Services and Use, v.4 n.1-2, p.37-47, April 1984
• 4: Cleverdon. C.W., and Keen, E.M. Aslib-Cranfield Research Project. Vol. 2. Test Results. Cranfield Institute of Technology, Cranfield. England, 1966. The report on the most thorough evaluation of automatic versus manual text-analysis methods ever carried out, using a collection of 1400 aeronautics documents.
• 5: Croft, W.B., and Harper, O.J. Using probabilistic models of document retrieval without relevance information. 1. Dot. 35, 4 (Dec. 1979). 285-295. Describes a method for using probabilistic considerations of term relevance for an initial collection search before any relevance information is available.
• 6: IBM World Trade Corporation. Storage and Znformafion Refrieval Sysfern (STAIRS}-General Iilformafion Manual. 2nd ed. IBM Germany. Stuttgart, Germany. Apr. 1972. Contains an early description of the IBM/STAIRS system.
• 7: Lancaster, F.W. Evaluation of fhe Medlars Demand Search Service. National Library of Medicine, Bethesda, Md., Jan. 1968. An impressive description of the in-house test of the Medlars search system carried out at the National Library of Medicine.
• 8: Lancaster, F.W. information Retrieval Systems: Clraracferistics, Testing, and EualuaGon. 2nd ed. Wiley, New York, 1979. A well-known textbook in information retrieval with an emphasis on system testing and evaluation.
• 9: Lovins, J.B. Development of a stemming algorithm. Mech. Transl. Comput. Linguist. II. 1-2 (Mar. and June 1968), 11-31. A detailed description of an automatic word-stemming algorithm.
• 10: Robertson, SE.. and Sparck Jones, K. Relevance weighting of search terms. 1. ASIS 27, 3 (May-June 1976), 129-146. Describes one of the main probabilistic information-retrieval models.
• 11: Salton, G. Automatic text analysis. Science 168, 3929 (Apr. 1970), 335-343. A survey of automatic text retrieval as of 1970.
• 12: G. Salton, Recent Studies in Automatic Text Analysis and Document Retrieval, Journal of the ACM (JACM), v.20 n.2, p.258-278, April 1973 [doi>10.1145/321752.321757]
• 13: G. Salton, A blueprint for automatic indexing, ACM SIGIR Forum, v.16 n.2, p.22-38, Fall 1981 [doi>10.1145/1095425.1095427]
• 14: G. Salton, A blueprint for automatic Boolean query processing, ACM SIGIR Forum, v.17 n.2, p.6-24, Fall 1982 [doi>10.1145/1095479.1095480]
• 15: G. Salton , M. E. Lesk, Computer Evaluation of Indexing and Text Processing, Journal of the ACM (JACM), v.15 n.1, p.8-36, Jan. 1968 [doi>10.1145/321439.321441]
• 16: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 17: Gerard Salton , Edward A. Fox , Harry Wu, Extended Boolean information retrieval, Communications of the ACM, v.26 n.11, p.1022-1036, Nov. 1983 [doi>10.1145/182.358466]
• 18: Salton, G.. Yang, C.S., and Yu, CT. A theory of term importance in automatic text analysis. I. ASK 26, 1 (Jan.-Feb. 1975), 33-44. Contains a description of term-discrimination theory and some retrieval results based on discrimination value weighting.
• 19: Sparck Jones, K. A statistical interpretation of term specificity and its application in retrieval. J Dot. 28, 1 (Mar. 1972), 11-21. Relates the usefulness of index terms to certain statistical term occurrence parameters.
• 20: Swanson. D.R. Searching natural language text by computer. Science 132, 3434 (Oct. 1960), 1099-1104. A pioneering small-scale test comparing an automatic text-search system with a conventional retrieval system based on manual indexing: probably the earliest result showing the superiority of automatic text searching.
• 21: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979

INDEX TERMS

Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods; Thesauruses
      H.3.3 Information Search and Retrieval
          Subjects: Search process; Retrieval models

General Terms:
Design, Experimentation, Measurement, Performance, Theory