User evaluation of the NASA technical report server recommendation service

ABSTRACT

We present the user evaluation of two recommendation server methodologies implemented for the NASA Technical Report Server (NTRS). One methodology for generating recommendations uses log analysis to identify co-retrieval events on full-text documents. For comparison, we used the Vector Space Model (VSM) as the second methodology. We calculated cosine similarities and used the top 10 most similar documents (based on metadata) as "recommendations". We then ran an experiment with NASA Langley Research Center (LaRC) staff members to gather their feedback on which method produced the most "quality" recommendations. We found that in most cases VSM outperformed log analysis of co-retrievals. However, analyzing the data revealed the evaluations might have been structurally biased in favor of the VSM generated recommendations. We explore some possible methods for combining log analysis and VSM generated recommendations and suggest areas of future work.

REFERENCES

• 1: Nelson Michael L. , Gottlich Gretchen L. , Bianco David J., World Wide Web Implementation of the Langley Technical Report Server, NASA Langley Technical Report Server, 2003
• 2: Nelson Michael L. , Gottlich Gretchen L. , Bianco David J. , Binkley Robert L. , Kellogg Yvonne D. , Paulson Sharon S. , Beaumont Chris J. , Schmunk Robert B. , Kurtz Michael J. , Accomazzi Alberto, The Widest Practicable Dissemination: The NASA Technical Report Server, NASA Langley Technical Report Server, 1995
• 3: Brewster Kahle, Harry Morris, Franklin Davis, Kevin Tiene, Clare Hart, and Robert Palmer, Wide Area Information Servers: An Executive Information System for Unstructured Files. Electronic Networking Research, Applications and Policy, 2(1), 1992, pp. 59--68.
• 4: Herbert Van de Sompel , Carl Lagoze, Notes from the Interoperability Front: A Progress Report on the Open Archives Initiative, Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, p.144-157, September 16-18, 2002
• 5: Michael L. Nelson, JoAnne Rocker, and Terry L. Harrison, OAI and NASA Scientific and Technical Information, Library Hi-Tech, 21(2), 2003, pp. 140--150.
• 6: Michael L. Nelson , Kurt Maly, Buckets: smart objects for digital libraries, Communications of the ACM, v.44 n.5, p.60-62, May 2001 [doi>10.1145/374308.374342]
• 7: Herbert Van de Sompel, Carl Lagoze, Michael L. Nelson, Simeon Warner, Implementation Guidelines for the Open Archives Initiative Protocol for Metadata Harvesting, 2002. Available at: http://www.openarchives.org/OAI/2.0/guidelines.htm.
• 8: J. Bollen and R. Luce, Evaluation of digital library impact and user communities by analysis of usage patterns, D-Lib Magazine, 8(6), 2002.
• 9: Philip K. Chan, Constructing Web User Profiles: A non-invasive Learning Approach, Revised Papers from the International Workshop on Web Usage Analysis and User Profiling, p.39-55, August 15, 1999
• 10: J. R. Anderson, A spreading activation theory of memory, Journal of Verbal Leaning and Verbal Behavior, 22, 1983, pp. 261--295.
• 11: Fabio Crestani , Puay Leng Lee, Searching the Web by constrained spreading activation, Information Processing and Management: an International Journal, v.36 n.4, p.585-605, July 2000 [doi>10.1016/S0306-4573(99)00073-4]
• 12: Johan Bollen, Group User Models for Personalized Hyperlink Recommendations, Proceedings of the International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, p.38-50, August 28-30, 2000
• 13: Allison Woodruff , Rich Gossweiler , James Pitkow , Ed H. Chi , Stuart K. Card, Enhancing a digital book with a reading recommender, Proceedings of the SIGCHI conference on Human factors in computing systems, p.153-160, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332419]
• 14: Steve Jones, Sally Jo Cunningham, Rodger J. McNab, and Stefan J. Boddie, A transaction log analysis of a digital library, International Journal on Digital Libraries, 3(2), pp. 152--169.
• 15: Michalis Sfakakis , Sarantos Kapidakis, User Behavior Tendencies on Data Collections in a Digital Library, Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, p.550-559, September 16-18, 2002
• 16: Jela Steinerova, In Search of Patterns of User Interaction for Digital Libraries, Proceedings of ECDL 2003, Trondheim Norway, pp. 13--23.
• 17: Houssem Assadi, Thomas Beauvisage, Catherine Lupovici, Thierry Cloarec, Users and Uses of Online Digital Libraries in France, Proceedings of ECDL 2003, Trondheim Norway, pp 1--12.
• 18: Tefko Saracevic, Digital Library Evaluation: Toward an Evolution of Concepts, Library Trends 49(3), 2000, pp. 350--369.
• 19: Miles Efron and Gary Geisler, Using Dimensionality Reduction to Improve Similarity Judgments for Recommendation, Proceedings of the Second DELOS Network of Excellence Workshop on Personalisation and Recommender Systems in Digital Libraries, Dublin City University, Ireland, June 18-20 2001.
• 20: Thomas Hofmann, Latent semantic models for collaborative filtering, ACM Transactions on Information Systems (TOIS), v.22 n.1, p.89-115, January 2004 [doi>10.1145/963770.963774]
• 21: Henry Kautz , Bart Selman , Mehul Shah, Referral Web: combining social networks and collaborative filtering, Communications of the ACM, v.40 n.3, p.63-65, March 1997 [doi>10.1145/245108.245123]
• 22: Rashmi R. Sinha and Kirsten Swearingen, Comparing Recommendations Made by Online Systems and Friends, Proceedings of the Second DELOS Network of Excellence Workshop on Personalisation and Recommender Systems in Digital Libraries, Dublin City University, Ireland, June 18-20 2001.
• 23: Jonathan L. Herlocker , Joseph A. Konstan , Loren G. Terveen , John T. Riedl, Evaluating collaborative filtering recommender systems, ACM Transactions on Information Systems (TOIS), v.22 n.1, p.5-53, January 2004 [doi>10.1145/963770.963772]
• 24: Ann Peterson Bishop, Logins and Bailouts: Measuring Access, Use, and Success in Digital Libraries, Journal of Electronic Publishing, 4(2), 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

General Terms:
Design, Experimentation, Human Factors, Measurement

Keywords:
digital libraries, recommendation servers, user evaluation