Direct: applying the DIKW hierarchy to large-scale evaluation campaigns

ABSTRACT

We describe the effort of designing and developing a digital library system able to manage the different types of information resources produced during a large-scale evaluation campaign and to support the different stages of it. In this context, we present DIRECT, the system which has been adopted to manage the CLEF evaluation campaigns since 2005.

REFERENCES

• 1: M. Agosti, G. M. Di Nunzio, and N. Ferro. The Importance of Scientific Data Curation for Evaluation Campaigns. In Proc. 1st Int. DELOS Conference., pages 157--166. LNCS 4877, Springer, 2007.
• 2: G. M. Di Nunzio and N. Ferro. DIRECT: a System for Evaluating Information Access Components of Digital Libraries. In Proc. ECDL 2005, pages 483--484. LNCS 3652, Springer, 2005.
• 3: M. Dussin and N. Ferro. Design of the User Interface of a Scientific Digital Library System for Large-Scale Evaluation Campaigns. In Second DELOS Conference - Working Notes. ISTI-CNR, 2007.
• 4: M. Zeleny. Management Support Systems: Towards Integrated Knowledge Management. Human Systems Management, 7(1):59--70, 1987.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

General Terms:
Design, Experimentation, Management, Performance

Keywords:
DIRECT, digital library system, large-scale evaluation campaigns, scientific data