Abstract

This paper exploits the Data, Information, Knowledge,Wisdom (DIKW) hierarchy as a framework for modelling the scientific data produced during large-scale evaluation campaigns for information retrieval systems in order to design a digital library system able to manage and support the course of such evaluation campaigns.

DIKW and the Information Space of an Evaluation Campaign

The Data, Information, Knowledge, Wisdom (DIKW) hierarchy [3] can be used to structure the scientific data produced during a large-scale evaluation campaign and to design a digital library system that manages such data in order to better support their curation, preservation, enrichment, and reuse [1], over four layers:

1. at the data layer there are raw, basic elements, partial and atomized, which have little meaning by themselves and no significance beyond their existence. Data are created with facts, can be measured, and can be viewed as the building blocks of the other layers. Despite the possibility of manipulation, a limited amount of actions can be performed with it;
2. the information layer is the result of computations and processing of the data. Information comes from the form taken by the data when they are grouped and organised in different ways to create relational connections: the word inform itself means etymologically to give shape, to form, thus entailing the notion of giving data a new shape by relating them;
3. the knowledge layer is related to the generation of appropriate actions, by using the appropriate collection of information gathered at the previous level of the hierarchy. It can be articulated into a language, more or less formal, such as words, numbers, expressions and so on, transmitted to others, or be embedded in individual experience, like beliefs or intuitions;
4. the wisdom layer provides interpretation, explanation, and formalization of the content of the previous levels. Wisdom is not one thing: it is the highest level of understanding, and is a uniquely human state. The previous levels are related to the past; with wisdom people can strive to the future.

This approach suggests that the information resources of an evaluation campaign are increasingly complex and are often the result of the processing of existing resources. The final aim is to better support researchers and developers by providing them with tools that facilitate their work and to promote knowledge transfer towards interested communities [2].

DIKW and the Actors of an Evaluation Campaign

It is possible to think of each campaign as a cycle where the information resources line up on a spiral staircase and allow the different types of actors involved in an evaluation campaign to move from data towards wisdom:

Click here for a larger view. After the image opens, click on it to zoom.

• the organizer manages the different aspects of an evaluation forum: he or she creates the experimental collections, the topics and the relevance assessments; provides the framework for the participants and for the assessors to perform the relevance assessments; and computes the different measures for assessing the performances of the submitted experiments as well as descriptive statistics and statistical tests to characterize the overall features of the submitted experiments;
• the participant tests his or her new algorithms and techniques, compares their effectiveness, and discusses sharing his or her proposals. Moreover, he or she should have the possibility of properly citing experiments and other information resources and getting a citation correctly resolved;
• the assessor contributes to the creation of the experimental collections by both proposing the topics and assessing the relevance of the documents with respect to those topics;
• the visitor needs to consult, browse, properly cite and access all the information resources produced during the course of an evaluation campaign and made available by the organizer in a meaningful fashion, which provides insights about the conducted experiments.

Distributed Information Retrieval Evaluation Campaign Tool (DIRECT) is a digital library system developed, adopted and tested in the Cross-Language Evaluation Forum (CLEF) since 2005 and designed according to this methodology.

Acknowledgements

The work reported has been partially supported by the TrebleCLEF Coordination Action, as part of the Seventh Framework Programme of the European Commission, Theme ICT-1-4-1 Digital libraries and technology- enhanced learning (Contract 215231).

References

[1] M. Agosti, G. M. Di Nunzio, and N. Ferro. A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns. In Proc. 1st International Workshop on Evaluating Information Access (EVIA 2007), pages 62-73. National Institute of Informatics, Tokyo, Japan, 2007.

[2] M. Agosti, G. M. Di Nunzio, N. Ferro, D. Harman, and C. Peters. The Future of Large-scale Evaluation Campaigns for Information Retrieval in Europe. In Proc. 11th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2007), pages 509-512. LNCS 4675, Springer, Heidelberg, Germany, 2007.

[3] M. Zeleny. Management Support Systems: Towards Integrated Knowledge Management. Human Systems Management, 7(1):59-70, 1987.