Abstract

We describe the effort of designing and developing a digital library system able to manage the different types of information resources produced during a large-scale evaluation campaign and to support the different stages of it. In this context, we present Distributed Information Retrieval Evaluation Campaign Tool (DIRECT), the system which has been adopted to manage the CLEF evaluation campaigns since 2005.

Overview

Scientific data, their management, access and reuse through citation, curation, enrichment, and preservation are essential components of scientific research. We consider Information Retrieval (IR) experimental evaluation as a source of valuable scientific data and we have proposed a data curation approach in order to better manage, preserve, interpret, and enrich the scientific data produced [1]. The result of our work is Distributed Information Retrieval Evaluation Campaign Tool (DIRECT), a Digital Library System (DLS) which has been developed, adopted and tested in the Cross-Language Evaluation Forum (CLEF) campaigns since 2005 [2, 3]. In designing the system, we have adopted the Data, Information, Knowledge, Wisdom (DIKW) hierarchy [4] in order to structure the information space involved in an evaluation campaign and to model the interaction of the different actors with the created information resources. We provided a demontration of the functioning of the DIRECT system and of the different tasks that can be accomplished with it.

Architecture of the User Interface

According to the investigation of user requirements and needs, the DIRECT user interface is designed to meet the following goals:

• to be cross-platform and easily deployable to end users;
• to be as modular as possible, clearly separating the application logic from the interface logic;
• to be intuitive and capable of providing support for the various user tasks described in the previous section, such as experiment submission, consultation of metrics and plots about experiment performances, relevance assessment, and so on;
• to support different types of users, i.e. participants, assessors, organizers, and visitors, who need to have access to different kinds of features and capabilities;
• to support internationalization and localization: the application needs to be able to adapt to the language of the user and his country or culturally-dependent data, such as dates and currencies.

The modularity of the components has enormous benefits when building interactive applications, since it helps the designer to better understand and develop each component and modify it without affecting the others. Therefore, we used the Model-View-Controller (MVC) approach as provided by the Apache STRUTS framework to clearly separate the following three layers:

• model layer: contains the underlying data structures of the application and keeps the state of the application;
• view layer: the way the model is presented to the user;
• controller layer: manages the interaction between the view and the input devices, such as the keyboard or the mouse, and updates the model accordingly.

For a larger view of Figure 1, click here. After the image opens, click on it to zoom.

Figure 1 shows the architecture of the DIRECT user interface, which is Web-based in order to be cross-platform and easily deployable and accessible without the need of installing any software on the end-user machines. The user interface is based on the Java Server Pages (JSP) technology; in addition, we developed a JavaScript library which provides event listeners, manipulation of the Document Object Model (DOM), and Asynchronous JavaScript Technology and XML (AJAX) support in order to make the user interaction more successful and responsive. In particular, AJAX allows us to make asynchronous calls to the server and to speed up the user interaction by loading only the requested portion of the data without requiring the user to download huge amounts of data in one time or to completely refresh a page when only a part of it has changed.

Moreover, the user interface is made more modular by using the STRUTS TILES templating framework, which allows for a rapid development and reuse of components. As shown in the figure, when the browser requests a page, the STRUTS engine asks the TILES engine to put together the page components, according to instructions provided by an eXtensible Markup Language (XML) configuration file. Then, TILES loads the JSP reusable code segments to create the page skeleton, adds the JavaScript libraries needed for enhancing the user interaction, fills the page with the contents provided by the STRUTS controller, applies the necessary Cascading Style Sheets (CSS) for formatting the page, and returns the dynamically created page to the View layer of STRUTS, which, in turn, sends it to the browser. Finally, we also support the internationalization and localization of the user interface by adapting it to the language and country of the user. This is implemented by using translation files according to the Java internationalization capabilities. The correct language and country are initially loaded according to the browser settings and, in the case of not supported locales, it falls back to a default configuration. The user interface has been translated in the following languages: Bulgarian, Czech, English, French, German, Indonesian, Italian, Portuguese, and Spanish.

DIKW Levels and User Interface Support

For a larger view of Figure 2, click here,

Presenting the demo of DIRECT, we can use Figure 2 as a map to present some of the interfaces developed at each level of the hierarchy. We highlight:

• at the data level, the interface developed to manage the experiment submission by the participants, and how a logical structure is suggested by the system to support the user's actions;
• at the information level, the interface enabling assessors to do relevance assessments, and how information can be created by the user over existing data;
• at the knowledge level, how the system helps the creation of relation between different information on the visitor's metrics and statistics page, and some plan for future investigations to allow the user to create his own scenario of knowledge;
• finally, at the wisdom level, we point out some possible future directions that suggest how the DIRECT system may support scientific production.

The interfaces presented during the demo session are summarized in Figure 3, focusing on interfaces made for the visitors to allow them to retrieve the results of a campaign using the DOI system.

For a larger view of Figure 3, click here

Acknowledgements

The work reported has been partially supported by the TrebleCLEF Coordination Action, as part of the Seventh Framework Programme of the European Commission, Theme ICT-1-4-1 Digital libraries and technology-enhanced learning (Contract 215231).

References

[1] M. Agosti, G. M. Di Nunzio, and N. Ferro. The Importance of Scientific Data Curation for Evaluation Campaigns. In Proc. 1st Int. DELOS Conference, pages 157-166. LNCS 4877, Springer, 2007.

[2] G. M. Di Nunzio and N. Ferro. DIRECT: a System for Evaluating Information Access Components of Digital Libraries. In Proc. ECDL 2005, pages 483-484. LNCS 3652, Springer, 2005.

[3] M. Dussin and N. Ferro. Design of the User Interface of a Scientific Digital Library System for Large-Scale Evaluation Campaigns. In Second DELOS Conference - Working Notes. ISTI-CNR, 2007.

[4] M. Zeleny. Management Support Systems: Towards Integrated Knowledge Management. Human Systems Management, 7(1):59-70, 1987.