Abstract

This paper describes my thesis research, which is concerned with scholarly event information services. Responding to three different information needs, the project is grouped around three tracks. Track one covers the scholarly event ontology and the data harvesting approach, which provide a backbone to describe, collect, organize, mine, and analyze scholarly events and related data. Track two covers mining scholarly meaningful information, which is an attempt to model evaluation metrics for scholars and scholarly events and determine scholars' prominence and the qualities of scholarly events. Track three covers a social network based information platform, which includes social network analysis of scholarly event data, a topic based knowledge organization system, and a research trend analyzing system. This paper focuses on the progresses made so far and what actions will be taken for each track.

Keywords

Scholarly Event, Conference, Evaluation Metrics, Knowledge Organizing System, Research Trends

1. Introduction

Scholarly events, such as conferences, workshops, and seminars, are some of the major scholarly communication channels. They provide scholars with opportunities to publish up-to-date scientific research results, to get feedback from scientific communities, and to exchange research interests and ideas with each other. Scholarly events demonstrate current research trends and, as such, they are valuable information sources from which researchers can obtain knowledge concerning research directions and trends. According to [1], a scholarly event is conceptualized as "a managed occasion for community learning, supporting both knowledge sharing and knowledge building." "Conference participants constitute a temporary knowledge building community whose practice is the negotiation of interpretations, agendas, methodologies, facts and so on." I define 'scholarly event' here as a sequentially and spatially organized collection of scholars' interactions with the intention of disseminating and sharing knowledge, exchanging research ideas, and performing related activities. The terms 'conference' and 'event' will be used to refer to scholarly events.

For paper publication, conferences--which are innovation-laden--are quicker while journals-- which are intended to settle a knowledge base--are often very long, and median time lag between conference publication and journal publication is 2-4 years[2]. Other comparative studies on obsolescence of science also partially explain why scientists go to conferences[3-5]. Conferences provide timelier research outlets to disseminate research findings to the community. As such, almost all researchers attend scholarly events as part of their academic practice.

1.1 Scholarly Information Needs

We can illustrate scholars' information needs with regard to scholarly events. First, information of a simple nature: which conferences will be (or have been) held on which topics and when; where they are held; who will organize them; what the due dates are for submissions. Though this is basic information, it is not often the case that a scholar gets a full and exhaustive picture of scholarly events held in the world. The reason for this is due to the sheer volume of scholarly events held by various academic societies and organizations - no one specific information channel has been fairly successful at keeping track of ever-growing conferences and providing their information to scholars.

Secondly, more in-depth scholarly meaningful information is needed: who is regarded as eminent in a specific topic; what scholars are best suited for questions, consultations, and collaboration; which conference is the most prominent in a field of study. Researchers (especially beginners or newcomers in a given field) and research performance evaluators (funding authorities) may have special interests. These information needs might be met partially at a minimal level since almost all conference websites list the event general chairs, organizing committee, program committee chairs and members, invited speakers, and/or award winners. However, since meaningful information regarding scholarly events is not provided by existing libraries in a sufficient and integrated manner, searching for relevant information can be a cumbersome endeavor.

The third information need is subject-context: how the scientific and technical research trends are changing; which research topics are emerging or obsolete; what scholars and which conferences are associated with whom in which context (topics, domain, or organization); which conference is closest to my latest research. Existing information systems do not illuminate these traits sufficiently.

1.2 Research Goal and Action Plan

This study's fundamental premise is the assumption that if researchers are able to get more comprehensive, meaningful, and context-sensitive information regarding scholarly events, then better scholarly communication will result. As such the thesis project's overall objective is to satisfy scholars' information needs as mentioned above. The first goal of this research is to build artifacts for satisfying scholars' basic information needs by collecting and archiving scholarly event information. A prerequisite for this task is to define an ontology, which enables describing, collecting, indexing, and managing scholarly event data. Also necessary is a system built to harvest event data in a semi-automatic way. The second goal is to meet scholarly meaningful information needs by building a corresponding platform. Available information, however, is not obvious or explicitly expressed. As such the focus should be on mining scholarly meaningful information from harvested data. The main task is to construct an algorithm and a system to enable users to measure and determine scholars' performance and the qualities of events. A prerequisite for this task is to model evaluation metrics for scholars and scholarly events' prominence and verify them through statistical analysis. The last research goal is to provide a platform to satisfy users' subject-context information needs. Therefore, the project will push for an automatic system to analyze and synthesize scholarly event data using social network metrics with special attention to subject context. Two sections below explain what has been done so far and goals in future work.

2. Current Work

2.1 Designing Scholarly Event Ontology

As a partial response to the first research goal (simple information needs), the first action of this project is to develop a scholarly event ontology with logical constructs across multiple domains in detail, capturing as many different event types as possible. The ontology is named 'SEDE', or Scholarly Event Description Ontology. The purpose of SEDE is to provide a 'backbone' for:

1. Describing, collecting and sharing scholarly event data and information,
2. Enabling software agents to extract and mine scholarly meaningful information from events,
3. Allowing event organizers and libraries to organize and manage event information for alerting relevant users more effectively,
4. Evaluating the performance of researchers and the quality of events, and determining their prominence index through stochastic evaluation metrics, and
5. Analyzing social relationships between events, subjects, organizations, and participants within/across domains.

To achieve this, SEDE should be i) both in human and machine-readable format; ii) interchangeable and evolvable; iii) scalable to describe events; iv) multi-functional. In order to ensure its functionalities, SEDE (in RDF/OWL format) has been modeled according to the following principles. Firstly, the preferred course is to, whenever possible, utilize existing metadata and ontologies rather than reinventing the wheel. This draws on earlier metadata and ontologies: SWRC[6] for research, FOAF[7] for people, and SKOS[8] for event topics, all of which SEDE extends to create a more comprehensive and evolvable ontology. Secondly, from a survey and analysis of various event types such as conferences, workshops, symposia, seminars, and webinars (seminars on the Web) as many scholarly event-specific properties and attributes as possible are extracted. Figure 1. shows the overall structure of the current SEDE ontology.

Although the design of SEDE aims to be uncontroversial and consistent with the generally accepted view of scholarly events, it is inevitable that some of the design decisions may be debatable. Therefore SEDE will be open for anyone to contribute to its improvement and modification via the thesis project wiki (www.eventography.org).

Figure 1. Overall Structure of the SEDE ontology

Designing an event ontology is not a new concept. The ESWC2006 Conference ontology[9] uses WordNet[10], FOAF[7], Geo/WGS84_POS[11], and SWRC[6] as ground namespaces; it has Announcement, Document, Event, Menu, Role, Sponsorship, Organization, Person, ResearchTopic, and SpatialThing (e.g., place) as its top classes. Though this description provides a suitable level of description power, it only fits the case of the ESWC2006 conference because no event information has exactly the same structure and terminology. For example, in some events a session is grouped around a specific time interval, but in others, the same term 'session' can mean a track grouped around a specific research topic or theme. In summary, the ESWC2006 ontology is designed at a level of description for the Semantic Web conference that does not support common and multi-domain grounds. On the other hand, the SEDE project attempts to build a more axiomatic scholarly event description approach. IUgo[12] extends the ESWC2006 conference ontology. It is designed to present data for discovery and retrieval of materials that relate to particular event at the user interface level. To this end, it subclasses the ESWC ontology to model its proprietary major classes and properties, while inheriting both the structures and problems of the ESWC2006 ontology, rendering it as insufficient be understood by entities that publish and harvest scholarly event data, and infer scholarly meaningful information from them. Simply put, scalability doesn't appear to have been a major concern when designing IUgo. ITTALKS was built to provide access to information technology related talk announcements[13, 14]. ITTALKS utilizes DAML for its knowledge base representation, reasoning, and agent communication. The primary intention of the system is to demonstrate the power of Semantic Web concepts for integrating intelligent agents to facilitate user and agent interaction. Its scenario is to provide the inferred information that corresponds to the user's profile, personal schedule, location at the time of the event, distance, and usage patterns by integrating service agents with online information resources. Though it may be effective in a small domain (IT) utilizing ontologies for computer understandable conversation, it is not generally applicable to all types of scholarly event and domains. On the other hand, the SEDE project covers not only event announcements but also all related entities (e.g., calls for submissions, topics, committees, event-to-event relationships, program schedule, presentation material,) to infer scholarly meaningful information.

2.2 Modeling Scholars and Scholarly Events Evaluation Metrics

Responding to the second type of information needs, novel evaluation metrics for scholars' performance and event's quality are modeled. Regarding scholar evaluation metrics, it is known that citation-based measures have many limitations--shortage in coverage, incorrectness of data in citation index databases, self-citations, and even negative citations, though several approaches (h-index[15], g-index[16], Usage Impact Factor[17]) have been proposed to make them more complete. In this light, studies on editorial board membership of journal to gauge scholar's performance are informative[18-25]. Among them, Urbancic[22] shows that institutions ranked in the top 25 in editorial board membership are also highly ranked for publishing productivity. Baker and Righter[23] also state that scientific productivity and eminence in medical science may be important factors for being asked as an editor. Social network analysis of the editorial boards of medical informatics and bioinformatics journals suggests that editorial board members may have high values in other metrics[24]. The research conducted by Zsindely et al[25] showed that the total citations of editorial board members positively correlated with the impact factor of journals. In general, editorial membership-based measures are objective and potentially useful. In addition, the dissertation advising impact metrics are proposed to reflect mentoring as one important aspect of scholarly productivity[26]. These studies all indicate that a scholar's academic performance somehow associates with the core group (defined below) membership of scholarly events. It is especially obvious that participating in a scholarly event as a core group member is beneficial to researchers' career development. So, many scientists list up their event's core group membership in curriculum vitae.

There are several studies on scholarly event ranking. Tveit[27] proposes a method for ranking academic event-related search results (published news stories, products, images, user groups, geographic areas) using citations of program/editorial committee members of conferences, workshops or journals. In this method the scores of the academic events are determined by the average numbers of citations and papers of committee members. Along similar lines, another method mines the characteristics of program committee (number of PC members, number of publications and co-authorship of PC members, centrality of PC members) to discover prestigious conferences from collected data sets[28-30]. The finding is that a good-quality conference has a comparatively higher number of PC members, papers, and co-authorships than a low-quality conference.

As previous works illuminate, this thesis project, in searching for an alternative and complementary evaluation method, determines a scholar's eminence and a scholarly event's prominence based on the data indexed in the SEDE ontology's knowledge base. The following two subsections explain the current outputs of this effort to model evaluation metrics.

2.2.1 Scholar's Prominence Evaluation Metrics

In general, leadership of scholarly events typically is organized by senior researchers. In this approach, scholars' prominence measures are deduced from the total number of memberships for core groups in events, their types and weights. A 'core group' takes its member(s) from the list that follows: Organizing Committee Members, Organizing Committee Chairs, Program Committee members, Program Committee Chairs, Session Chairs, Award Winners, and Invited Speakers. Here, as an extended definition, is a proposal for a model to determine a scholar's prominence in a given subject field. Definition 1 (Subject Prominence Index of Scholar S) Let T be the core group, t be the core group type (), Kt be, for scholar S, the number of memberships for core group types in scholarly events, Wt be the weight assigned to core group type t, (N|F) be the total number of scholarly events in a given subject field F, and be the normalization factor to make consistent. Then, the Subject Prominence Index of Scholar S is:

Further, scholar S1 is said to be more prominent than scholar S2 if . Where, .

2.2.2 Scholarly Event's Prominence Evaluation Metrics

A Scholarly Event's prominence is determined by how many prominent scholars participated in the core group of a given scholarly event. See below for how to determine a Scholarly Event's Prominence in a specified subject field.

Definition 2 (Subject Prominence Index of Scholarly Event E) Let P(S) be the core group member's (Scholar S) Subject Prominence Index as in Definition 1, s be an instance of scholar S (), (C|F) be the total number of core group members for a scholarly event that belong to a specific subject field F, and be the normalization factor to make consistent. Then, the Subject Prominence Index of Scholarly Event E is:

Furthermore, Event E1 is said to be more prominent than Event E2 if . Where, .

2.2.3 Scholarly Event Series' Prominence Evaluation Metrics

A Scholarly Event Series' prominence is determined by how many prominent scholarly events there are in a given scholarly event series. See below for how to determine a Scholarly Event Series' Prominence in a provided subject field.

Definition 3 (Subject Prominence Index of Scholarly Event Series ) Let P(E) be the Subject Prominence of Scholarly Event E as in Definition 2, g be an instance of Scholarly Event Series G (), (G|F) be the total number of scholarly event instances belonging to Scholarly Event Series in a given subject field F, andbe the normalization factor to make consistent. Then, the Subject Prominence of Scholarly Event Series is:

Furthermore, Event Series is said to be more prominent than Event Series if . Where, .

It should be noted that the proposed evaluation method may not be fully sufficient to reflect scholarly events' qualities. As a possible improvement, such evaluation factors as paper acceptance rates can be added.

3. Future Work Plan

3.1 Analyzing the Core Group of Scholarly Event

As to the plausibility of the evaluation models in section 2.2, little evidence has been found regarding the implications of core group membership for scholarly events. Thus, it's necessary to investigate whether core group membership in scholarly events can be used as a prominence marker of scholars' research achievement. For this purpose sizable scholarly event data will be collected. (The research subject will be 'life sciences' because they have sheer volume of data sets). For each event, core group members' names, affiliations and their types (e.g., PC chair, invited speaker) will be identified. Then, for each scholar, citation measures will be monitored using a citation database. The hypothesis is that core group membership has positive correlations with better performance than non-core group's scholars in citation measures and scholars' citation measures vary with core group types. If it could be demonstrated that core group membership correlates with better performance, core group membership can be applied in scholarly events to assess a scholar's prominence. Consequently, these findings might have theoretical implications for scholarly event evaluation. It might be debatable to use citation measures because they have many pitfalls. One may also use alternative ways to measure a core group member's academic achievements by identifying his/her membership in other prominent groups such as the 1000 Biologists, National Academy of Science, and so on. However, despite the criticism, this method provides important evidence of academic productivity when combined with other indicators [31]. And in reality, the most useful services to date are citation databases. In addition, the intention of this research is not to compete with citation measurement, but rather to provide complementary evaluation metrics for scholars' performances and events' quality.

3.2 Designing and Implementing a Scholarly Event Data Harvester

To achieve three research goals, data must be aggregated and indexed according to the SEDE ontology. For efficient data collection, a harvester--comprised of a focused crawler and a data extractor--needs to be developed, which will automatically identify scholarly events' web sites and fetch their URLs, and then extract and add all found relevant information to the SEDE knowledge base for post-processing. Of course, some specific details regarding general information are often posted to scholarly event listing services[32-34]. Or short text 'call for papers' announcements are circulated via email or other forums. However, the bulk of detailed information for events is typically posted on the event websites themselves. Thus information such as the people making up committees, lists of topics for the event, and the event program schedule, presentation material are largely left out of listing and alerting services. For this reason, an event harvester should be devised.

The event focused crawler basically finds the links that are likely to be most relevant to the scholarly events. For this function, the basic approach is to build an 'event corpus' comprising of terms such as 'conference', 'workshop', 'call for paper', and so on. Visiting a specific web page, it compares words in the page and those in the event corpus. Then it finds a conference's main page and its sub pages by parsing URL directory separators, '/'. Next, it again compares words with the corpus for each page to see if that sub page indeed belongs to the event page. For obsolete URLs, proceedings archives such as DBLP[35] and ACM[36] provide complementary resources. They provide additional information such as paper acceptance statistics as well. The crawler will exploit and leverage them as appropriate. Regarding automatic extraction of information, several studies have been reported[37-40]. McCallum[38] employed information extraction to extract bibliographic information such as authors, title, journal and publication date from scientific articles, in order to build a computer science research paper directory automatically by using machine learning techniques. Shneider[40] describes an application of information extraction for call for papers. He employed Conditional Random Fields for the task of extracting key information such as conference names, titles, dates, locations and submission deadlines from CFPs. And he demonstrated the effectiveness of layout features in the absence of grammatical structure, which is typical in CFPs. However, the results show that the performance was at best 72.7%. Although automatic techniques clearly have the advantage that they can be run on large text corpora with little human effort, automatic information extraction using natural language processing techniques have not seen very good results.

A significant problem in scholarly event websites is that their style and format come in different forms. There is no standard for the posting of this varied information; conference websites will often post this information on different kinds of pages, and in different contexts (different arrangements on the page and different typographical distinctions). This is coupled with the fact that such information as dates, topics, and lists of people, are not presented in a standardized format, nor marked up in any way. Rather, the information often has only typographical clues as to their significance. Furthermore, all pages are designed in different ways and, coming from different countries of origin, tend to list information and use vocabulary differently. Due to the limitations of fully automatic information extraction techniques and the heterogeneous nature of event web pages, this project's initial strategy is to make use of a more low-tech method of data extraction, which utilizes manually defined patterns of text content and HTML formatting based on general conventions for listing data in human-readable formats on the web. Later, machine learning algorithms will be investigated, though machine learning approaches do not guarantee perfect automatic data extraction. As such, the current strategy is to use a semi-automatic approach.

3.3 Building Social Network Based Information Platform

To meet subject related information needs the thesis project has a plan to design and implement a social network-based information platform by exploiting SEDE's instances. The following are the deliverables for this action plan:

1. Event Topics Distance/Similarity Calculation System: In SEDE, event topics and session (track) topics are encoded in SKOS[8] standard; it aims to automatically measures and determines subject distances and similarities among events and event series,
2. Research Trend Analyzing System: Enables users to analyze research trends, overview continuously changing research topics, check hot and obsolete topics, and observe conversions in practice among various academic disciplines,
3. KOS Learning System: Aims to build alternative knowledge organization systems (KOS) for arbitrarily-specified subjects by the user (i.e., cell biology, medicine, mathematics). In so doing this, the project will explore links to existing external KOSs (i.e., LexGrid[41], MeSH) to construct Knowledge Grid as projected in [42, 43].
4. Academic Entities' Networking System: Delineates academic entities' social relationships. To this end, researchers' and research groups' co-participations in a given set of events (series) will be analyzed to connect and group each other. Social Network Analysis metrics will be calculated to determine their standing in their communities.

4. Research Implications

The proposed information platform can enhance library services by providing users with more comprehensive and organized scholarly information. Particularly the SEDE ontology will provide a background for eventography, which would be an alternative field of bibliography or scientography[44]. In addition, deliverables from proposed evaluation metrics (an alternative scholar's performance measuring method) will provide a background for eventometrics, which will be a sub field of scientometrics. This research embodies the author's hope to contribute to efficient scholarly communication.

5. Acknowledgement

This study was supported by the Korean Research Foundation Grant funded by the Korean Government (Ministry of Education, Science and Technology), KRF-2006-511-H00001.

The work described here is joint work with my advisor, Hong-Gee Kim, at Seoul National University. Charles Borchert provided expert assistance with the information extraction.

6. References

[1]	N. Jacobs and A. McFarlane, "Conferences as Learning Communities: Some Early Lessons in Using 'Back-channel' Technologies at an Academic Conference - Distributed Intelligence or Divided Attention?" Journal of Computer Assisted Learning, vol. 21, no. 5, pp. 317-329, 2005.
[2]	M. Montesi and J.M. Owen, "From Conference to Journal Publication: How Conference Papers in Software Engineering are Extended for Publication in Journals," Journal of the American Society for Information Science and Technology, vol. 59, no. 5, pp. 816-829, 2008.
[3]	M. Luwel and H.F. Moed, "Publication Delays in the Science Field and Their Relationship to The Ageing of Scientific Literature," Scientometrics, vol. 41, no.1-2, pp. 29-40, Jan. 1998.
[4]	W. Glanzel and U. Schoepflin, "A Bibliometric Study on Ageing and Reception Processes of Scientific Literature," Journal of Information Science, vol. 21, no. 1, pp. 37-53, 1995.
[5]	V. Lariviere, E. Archambault, and Y. Gingras, "Long-term Variations in the Aging of Scientific Literature: From Exponential Growth to Steady-state Science(1900-2004)," Journal of the American Society for Information Science and Technology, vol. 59, no. 2, pp. 288-296, 2007.
[6]	Y. Sure, S. Bloehdorn, P. Haase, J. Hartmann, and D. Oberle, "The SWRC Ontology - Semantic Web for Research," in Proceedings of the 12th Portuguese Conference on Artificial Intelligence (EPIA 2005), Vol. 3803. Springer, Covilha, Portugal, pp. 218-231, 2005.
[7]	D. Brickley and L. Miller.: "Friend of a Friend (FOAF) Vocabulary," 2006. [Online]. Available at http://xmlns.com/foaf/spec/
[8]	A. Miles and D. Brickley, Eds., "SKOS Core Vocabulary Specification. W3C Working Draft 2." November 2005. [Online]. Available: http://www.w3.org/TR/2005/WD-swbp-skos-core-spec-20051102/
[9]	T. Heath, "ESWC2006 Conference ontology," 2006. [Online]. Available: http://www.eswc2006.org/technologies/ontology-content/2006-09-21.html
[10]	G.A. Miller, "WordNet: A Lexical Database for English," Communications of the ACM, vol. 38, no. 1, pp. 39-41, 1995.
[11]	D. Brickley, "Geo," 2003. [Online]. Available: http://www.w3.org/2003/01/geo/
[12]	N. Rogers, M. Jones, P. Shabajee, and J. Tredgold, "Iugo Deliverable Report: Project Final Report," Institute for Learning and Research Technology. Available: http://iugo.ilrt.bris.ac.uk/iugo/docs/iugo_final_report.pdf
[13]	F. Perich, R.S. Cost, T. Finin, A. Joshi, Y. Peng, C. Nicholas, H. Chen, L. Kagal, Y. Zou, and S. Tolia, "ITTALKS: An Application of Agents in the Semantic Web," The Workshop on Engineering Societies in the Agents' World, Prague, Czech Republic. Lecture notes in computer science , vol. 2203, pp. 175-193, 2001.
[14]	R.S. Cost, T. Finin, A. Joshi, Y. Peng, C. Nicholas, I. Soboroff, H. Chen, L. Kagal, F. Perich, Y. Zou, and S. Tolia, "ITtalks: A Case Study in the Semantic Web and DAML+OIL," IEEE Intelligent Systems, vol. 17, no. 1, pp. 40-47, Jan/Feb. 2002.
[15]	J.E. Hirsch, "An index to quantify an individual's scientific research output," in Proceedings of the National Academy of Sciences, vol. 102, pp. 16569-572, 2005.
[16]	L. Egghe, "An Improvement of the H-index: the G-index," ISSI Newsletter, pp. 8-9, 2006.
[17]	J. Bollen and H.V. de Sompel, "Usage Impact Factor: The Effects of Sample Characteristics on Usage-based Impact Metrics," Journal of the American Society for Information Science and Technology, vol. 59, no. 1, pp. 136-149, Jan. 2008.
[18]	K.C. Chan and R.C.W.Fok, "Membership On Editorial Boards and Finance Department Rankings," The Journal of Financial Research, vol. 26, no. 3, pp. 405-420, 2003.
[19]	J.D. Gibbons, "U.S. Institutional Representation on Editorial Boards of U.S," The American Statistician, vol. 44, no. 3, , pp. 210-213, 1990.
[20]	K.C. Chan, H.-G. Fung, and P. Lai, "Membership of Editorial Boards and Rankings of Schools with International Business Orientation," Journal of International Business Studies, vol. 36, pp. 452-469, 2005.
[21]	J.D. Gibbons, and M. Fish, "Rankings of Economics Faculties and Representation on Editorial Boards of Top Journals," The Journal of Economic Education, vol. 22, no. 4, pp. 361-372, 1991.
[22]	F.R. Urbancic, "Editorial Board Representation: An Alternative Method for Ranking Real Estate Programs," Journal of Real Estate Practice and Education, vol. 7, pp. 53-63, 2004.
[23]	P. Bakker and H. Rigter, "Editors of medical journals: Who and from where," Scientometrics, vol. 7, no. 1-2, pp. 11-22, January 1985.
[24]	B. Malin and K. Carley, "A Longitudinal Social Network Analysis of the Editorial Boards of Medical Informatics and Bioinformatics Journals," Journal of the American Medical Informatics Association, vol. 14, no. 3, pp. 340-348, 2007.
[25]	S. Zsindely, A. Schubert, and T. Braun, "Citation Patterns of Editorial Gatekeepers in International Chemistry Journals," Scientometrics, vol. 4, no. 1, pp.69-76, 1982.
[26]	G. Marchionini, P. Solomon, C. Davis, and T. Russell, "Information and Library Science MPACT: A Preliminary Analysis," Library & Information Science Research, vol. 28, no. 4, pp. 480-500, 2006.
[27]	A. Tveit, "Ranking Academic Event Related Search Results using Event Member Metrics," World Intellectual Property Organization. [Online]. Available: http://www.wipo.int/pctdb/en/wo.jsp?WO=2007070844&IA=US2006062074&DISPLAY=STATUS
[28]	Z. Zhuang, E. Elmacioglu, D. Lee, and C.L. Giles, "Measuring Conference Quality by Mining Program Committee Characteristics," in Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, New York, NY: ACM, 2007, pp. 225-234.
[29]	S. Yan and D. Lee, "Toward Alternative Measures for Ranking Venues: a Case of Database Research Community," in Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, New York, NY: ACM, pp. 235-244, 2007.
[30]	E. Elmacioglu and D. Lee, "Oracle, Where Shall I Submit My Papers," Forthcoming in Communications of the ACM. Available: http://nike.psu.edu/publications/cacm07-oracle.pdf
[31]	J. Lundberg, "Bibliometrics as a Research Assessment Tool-impact Beyond the Impact Factor," Ph. D. dissertation, Karolinska Institutet, 2006. Available: http://diss.kib.ki.se/2006/91-7140-965-3/thesis.pdf
[32]	EventSeer. Availble: http://eventseer.net/
[33]	All CONFERENCES.COM. Available: http://www.allconferences.com/
[34]	Conference Alerts. Available:http://www.conferencealerts.com/
[35]	M. Ley, "The DBLP Computer Science Bibliography." Available: http://dblp.uni-trier.de/
[36]	ACM Digital Library. Available: http://portal.acm.org/dl.cfm
[37]	F. Peng and A. McCallum, "Accurate Information Extraction from Research Papers Using Conditional Random Fields,"In Proceedings of the Human Language Technology Conference, Boston, MA, May 2-7, 2004,
[38]	A. McCallum, K. Nigam, J. Rennie, and K. Seymore, "A machine learning approach to building domain-specific search engines," in Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (IJCAI-99), San Francisco, CA: Morgan Kaufmann Publishers Inc., 1999, pp. 662-667.
[39]	K.-M. Schneider, "Using Information Extraction to Build a Directory of Conference Announcements." In Fifth International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2004), vol. 2945, Seoul, Korea, pp. 521-532, 2004.
[40]	K.-M. Schneider, "Information Extraction from Calls for Papers with Conditional Random Fields and Layout Features." In 16th Irish Conference on Artificial Intelligence and Cognitive Science (AICS 05), Portstewart, Northern Ireland, pp. 267-276, 2005.
[41]	T.M. Johnson, H.R. Solbrig, D.C. Armbrust, and C.G. Chute, "LexGrid Editor: Terminology Authoring for the Lexical Grid," AMIA Annu Symp Proc., 2005.
[42]	H. Zhuge, The Knowledge Grid. World Scientific, 2004.
[43]	M. Cannataro and D. Talia, "The Knowledge Grid," Communications of the ACM, vol.46, no.1, pp. 89-93, Jan. 2003.
[44]	E. Garfield, Essays of an Information Scientist: Towards Scientography. Philadelphia: ISI Press, 1986.

Back to Top