IEEE TCDL Bulletin
 
space space

TCDL Bulletin
Fall 2007
Volume 3   Issue 3

 

Studying Academic Lawyers' Information Seeking to Inform the Design of Digital Law Libraries

Stephann Makri
UCL Interaction Centre, 31-32 Alfred Place, London WC1E 7DP, UK
<S.Makri@ucl.ac.uk>

 

Abstract

We report findings from the initial phase of our study on legal information seeking, which comprised a series of semi-structured interviews and naturalistic observations of academic law students and staff looking for electronic legal information. This study has the long-term aim of informing the design of digital law libraries. Participants found it difficult to use digital law libraries – such difficulty arising from poor knowledge of the digital library system rather than from poor general electronic research skills. Hazy and faulty system-related knowledge were rife, suggesting the need for academic lawyers to understand more about the digital library systems that they use (within-systems knowledge). These lawyers chose to rely primarily on one major digital law library for legal information seeking. Their preference was often based upon vague or flawed rationale and suggests the need for academic lawyers to appreciate the situations in which different electronic resources might be useful (between-systems knowledge).

 

Introduction and motivation for research

Our work is motivated by the desire to support digital library users in 'getting to grips' with electronic resources. More specifically we are motivated by the wish to support users in understanding how to use, and in which situations it is appropriate to use, particular digital libraries and other electronic resources.

This work focuses on lawyers as a specific category of user: a user population that have been traditionally regarded as having poor research skills [5]. Electronic research skills are no exception: Howland and Lewis [7] surveyed U.S. law firm librarians to examine the quality and extent of the electronic legal research skills of summer clerks and first-year associates. They found that these graduates were unable to efficiently or effectively research issues that appear routinely in actual legal cases and concluded that they were not efficient or cost-effective users of LexisNexis and Westlaw (the two biggest digital law libraries in terms of case, legislation and journal coverage). This was despite all of the students having received some training on how to use the libraries while in law school.

Digital libraries have traditionally been regarded as difficult to use [4] and based on our Contextual Inquiry interviews and observations with academic lawyers, digital law libraries such as LexisNexis Professional and Westlaw are no exception. We believe that this difficulty of use contributes to the problems that lawyers face with electronic legal research. Furthermore, we argue that developing better research skills goes hand-in-hand with developing an understanding of the electronic environments in which these skills must be practiced. Our current work is focused on gaining a better understanding of legal academics' and professionals' information-seeking behaviour when using existing electronic resources. This understanding will then be used to inform the design of user-centred information-seeking support for digital law libraries (and potentially the design of the libraries themselves).

Related Work

There are very few studies that focus on legal information seeking and even fewer user-centred studies on legal research with a focus on digital law libraries or other electronic legal research tools. In this section, we briefly review a handful of studies that focus on the usage and perceptions of digital libraries; in a transaction log-based study, Yuan [13] monitored the LexisNexis Quicklaw searches of a group of law students over the period of a year. Yuan examined several aspects of their searching behaviour, including the increase of their command and feature repertoires, their change in language usage, increase of search speed and change of learning approaches. Yuan found that experience did not result in searchers making fewer errors or being able to recover from more errors. Yuan also found that although participants with higher levels of Quicklaw experience used a greater variety of commands and features than those with lower levels of experience, some commands remained rarely or never used. Despite this, however, participants were able to accomplish many tasks by knowing a core set of commands and features.

In order to identify the 'fit' of the same digital library to various organisations, Elliott and Kling [6] conducted a qualitative study into digital library usage. They interviewed forty-six legal professionals based in three courtrooms in the same county within the California Superior Courts System. Each of the three courtrooms provided access to different degrees of technology to support legal research. Regarding attitudes to computer usage, Elliott and Kling found there to be two camps of legal professionals; one highly enthusiastic towards computer usage and another who view computer usage with derision. The deputy law librarian for the District Attorneys Office pointed out that lawyers will go to "what they are comfortable with." This is apparently despite the fact that all of the lawyers in the study had access to periodic training classes for all digital library systems. Indeed, regarding electronic information seeking training attitudes, the authors found that lawyers prefer one-to-one assistance with digital libraries rather than group training and made recommendations for increased tailored training assistance. They found that most lawyers were lost when attending formal classes (or simply did not want to allocate time to take such classes).

Andrews [1] examined perceptions of LexisNexis (and their perceptions of the training that they received) by administering questionnaires and structured interviews to eighteen legal professionals and law librarians. Andrews asked interviewees about the usability of the Lexis system as it stood in 1994 (which, it should be noted, has changed significantly since then). The user interface was regarded as a significant barrier to usage (although, interestingly, the author highlights that suggestions to make the interface more in-line with standard PC/Mac interfaces seem to be based on legal professionals comparing Lexis to software packages of the time rather than to other online systems).

Sutton [11] examined the legal literature in order to explore how lawyers construct 'mental models' of the law. Sutton notes that these mental models evolve through iterative interaction between lawyers and the corpus of law as well as between lawyers and the legal system. Sutton suggests that Lexis and Westlaw can be enhanced in order to support users in building a 'mental model' of the law. Unsurprisingly, these enhancements are user-focused (probably because Sutton's study was also 'user-focused' in the sense that his motivation appears to be helping lawyers to better understand their work domain of law). Sutton argues that "Lexis and Westlaw should embrace a dynamic behavioural model of system users and assist mental model building at all points along the knowledge continuum from base-level modeling through context-sensitive exploration to model disambiguation." He suggests that support should be provided to guide the user "from the most general topical outline through treatise/commentary responses to primary sources themselves, based on a legal schema of increasing complexity." In a similar vein to Sutton's argument that digital law libraries should support users in building a mental model of the law, we argue that digital law libraries (and digital libraries in general) should also support users in acquiring and refining a mental model of the system. The value of supporting users in this endeavour is highlighted by Borgman [3].

Methodology

The initial phase of our study comprised a series of semi-structured interviews and naturalistic observations of twenty-one academic law students, ranging from first year undergraduates to final year doctoral students. This spread of academic participants was recruited to provide complementary theoretical perspectives at each stage of UK university-level academic legal research. All of the students were studying at a large London university for an LLB (4 first year, 3 second year and 2 final year students), LLM (8 students), or PhD (2 students). In addition two participants, who were studying a vocational Legal Practice Course elsewhere, were included in the study in order to complete the theoretical picture. We also interviewed six members of academic staff working at the same academic institution. These members of staff included one Senior Research Fellow, two Lecturers, two Senior Lecturers and one Professor of Commercial Law.

Participants were informed that the study would focus on how they look for legal information as part of their work, and interviews and observations were conducted based on the Contextual Inquiry approach (see [2]). Interviews began with a set of introductionary questions, focusing on what stage the lawyer was at in their academic career, the nature of legal research involved, the electronic resources used and how they choose when to use a particular resource. These introductionary questions were followed by a naturalistic observation where participants were asked to find some electronic legal information that they currently need as part of their academic work. If a participant was unable to think of a pressing research need, they were directed to think back to a recent time when they needed to look for legal information and, if possible, to show the investigator how they went about finding that information.

Participants were asked to think aloud whilst using the computer, explaining what they were doing as they were doing it and were told to verbalise any thoughts going through their heads. Whilst they were using the electronic resources (predominantly the digital library resources LexisNexis Professional and Westlaw), the researcher asked seemingly innocuous but probing questions designed to uncover details about their knowledge of the digital library system and details about the information that they were expecting to find. These questions took the form of opportunistic 'how,' 'what' and 'why' questions. Interviews were transcribed and analysed using Strauss and Corbin's Grounded Theory approach [10].

Findings

In the first phase of our study, this group of academic lawyers found it difficult to find the information that they were looking for when using digital law libraries such as LexisNexis Professional and Westlaw. We found that much of this difficulty arose from poor knowledge of the digital library system itself rather than from poor research skills in general. We identified three broad types of system and information source-centred knowledge that academic lawyers held: awareness knowledge (which resources exist to help locate certain materials), access knowledge (whether they have access to certain materials and, if they do, how they might go about doing so) and usage knowledge (how to use the electronic resource). Faulty knowledge and misconceptions were rife across all types of knowledge that were identified.

To exemplify faulty awareness knowledge, a Professor of Commercial Law suggested that it is not possible to perform subject searches on Westlaw (i.e., to search for legal materials related to a particular subject such as 'jurisprudence'). He assumed that only searches where the precise details about the material required are known (such as when searching for a legal case such as Donoghue v Stevenson) could be performed using Westlaw:

"It just doesn't help me... I can't recall now, but I'm not prepared to analyse the problem too much... whether it even has the capability of finding anything that's reasonably discoverable on a particular subject. I'm not sure, so I go straight into Google... punch in the phrase, see what comes out. You can do a word search once you're in a case, but if you wanted to find all the cases on a particular subject, I don't think you can use Westlaw for that. That's what I would really want." – R6

As an example of faulty access knowledge, one Senior Research Fellow incorrectly believed that the company behind Westlaw had developed both a U.S. and U.K. version of the digital law library. He believed that U.S. materials could best be found by logging in via www.westlaw.com rather than through the .co.uk URL and commented that doing so would sometimes allow him to access non-UK legal materials by 'bypassing' the university's subscription to Westlaw:

"This is Westlaw America and sometimes it lets you login now and sometimes it doesn't. So sometimes we could login and bypass our subscription and get access to whatever else is going on in the world. It seems to recognise [that we're in the UK] because it's got 'Westlaw UK' [on the menu bar]." – R3

In reality, Westlaw produce only one version of their digital law library and this is the same irrespective of the URL that the user logs in via. This misconception, however, highlights the fact that finding non-UK materials is difficult when using Westlaw; Senior Research Fellow R3 remained unaware that only one version of Westlaw exists and that non-UK materials are available from this version (albeit through a rather complicated interaction path).

The broader category of usage knowledge was found to include several sub-categories. These included knowledge about:

  • The coverage of the digital library (what types of materials the system can be used to find and which parts of the system can be used to find them).
  • The content/structure of materials within the library.
  • The authority of the digital libraries and materials within the library.
  • How to search the digital library.

Hazy and faulty knowledge surrounding the coverage of the library often included not knowing where within the system to go in order to find certain types of material or to perform a certain type of search. For example, many academic lawyers found the segmented fields in Westlaw and LexisNexis Professional to be confusing. Coverage knowledge was also hazy surrounding the materials within the library, for example some academic lawyers raised their uncertainty about why Westlaw presented the full-text of some legal journal articles but only brief details about others.

Regarding the content and structure of materials within the library, knowledge was far less hazy. As Professor of Commercial Law R6 commented, academic lawyers are likely to soon become familiar with the 'anatomy' of legal materials such as cases. One first year undergraduate student, for example, explained that a list of keywords is available in the headnote of a legal case and that the case can be retrieved by including one of those keywords as a search term within the 'headnote' segmented field search box in LexisNexis Professional:

"On the top of them [case reports] you've got a list of keywords and you want to get a particular keyword to come up." – S5

Knowledge about the authority of materials was also less hazy than knowledge about the coverage and structure of the library. Academic lawyers were generally aware that authenticated resources such as digital law libraries were likely to hold more authoritative content and that a Google search, albeit useful for gaining an overview of a legal subject or finding information relating to current affairs, would not be appropriate to find authoritative legal materials. Knowledge about authority also extended to documents within digital law libraries. For example one PhD student chose only to read the full-text of journal articles that he considered to be from 'serious journals':

"So far it's very disappointing, because we can't find anything interesting about 'utmost good faith' in insurance from a serious journal." – S4
"How do you know if it's from a serious journal?" (Asked by investigator).
"Well, I know all the journals from experience that are not serious... like Law Gazette. Basically, they're just news... not academic... they're very short." – S4

Finally, regarding knowledge about how to search the system, academic lawyers' knowledge was varied. One or two academic lawyers had very poor search-related knowledge. For example, one postgraduate student did not retrieve any results as part of her search and proceeded to use Westlaw's 'natural language search' facility using traditional search terms and Boolean connectors. Many lawyers commented on the need to be 'exact' or 'precise' when searching digital law libraries such as LexisNexis Professional or Westlaw. For example, one undergraduate student highlighted the need to type in search terms exactly as they are stored in the database in order to match the terms to any resultant documents:

"Although in Westlaw it appears as though it [the document] doesn't exist, it does actually exist although you've just not typed in exactly what it wants you to type in. You have to type it in exactly how it's specified there. I just know this now from practicing a bit, but I guess if you didn't know it it's a bit of a pain because you get a bit stuck as to what to do and you think 'oh well, the resources aren't there so how am I meant to find it?'" – S13

A small minority of academic lawyers had rather sophisticated search knowledge. A PhD student, for example, highlighted the existence of several Boolean connectors in both Westlaw and LexisNexis Professional. However, none of the academic lawyers in our study demonstrated the use of any search syntax above the use of inverted commas to denote phrases and the connectors 'AND' and 'OR.' The PhD student explains that this is partly because he is 'not very good' at constructing advanced Boolean queries and partly because he would prefer to manually filter documents rather than use advanced queries to do so:

"I can set rules, but I'm not very good at that. Partly because I don't really want to rely on electronic systems to filter for me. So I know you can, for example, define how many times 'Carter' and 'Boehm' [the search terms the participant had recently been using] have to show up or how close they should be together." – S4.

To conclude, we have presented evidence that hazy and faulty knowledge are rife across the categories of knowledge that were identified in our study and across the entire academic spectrum. This suggests the need for academic lawyers to understand more about the digital library systems that they use (i.e., to acquire or strengthen their 'within-systems knowledge').

Discussion of Findings in Relation to Previous Work

Perhaps more worrying than the widespread faulty and hazy system-related knowledge, and in line with the findings of Yuan and of Elliott and Kling, was that academic lawyers do not delve beyond the basics of digital library systems and were often unwilling to go to training classes on how to use digital law libraries despite being aware that these classes had been available to them. Whilst this may be understandable, since they are not approaching the system with the intention of learning how the system works but with the intention of satisfying a pressing research need, understanding 'beyond the basics' of digital libraries is essential for lawyers as they move up the academic ladder.

In line with Elliott and Kling's findings that lawyers will go to the digital resources 'they are comfortable with,' we found that although this group of academic lawyers often used several electronic resources in a complementary fashion to conduct legal research, they often chose to rely primarily on one of either LexisNexis or Westlaw for conducting their legal research. This reliance on either Lexis or Westlaw was often independent of the information task at hand and their preference was often based upon vague rationale for always choosing one digital library over the other:

"Sometimes if I don't find something on Westlaw I will look into LexisNexis [Professional]...because Westlaw is more convenient... it's more easy to surf... and for some reason which I can't really explain, I always look first on Westlaw and then if I can't find it I'll go onto LexisNexis. It's probably because I'm more used to using Westlaw so I would only resort to LexisNexis in case of emergency." – S6

Sometimes this rationale for choosing one digital law library system over another was not vague, but flawed. For example one first year undergraduate student believed that it was not possible to copy and paste from the full-text of a case on Westlaw into a Microsoft Word document and therefore expressed a preference for using LexisNexis Professional. In fact, it is possible to copy to the clipboard from both digital library platforms:

"If it's a case that's on both [LexisNexis Professional and Westlaw] then it's probably better to get it on LexisNexis because then you can copy and paste the case into a Word document whereas with Westlaw it's more complicated... you have to e-mail it to yourself and then cut and paste and print." – S5

Indeed, much of academic lawyers' rationale for choice of system referred to 'the interface' and may be symptomatic of problems or barriers caused by usability (as suggested by Andrews [1]). It is more difficult, however, to ascertain where flawed rationale for choosing systems stems from as it not only might be influenced by usability barriers, but also by other factors based on users perceptions as a result of system use. Similarly it is clear that poor knowledge of individual systems plays a part in the creation of incorrect assumptions. However, so might usability issues or a host of other inter-dependent factors such as knowledge of the law and knowledge of the legal research process.

A potentially more serious finding regarding the use of multiple electronic resources was that there was often an incomplete, incorrect or sometimes outright lack of awareness of concrete differences between digital law libraries:

"I think on LexisNexis [Professional] I can simply type in the citation of the case but I think in Westlaw sometimes I have to type in the name of a case... I think they're basically similar because if you find the same case on the two sites, the text will be the same." – S1

Academic lawyers' lack of knowledge of the similarities and differences between individual digital law libraries might well play a part in their incorrect assumptions about the way that individual systems work. This finding also suggests the need for academic lawyers to gain an understanding of the similarities and differences between digital law libraries in order to appreciate the situations in which different electronic resources might be useful ('between-systems knowledge').

Overall, these findings highlight the need for digital law libraries to provide system-related knowledge support. Just as Sutton [11] argues that digital law libraries should be designed to allow lawyers to form a 'mental model' of the work domain, we believe that digital law libraries should also support users in forming a mental model of the systems that they use to find information; information that can then be used to support users' models of the work domain. Indeed, we believe that 'getting to grips' with electronic legal resources, particularly digital law libraries, might be just as important as getting to grips with the legal domain itself.

Proposed Future Work and Issues for Discussion

Aside from our planned work with practicing lawyers, we have recently conducted complementary interviews with other stakeholders. These interviews promise to be a useful theoretical basis from which to triangulate and hence better validate our findings. We have interviewed three Law Librarians working at the academic and vocational institutions that our academic participants attended, two law librarians working at a large London law library and one member of training staff from each of the two major digital law library providers (LexisNexis Butterworths and Westlaw UK). All of these stakeholders provide an element of support to academic lawyers conducting legal research and hence have been demonstrated to hold perceptions and opinions on the difficulties that they face and why they behave in the way that they do. In addition, and particularly in the case of law librarians, we have found that stakeholders share a similar motivation to ours and hence have useful opinions that might guide our future design work. Indeed, we plan to feed our observations on academic lawyers' behavior back to major stakeholders in the form of a representative set of scenarios (see Rosson & Carroll [9]) and use these to drive our future design work.

We are encouraged by the first phase of our study because the data obtained is rich and has yielded some interesting findings. However, the challenge now remains for us to use these findings to inform the design of digital law libraries. Through an analysis of the categories of electronic resource and information source-related knowledge that lawyers have about the digital law library systems that they use, our study has provided insight into the content of the knowledge that must somehow be implemented into the design of system to support legal information-seeking. However, a significant challenge remains with regard to ascertaining exactly how this content might feed in to the design of a prototype system and what form such a system might take. There are two potential (and partially overlapping) approaches that we might consider:

  1. The approach of embedding knowledge relating to electronic resources and information sources into the design of digital law library systems. This approach involves better designing digital law libraries in order to support the development of users' mental models of the system (and hence supporting users to acquire and strengthen positive knowledge related to the categories of knowledge identified in this study and to address and/or avoid the creation of faulty or hazy knowledge related to these categories). An example of a system exemplifying an aspect of this approach (understanding the effects of the search syntax entered on the search results returned) has been implemented in Muramatsu & Pratt's 'transparent query system' [8]. This system includes lightweight visualisations of key search term transformations. For example, the system displays common words in the current search that have been excluded by the search with a line through them, in a frame above the results page.
  2. The approach of integrating a form of electronic information-seeking support into the design of digital law library systems. Such support would aim to make explicit knowledge relating to electronic resources and information sources that might otherwise impede information-seeking. An example of this approach is exemplified by Stelmaszewska et al. [12], who designed a context-relevant integrated help system for digital libraries that provides feedback to users on the effects of their search results with reference to the search terms that they have input. For example, most searches that result in several thousand hits will result in a tip that suggests to the user that they narrow their search.

Both approaches are aimed at supporting the development of users' mental models of the digital law library system and may also potentially support the refinement of their mental models of legal information seeking in general. In addition, we hypothesise that both approaches can also help users to acquire and strengthen positive knowledge and avoid/address faulty and hazy knowledge. However whilst these approaches are subtly different, they share the common ethos of making the process of understanding digital libraries more transparent to users. We therefore hope that by following either of these approaches, we will be able to make digital libraries easier to 'get to grips' with. Further research with our user group of lawyers is, however, required in order to provide us with an indication of which design direction we might take in order to produce a system that is truly usable, useful and used.

Digital libraries have the potential to be both barriers and gateways to knowledge. Therefore despite the potential challenges we have discussed, we believe that helping users to get to grips with digital libraries is of the utmost importance. We would like to see a future where digital libraries are designed with the user in mind, where digital libraries are designed not only to help users progress on their information journey, but to support users in getting to grips with the system along the way. We believe that our work aims to take a small but nevertheless important step towards such a future.

References

1. Andrews, C.J. (1993). User Perceptions of CALR. Unpublished MSc thesis, City University, London, UK.

2. Beyer, H. & Holtzblatt, K. (1998). Contextual Design. Morgan Kauffman, San Francisco.

3. Borgman, C. (1988). The User's Mental Model of an Information Retrieval System. International Journal of Man-Machine Studies 24(1), pp. 47-64.

4. Borgman, C. (2000). From Gutenberg to the Global Information Infrastructure. Cambridge, MA. MIT Press.

5. Callister, P.D. (2003). Beyond Training: Law Librarianship's Quest for the Pedagogy of Legal Research Education. Law Library Journal, 95(1), pp. 7-45.

6. Elliott, M. & Kling, R. (1997). Organizational Usability of Digital Libraries: Case Study of Legal Research in Civil and Criminal Courts. JASIS 48(11), pp. 1023-1035.

7. Howland, J.S. & Lewis, N.J. (1990). The Effectiveness of Law School Legal Research Training Programs. Journal of Legal Education, 40, pp. 381-391.

8. Muramatsu, J. & Pratt, W. (2001). Transparent Queries: Investigating Users' Mental Models of Search Engines. In Proceedings of the 24th annual international ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 217-224. New Orleans, Louisiana, USA. ACM Press.

9. Rosson, M. & Carroll, J. (2002). Scenario-based Design. In Jacko, J. & Sears, A. (Eds.), The Human-computer Interaction Handbook. Lawrence Erlbaum, 2002, pp. 1032-1050.

10. Strauss A. & Corbin J. (1998). Basics of Qualitative Research. Sage. London, UK.

11. Sutton, S.A. (1994). The Role of Attorney Mental Models of Law in Case Relevance Determinations: An Exploratory Analysis. JASIS 45(3), pp. 186-200.

12. Stelmaszewska, H., Blandford, A. & Buchanan, G. (2005). Designing to change users' information seeking behaviour: a case study. In S. Chen & G. Magoulas (Eds.) Adaptable and Adaptive Hypermedia Systems. 1-18. London: Information Science Publishing.

13. Yuan, W. (1997). End-user Searching Behavior in Information Retrieval: A Longitudinal Study. JASIST. 48(3), pp. 218-234.

 

© Copyright 2007 Stephann Makri

Top | Contents
Previous Article
Next Article
Home | E-mail the Editor