ABSTRACT

Wathen & Burkell [25] provide a motivation for orienting a research agenda that examines the effects of messenger/medium on credibility assessment: "[a]ccording to persuasion theories, a user who has a high need for information will overlook weak peripheral cues. If the information-seeking episode is more casual, surface characteristics will have more influence" (p. 141). These claims (i.e., that the medium impacts credibility judgments, and that credibility judgments based on surface characteristics of the medium are tempered by information need) are central to the research proposed in this dissertation.

The proposed research also draws on social psychological theories of dual-processing, which state that information processing outcomes are the result of an interaction "between a fast, associative information-processing mode based on low-effort heuristics, and a slow, rule-based information processing mode based on high-effort systematic reasoning" [5, p. ix]. A specific dual-process theory, Chaiken's heuristic-systematic model, is utilized in this research to describe the credibility judgment process [3].

The aim of this dissertation is to combine the theories proposed by Wathen & Burkell and Chaiken to explain the effects of peripheral cues (of the information source being evaluated) and affective state (of the individual performing the evaluation) on credibility judgments [25, 3]. Specifically, this dissertation proposes two studies to examine the effect of individual involvement in the information seeking process on the perceived credibility of the information sources retrieved during the process. By examining how level of involvement (e.g., high or low) interacts with low-effort examinations of peripheral cues like source reputation (Study 1) and media format (Study 2), a better understanding of the complex and multifaceted process of credibility judgment can be attained.

Categories and Subject Descriptors

H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval – Selection process; H.3.7 [Information Storage and Retrieval]: Digital Libraries – User issues; H.5.2 [Information Interfaces and Presentation]: User Interfaces – User-centered design.

General Terms:

Measurement, Design, Experimentation, Human Factors, Theory.

Keywords:

Digital libraries, credibility judgments, authority, interface design, no results found, null result sets, usability, message, messenger.

1. INTRODUCTION

What is a credibility judgment? Intuitively, it is the act of deciding whether to place trust in a new piece of information and its author(s). In order to function in an information-rich environment, we implicitly and explicitly sort the constant barrage of information we are presented with through a series of credibility judgments. These credibility judgments do not determine what we know, but rather what we believe—in some sense they are the buffer that structures our understanding of the world around us.

A credibility judgment is defined herein as an individualized process, where a person sorts and categorizes a wealth of information sources into a hierarchy of plausibility—e.g., Wilson’s cognitive authority approach [26] adopted by Rieh & Belkin [22], or Charmaz’s constructionist approach [6] adopted by Lloyd [16]. Framing a credibility judgment as a process of pattern matching and selection emphasizes the primacy of the individual making the credibility judgment rather than the particular aspects of the information source being evaluated; this approach highlights how credibility is a perceived quality that varies between individuals, dependent upon (but sometimes in spite of) the impartial qualia¹ of information sources. This constructionist approach differs from more traditional² and mathematical research in the area of document relevance that focus primarily on the attributes of documents that influence credibility [8, 15]. In general, this paper adopts this individualized perspective when talking about credibility judgments because of the focus on the unique and situated nature of the process [9, 10]. It also treats a credibility judgment as a process that occurs between an individual and an information source (instead of a one-way communication from source to reader), and over time and in different contexts. While this perspective does not imply that the information source in question has agency, it highlights the importance of context on its interpretation. Because of the focus on perception and perceived qualities of information sources, much of the document qualia emphasized in mathematical definitions can be seen in terms of the perceptions of the individual making the credibility judgment. To highlight the distinction being made here, consider the act of determining whether a new email is real or spam (i.e., whether it is credible or not). Several implicit questions are asked in this process, including: Is the sender known? Is the message content believable? And is the format of the message proper (e.g., grammar, capitalization, and spelling)? In the mathematical framework, the credibility of the information source (i.e., the email message and by extension the sender) is a fixed attribute communicated to the information receiver (i.e., the person receiving the email) through its format, presentation, and content; the information receiver interprets this data³ to arrive at a conclusion regarding the source’s credibility. In the constructionist perspective, the credibility judgment is conceptualized as a process occurring within the mind of the information receiver. The same components of the information source are analyzed (e.g., authorship, presentation, and content), but rather than being seen as fixed elements of the information source, they are instead perceived qualities—there is no claim that they represent concrete truths. In other words, this perspective echoes Rees and Saracevic’s observation that relevance judgments are subjective and not inherent to a document [20]. By broadening the discussion of credibility beyond its representation as a quale of information, we can more clearly focus on the individual and situated nature of credibility judgments.

Information-seeking is by its nature a process of selection and consumption; individuals are bombarded, either through senses or thoughts, with new information, and whether or not they digest that information, whether or not they decide to treat it as worthwhile and credible, relies on an intricate and multifaceted process of triage. How can this complicated process of information triage be evaluated and understood? How do new, more interactive networked environments (e.g., the Web) change individuals’ behavior and expectations of the information they deal with? Further, what effect do supporting factors (such as poorly designed interfaces and the reputation of information sources) have on the interpretation of the credibility of actual contents? And what about individual factors, such as an individual’s affective state, level of self-efficacy, epistemological belief structure, and level of expertise?

2. MOTIVATING EXAMPLE

Wathen & Burkell provide a motivation for orienting a research agenda that examines the effects of messenger/medium on credibility assessment:

Further, they posit that “[a]ccording to persuasion theories, a user who has a high need for information will overlook weak peripheral cues. If the information-seeking episode is more casual, surface characteristics will have more influence” [25, p. 141]. These two claims (i.e., that the medium impacts credibility judgments, and that credibility judgments based on surface characteristics of the medium are tempered by information need) are central to the research proposed in this dissertation.

As will be discussed below, much of the past research into the perceived credibility of information sources is either of a qualitative nature or theoretical, but not quantitative (with the exception of a focused research agenda performed by communications scholars on the perceived credibility of a specific media, newspaper articles). The qualitative data that exists provides instrumental insight into the information seeking behaviors of individuals but lacks the generalizability that quantitative research offers. Because of this disproportionate balance in the literature, a quantitative methodology examining Web-based media is proposed in this dissertation to explore the factors that affect the credibility judgment process.

Figure 1. Proposed theoretical model for how users judge the credibility of on-line information (reproduced from Wathen & Burkell, 2002, click here for a larger view of the image).

Wathen & Burkell propose a theory of the credibility judgment process that separates the evaluation process into distinct segments [25]. In the theory, the process begins with low-effort examinations of peripheral cues (e.g., appearance, design, organization, and source reputation) before continuing on to a more high-effort analysis of the actual content of the information source (see Figure 1 for more detail).

The proposed research also draws on social psychological theories of dual-processing,⁴ which state that information processing outcomes are the result of an interaction “between a fast, associative information-processing mode based on low-effort heuristics, and a slow, rule-based information processing mode based on high-effort systematic reasoning” [5, p. ix]. A specific dual-process theory, Chaiken’s heuristic-systematic model, is utilized in this research to describe the credibility judgment process [3].

The aim of this dissertation is to combine the theories proposed by [25] and [3] to explain the effects of peripheral cues (of the information source being evaluated) and affective state (of the individual performing the evaluation) on credibility judgments. Specifically, this dissertation proposes two studies to examine the effect of individual involvement in the information seeking process on the perceived credibility of the information sources retrieved during the process. By examining how level of involvement (e.g., high or low) interacts with low-effort examinations of peripheral cues like source reputation (Study 1) and media format (Study 2), a better understanding of the complex and multifaceted process of credibility judgment can be attained.

3. LITERATURE REVIEW

3.1 Description of Domain

The semi-anonymous, self-publishing aspects of the Web challenge our traditional methods of establishing trust by reputation and authority. Traditionally, we consumed information (in print) in a centralized and hierarchical framework that created a clear dichotomy between content producers and content consumers⁵ . It is this framework that had customarily guided individuals’ perceptions of trustworthiness. In short, we placed our trust in “experts” whose expertise was vetted by others. In a qualitative study examining searching behavior and perception of information quality, Rieh demonstrated that in a networked world where information sources are far more varied and numerous, and where barriers to participation have been reduced to the extent that the lines between producer and consumer have been blurred, individuals must increasingly decide for themselves who the experts truly are [21]. Previous research has provided evidence that many individuals are underprepared to judge the credibility of online information sources that lack gatekeepers [1, 17, [24]. Whereas traditional publishing methods (e.g., print media) rely on an infrastructure of editorial review and monetary publishing restrictions, the Web, as a whole, lacks such gatekeepers; it is therefore more susceptible to a unique set of credibility pitfalls, such as misrepresentation, plagiarism, and utter falsehoods [12, 14]. As information consumers, how do we decide what and whom to believe?

3.2 Information Overload and Filtering

The difference in reliability between online information sources and traditional media is often the existence of gatekeepers, who act as filters. The high costs (both monetary and in the amount of time and human resources required to produce a document) associated with publishing in traditional media outlets has led publishers to construct their own information filtering system to ensure their profitability and survival. Five-hundred years ago, when the moving type printing press was a new invention, the very existence of a printed book implied that it was valuable and credible; today, the printed word is everywhere, and most individuals can make their own contributions to the growing multitude with little regard to cost. As demonstrated by Amsbary & Powell, the credibility judgments made in a mindset of information scarcity simply cannot be carried over into an environment of information overload [1]. Because of the massive amounts of available information with questionable authority, individuals must exact a process of information triage upon the barrage of materials available to them. Rieh & Hilligoss provide evidence that credibility judgments made on the Web are both more frequent and more perfunctory [23]. An individual cannot spend a great deal of time evaluating each source s/he is presented with and, because of this, more flippant and extemporaneous techniques (such as evaluation based on interface design) are being used.

3.3 Wathen & Burkell’s Credibility Model

Wathen & Burkell posit a three-stage theoretical model of the credibility judgment process, highlighting the effects of surface credibility (the impact of the messenger), message credibility (composed of source and message credibility), and finally the impact of the cognitive state of the individual making the credibility judgment [25].

The first stage, the evaluation of surface credibility, relies on an examination of surface characteristics such as appearance/presentation, usability/interface design, and organization of information. This stage answers these implicit questions: “Does the source appear professional? Can I find what I want quickly and easily?” The evaluation of surface credibility is the quickest of the three stages, in that it relies on heuristic judgments of visual cues that can be made without a thorough reading of the material in question.

The second stage, the evaluation of message credibility, assumes that some level of reading the content has occurred, and relies on an examination of source characteristics (e.g., expertise, trustworthiness, and credentials) and message characteristics (e.g., content, relevance, currency, and accuracy). This stage is most concerned with perceived trustworthiness and expertise [11], and answers this implicit question: “If the site has the information I’m looking for, will I believe it?” The evaluation of message credibility is a more in-depth process than the evaluation of surface credibility, but still does not require a thorough reading of the material; source and message cues can be determined by, e.g., a past familiarity with the author of the material, an examination of the date of publication, or the overall relevance to the individual’s information needs.

The third and final stage, the evaluation of content, is the most situated (i.e., situationally-specific) of the three stages, in that it places paramount importance on the cognitive state of the individual making the credibility judgment and how it interacts with both surface and message credibility. The evaluation of content answers such implicit questions as: “How badly do I need the information? How familiar am I with the topic? Am I ready to believe this information? Am I ready to act on it?” While we may expect individuals performing the first two stages of the credibility judgment process to exhibit some similarity in their judgments (since aspects of the information source in question, such as a lack of organization, tend to be consistently perceived across individuals), credibility judgments reached in the third stage are primarily dependent on the cognitive state of the individual, and are thus very much dependent on that individual’s specific context⁶ . Further, this final stage is the most time-intensive of the three stages, since it requires a thorough understanding of the material to address the questions asked. Content evaluation is a more involved process that may not occur if an individual is deterred by, for example, a source’s unfavorable design. Wathen & Burkell’s model implies that an individual’s cognitive and affective state is most important for the judgments made at this third stage, but this proposal adopts the stance that they are factors that influences each of the three stages. Thus the third stage is defined as a thorough analysis of the material in question, with the analysis reliant on an individual’s cognitive and affective state.

Wathen & Burkell propose (but do not test) that if an individual determines that an online source does not meet an appropriate level of credibility at any one stage, then s/he will leave the site without further evaluation. Further, they speculate that this “easy to discard” behavior is indicative of information-rich environments, where the assumption is that many other potential sources of information exist, and spending too much time on any one source is potentially wasteful. The organization of this model relies on this notion of temporal scarcity, in that it proceeds from the quickest and most general evaluations to the longest and most in-depth evaluations.

3.4 Dual-Process Theories

Dual-process theories [3, 4, 7] focus on the notion that information processing outcomes (e.g., credibility judgments) are the result of an interaction between two qualitatively different processes [5]. As emphasized by Chaiken & Trope, “the common distinction in dual-process models is between a fast, associative information-processing mode based on low-effort heuristics, and a slow, rule-based information-processing mode based on high-effort systematic reasoning” [5, p. ix]. These theories tend to be proposed in opposition to the essentializing notions of single process theories that over-explain the importance of a distinct factor that leads to an outcome, and in so doing ignore the synergy that emerges from interrelated processes.

Hilligoss & Rieh [23] point to two particular dual-process theories that are well-suited for describing the process of persuasive communication at the base of information evaluation: Petty & Cacioppo’s [19] elaboration likelihood model (ELM) and Chaiken’s [3] heuristic-systematic model (HSM). The ELM describes an individual’s attitudinal changes as they evaluate information sources, and shows how these changes lead to the individual focusing on either (1) peripheral cues, or (2) the content itself. The HSM describes how an individual’s motivation and ability leads to a focus on either (1) low-effort heuristic processing, or (2) high-effort systematic processing. In both cases, (1) reflects an evaluation of the messenger (e.g., reputation, design), while (2) reflects an evaluation of the content of the message.

3.4.1 Applying Dual-Process Theories to Credibility.

Wathen & Burkell’s three-stage model can be viewed as a dual-process model since it involves an interaction between low-effort evaluations (of surface and source) and high-effort evaluations (of content). The first two stages, the evaluation of design features and message features (e.g., reputation), can be categorized as low-effort processes since they are dependent on relatively immediate perceptions of quality rather than on a process of high-effort systematic reasoning. The third stage (evaluation of content), on the other hand, requires a focused effort to understand the information being communicated by an information source, and as such requires a comparatively large amount of effort to arrive at a judgment of credibility.

One important example of the applicability of dual-process theories to online credibility judgments is the effect that involvement (alternatively, investment or motivation) has on these judgments. Casual and directed information-seeking episodes can lead to a difference in the amount of importance ascribed to, e.g., design evaluation. A single process theory that examines the effects of poor interface design on credibility judgments may directly conclude that poor design leads to a decrease in perceived credibility [2]. But this simple analysis does not take into account highly motivated/invested users who are willing to overlook bad presentation in order to find applicable information, nor does it explain novice users with low self-efficacy who may be more likely to ascribe interface difficulties to their own lack of expertise.

3.4.2 The Heuristic-Systematic Dual-Process Model.

Dual-Process theories have yet to be applied to credibility judgments in the research literature, but Rieh & Hilligoss [23] and Metzger [18] point to the applicability of their use. This line of inquiry is arguably a fruitful method for understanding the relationship between aspects of information sources that increase credibility, and the particular characteristics individuals bring to the information-seeking context that impact their ability to make informed credibility judgments. Chaiken’s [3] heuristic-systematic model is particularly suited to explaining credibility judgments in an online setting because of its distinction between quick, low-effort heuristic evaluation and involved, high-effort systematic analysis; the theoretical frameworks of credibility reviewed here all describe low- and high-effort components of the judgment process (typically split between messenger and message evaluation, respectively).

Chaiken’s original experiment involved university undergraduates (N=207) expressing their position on an argument they read (either advocating sleeping less than eight hours a night, or moving from a two-semester system to a trimester system at the university in question) in an interview transcript in which an interviewee expressed his opinion on the argument. After the initial session, participants were contacted via phone to assess their agreement with the positions advocated in the original argument. Two independent variables were examined: perceived consequence (i.e., investment in the topic) and communicator likability (i.e., the likability of the individual making the argument in the interview transcript). Perceived consequence was manipulated by telling participants that they would be involved in a discussion session following the reading of the interview; the high-involvement group was told that they would be discussing the argument presented in the interview transcript, and the low-involvement group was told they would be discussing an unrelated topic. Communicator likability was manipulated by including (in the interview transcript) an opinion of undergraduate students expressed by the individual making the argument; the “likeable” group read that the interviewee had a positive view of undergraduate students, and the “unlikable” group read that the interviewee was disdainful of undergraduates. Results from the experiment showed that high-involvement participants were more likely to change their opinion, regardless of communicator likability, and low-involvement participants were more likely to change their opinion only if presented with a likable communicator. Chaiken concluded that high-involvement led to more high-effort, systematic information-processing (i.e., evaluating the persuasive argument for its merits), and thus was less susceptible to low-effort, heuristic cues like an unlikable communicator; and low-involvement led to more low-effort, heuristic information-processing (i.e., evaluating the likability of the communicator).

3.5 Composite Theory of Credibility

The research method presented here is intended to determine the effects of individual involvement (reader motivation) on the credibility judgment process, and how level of involvement leads individuals to make judgments based on low-effort heuristics (e.g., reputation, media format) or high-effort systematic reasoning (e.g., evaluation of content). Wathen & Burkell’s theory posits that inferring low credibility in either of the first two stages will result in no further stages being examined. When viewed through the lens of a dual-process theory, this means that individuals with low information need (i.e., low involvement) will not arrive at an examination of content, because they will have made a credibility determination during the heuristic examination of surface characteristics in the first two stages⁷ . The primary purpose of the two studies proposed here is to better understand the interaction between the variables of reputation and involvement (Study 1) and format and involvement (Study 2) on people’s perceptions of credibility by comparing participants’ credibility judgments across contrasting groups. To date, no empirical support has been provided. The two proposed experiments are intended to test the hypotheses drawn from both Wathen & Burkell’s [25] theory of credibility judgments and Chaiken’s [3] heuristic-systematic dual-process model, specifically that credibility judgments are influenced by media format and reputation in addition to the veracity of the content, and especially so in cases of low individual involvement. The independent variables of interest are: media format (print or electronic); media type (PDF or HTML); reputation of source (Wikipedia or Britannica); and the participant’s individual involvement in the research task (high or low). The dependent variable of interest is the perceived credibility of the information source compared to other information sources (rank order of high to low). Two studies are proposed. Individually, these two studies focus specifically on the variables of interest and should provide more precise information concerning the underlying psychological mechanisms and environmental applications of the independent variables (media format, reputation, and involvement). The results of Study 1 address the interaction between source reputation (positive vs. negative) and participants’ level of involvement (high vs. low) in affecting the degree of credibility. The results of Study 2 will provide a concise theory for how each level of media format (print vs. electronic) interacts with each participant’s level of involvement (high vs. low) in affecting the degree of credibility. Both studies will provide researchers, librarians, designers, and laypersons a better understanding of how design and reputation in combination with user involvement can affect the credibility of information sources.

4. REFERENCES

Footnotes