Verbal Protocol Analysis

Verbal protocol analysis (VPA) is a qualitative, process-tracing technique whereby participants think aloud while engaging in a task, arriving at a decision, or making a judgment. Verbal protocols are typically content coded or examined in terms of the cognitive processes used. Although the use of VPA is quite rare in industrial and organizational psychology, some have argued that its absence is a detriment to our science. This is especially apparent because its use has substantially increased the understanding of interesting phenomena in a number of other fields, such as cognitive science, education, and human factors psychology.

Conducting Verbal Protocol Analysis

Like any data-collection or -analysis technique, there is no best way to conduct VPA; however, the following general principles likely apply to most uses.

• Collect concurrent data. Research strongly suggests that data collected in real time are superior to those collected after the fact; therefore, whenever possible, have participants vocalize their thoughts as they engage in the task.
• Record the data. Early forms of VPA relied on the experimenter’s notes because recording equipment was not readily available. However, the ubiquity of analog and digital recording equipment makes recording protocols simple and cost-efficient.
• Transcribe the data. Although qualitative data are never easy to manage, computer programs are available to provide assistance, but only when data are fully transcribed.
• Plan your work. Have an idea of what you are looking for before you immerse yourself in the data. As with any research technique, this means drawing from relevant theory, making specific predictions about what you expect to find, and specifying defensible ways to determine whether your predictions were met.

Uses in the Organizational Sciences

Although VPA is certainly considered an unorthodox research method among most organizational scientists, it is not completely unheard of. A handful of studies have used VPA in an organizational context, and in most cases, it has helped to answer questions that are firmly rooted in traditional organizational science domains but require information that traditional methods simply cannot provide.

A good example is the use of VPA to investigate the processes that individuals use when deciding whether to apply for a given job. The researcher could use more traditional techniques, such as self-report, by asking participants what information they think they typically pay attention to while reading job ads, or a type of policy-capturing methodology to elicit similar information. However, VPA is likely a better choice because it allows the researcher to directly assess the real-time reports of participants’ strategies, a benefit that few methodologies offer. Other examples of the use of VPA in the organizational sciences include examining the thought processes that job seekers use to evaluate potential employers’ reputations and assessing the construct validity of the organizational culture profile through VPA alone, which brings up an interesting point regarding the potential use of VPA as a psychometric aid.

Traditional psychometric assessment devices are necessary for the further development of quality measurement systems, but they may not be entirely sufficient. This is especially true when assumptions are made regarding the cognitive processes that individuals use when responding to a given instrument. The importance of this sentiment—often referred to as cognitive process validity—was recently recognized by the Society of Industrial and Organizational Psychology, which argued that such assumptions should be empirically tested before instruments are used in applied contexts. Verbal protocol analysis provides one possible means of doing this.

Traditional psychometric tools provide little information as to why specific items are behaving poorly. Thus, psychometricians are often left with a difficult choice: (a) blindly rewrite the items in question, retest the entire scale, and rerun the analyses; or (b) throw out the problematic items. Not surprisingly, many psychometricians often choose the second option. However, a small amount of verbal protocol data gathered from participants while they are completing an instrument may allow researchers to pinpoint the precise cause of the unexpected item performance. For example, VPA may indicate that some of the items are confusing, some terms are unknown to the target sample, some questions lack important situational context, or the scale is too long and participants become bored and careless toward the end. These are all insights that are typically not available with traditional psychometric assessment tools—but, of course, there are some caveats.

Cautions and Limitations

The primary critiques of VPA can be boiled down to two arguments: (a) Verbalizations show only a moderate relationship to actual behavior, and (b) verbalizing one’s thoughts fundamentally alters the cognitive processes that are typically used while engaging in a task. These claims are certainly not without merit; a number of studies show at least partial support for them. However, counter analyses also show that these limitations are not attributable to VPA as a data-collection tool per se; rather, they may be the result of sloppy data-collection procedures. That is, the use of VPA does not guarantee good or bad data; instead, the quality of the data is determined mainly by the quality of the procedures used to gather them. If caution, common sense, and rigorous data-collection methods are used, there is no specific reason why quality verbal protocol data cannot be used to examine a host of interesting phenomena.

Although VPA certainly has the potential to be a useful tool in the organizational sciences, it is by no means a silver bullet. Like any form of data, it is most effective when it is used in conjunction with data from a variety of sources. Researchers who are interested in using VPA should be forewarned that the resulting data are not necessarily easy to manage. Not only is it almost always necessary to transcribe the data before engaging in qualitative analyses; it is also often necessary to enter quantitatively coded data into a traditional statistics package such as SAS or SPSS so that basic analyses can be conducted. Finally, data collection tends to be quite slow because the data are generally collected one person at a time; therefore, it often takes days or even weeks to get a sample size as large as those that can be easily obtained by most quantitative techniques in one session.

Summary

Verbal protocol analysis is an underused but potentially valuable qualitative data-collection tool whereby participants think aloud while engaging in a task or behavior. It has been shown to provide unique and valuable insight into the cognitive processes that individuals used in a variety of settings. Though it is certainly not without its critics, this technique has stood up to the majority of critiques, and it is now considered a relatively orthodox tool in a variety of fields. Though it will certainly never replace any traditional organizational research tool, it does have the potential to bolster claims about the cognitive processes that individuals use while engaging in relevant behaviors. Similarly, it may also be a useful psychometric assessment tool; however, further research is required in this area before any firm conclusions can be reached about its actual incremental value over and above traditional techniques.

References:

1. Barber, A. E., & Roehling, M. V. (1993). Job postings and the decision to interview: A verbal protocol analysis. Journal of Applied Psychology, 78, 845-856.
2. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (2nd ed.). Cambridge, MA: MIT
3. Green, A. (1995). Verbal protocol analysis. The Psychologist, 8, 126-129.
4. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
5. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.