Reference Is Better Than We Thought

What is good reference service? What are the desirable outcomes? Is the quality of reference service most dependent on the library, the librarian, or the user? These are questions that have been asked by reference librarians for at least three decades-even longer if you count Edith Guerrier's 'The Measurement of Reference Service,' published in the July 1936 issue of LJ (p. 529-531).

Starting in the late 1960s, reference researchers like Charles Bunge, Thomas Childers (articles by both of whom appeared in this magazine), and the late Terry Crowley have argued about the answers to these questions. They have most often defined quality reference service in one of three ways: accuracy, utility, or user satisfaction, but many other theories have been suggested. Herbert Goldhor's different theory, for example ('Performance = Accuracy = Staff Ability + Library Collection') reflects the thoughts of many writers on this topic. Of course, Bunge might have preferred to say, 'Performance = Efficiency = Accuracy/Time = Staff Ability + Library Collection.' Crowley would state it thus: 'Performance = Accuracy = Library Collection + Staff Ability = Budget.'

The point is that in prior studies of reference service, accuracy has usually meant some number of judges scoring ten or 20 so-called typical questions on a scale ranging from 'completely answered' to 'not answered at all.' The results of these studies, unfortunately, are all too familiar: half-right reference service. This outcome was dubbed the '55 percent rule' by Peter Hernon and Charles McClure ('Unobtrusive Reference Testing,' LJ 4/15/86, p. 37-41).

Much work has been devoted to the scoring of questions (see, for example, Cheryl Elzy and others' 'Evaluating Reference Services in a Large Academic Library.,' College & Research Libraries, 9/91, p. 454-465) and John Richardson and Rex Reyes, 'Government Information Expert Systems.,' College & Research Libraries, 5/95, p. 235-247). The results, however, still come up the same: half-right reference service.

On the other hand, utility and user satisfaction have been measured by exit surveys as the user leaves the library-also with now-familiar results. Users found at least something they wanted, so they expressed very high satisfaction with the outcome of their reference encounter. These results seem contradictory. How can half-right reference service be found to have high utility and give great satisfaction to users?

Citations to nearly 1000 studies of various aspects of reference service are listed at purl.org/net/reference. If you only want to read a few, try Kenny Crews's 'The Accuracy of Reference Service.' (Library & Information Science Research, 7/88, p. 331-355) and Matthew L. Saxton's 'Reference Service Evaluation and Meta-Analysis.' (Library Quarterly, 7/97, p. 267-289). After good, critical reviews of this literature, both authors found a lack of agreement on the definition of reference service and inconsistent operational definitions of both the independent and outcome variables. They discovered bias owing to a lack of random sampling, simplistic statistical procedures, low sample sizes, little repetition of prior studies to confirm or dispute earlier findings, and a lack of attention to theory.

Most prior studies have found that the three most important outcomes (i.e., accuracy, utility, and satisfaction) appear to be almost totally unrelated, which suggests that they are driven by different underlying factors. For instance, users often indicated that they were satisfied even when they did not receive a useful response from a librarian. Users also indicated they received useful information even when it was inaccurate.

Many of us have found it difficult to understand such results. It just doesn't make sense that our users would be so happy when they received half-right reference service. Indeed, many reference librarians have wanted to disagree with such research results in letters to journals and at conferences. Until now, however, it has been difficult to argue with that research without some solid alternative evidence.

I had several conversations with Terry Crowley in the late 1980s while I was doing my own research on this topic. Crowley wanted to believe that this contradictory situation was a methodological artifact. He maintained that someone, some day, was going to resolve this apparent conflict. In his 'Half-Right Reference: Is It True?' (RQ, Fall 1985, p. 59-68), Crowley posed the right question.

It's not true, and now we know why. Although Crowley didn't live long enough to learn all the details, it turns out that so-called typical, 'fact-type' queries used in all of the previous accuracy studies were only representative of half of all real queries received at reference desks. Matthew Saxton and I coauthored a new study of 12 different public libraries in southern California (Anaheim, Azusa, Beverly Hills, El Segundo, Glendora, Heritage Park Regional Branch, San Juan Capistrano, Pomona PL, Santa Monica, South Pasadena, Torrance, and Yorba Linda). We found that the so-called '55 percent rule' has never been tested against a truly representative field sample. In 90 percent of the cases in this examination, a panel of reference experts determined that librarians recommended an accurate source or an accurate strategy in response to a user's query.

The most important factor predicting accuracy was the difficulty of the query. This finding is intuitively obvious-it makes sense. Earlier work didn't make sense. The reference service performance model was overly simplistic, samples were way too small, and the test questions simply were not truly representative of real-world reference questions.

For the first time, we now have a study with a sophisticated model, one of the largest samples ever (9,274 persons inquiring for assistance), and questions drawn from the library users' realm-all employing the latest statistical techniques.

It's also important to mention some of the other significant findings from this study. For instance, library users are more satisfied by those librarians who actively practice the reference skills outlined in the Reference and User Services Association (RUSA) 'Guidelines for Behavioral Performance of Reference and Information Services Professionals' (RUSA: American Lib. Assn., 1996). They include inviting queries, expressing interest, listening critically, and verifying user satisfaction.

The reference model should now read 'User Satisfaction = Librarian Behavior [based on those RUSA guidelines].' Furthermore, these findings also suggest that the RUSA guidelines are really customer service guidelines, that they are reliable and valid.

Finally, the probability of an individual finding valuable and complete information is not only dependent on the librarian's reference skills but is also predicted by the user's familiarity with the library and his/her level of education.

Reference librarians should stop berating themselves. We are doing reference work much better than we thought we were.

More utility measures for the assessment of reference should be put in place in the future, particularly if one is trying to distinguish between good and poor service. Students of reference service should learn about the existence of multiple performance outcomes (i.e., accuracy, utility, and satisfaction) and to recognize that each outcome is driven by different factors.

As we all know, many people with computers make use of brute force search engines without the help of reference librarians. This change in user behavior is called 'disintermediation.'

Our new study shows that we need to view reference service as the same interpersonal process as was envisioned by Samuel Swett Green ('Personal Relations Between Librarians and Readers,' LJ 11/30/1876, p. 74-81). This is especially important as we move toward web-based, 24/7 virtual reference enterprises. We might make old Samuel S. Green proud!