Beyond the Numbers

Look how users are impacted, say Edwin S. Clay III and Patricia Bangs, and you will discover, and be able to convey, the real worth of libraries

By Edwin S. Clay III and Patricia Bangs -- netConnect, 1/15/2006

Librarians crunch numbers. We collect data in our physical environments, such as circulation stats, door counts, or program attendance. We are learning to collect data in our electronic environments and are even becoming adept at manipulating numbers to reflect the return on investment that governing boards and officials can appreciate. The one question we still have trouble answering with numbers, however, is the most essential in this era of accountability: “So what?” We must be able to quantify the value of the services we provide to our customers and those who fund us.

Libraries traditionally collect data about staff activities. How many books circulate, how many programs are presented, and how many reference questions are answered are all output measures. Libraries also measure the costs of providing services and resources, or inputs. Inputs include resources devoted to a program; collections, materials, equipment, and supplies; staff and volunteer time; community partners; facilities, computers, and online access; and even limitations, such as laws, regulations, and funding requirements. Inputs help libraries measure efficiency.

A higher measure

According to library consultant Rhea Rubin, another way to evaluate library services is through user-oriented measures known as outcomes (see “Measure Your Worth,” p. 10). The power of outcomes is in the revelation of whether you have touched lives with the work you do. Rubin explains, “Outcome measurement is a user-centered approach to the planning and assessment of programs or services that are provided to address particular user needs and designed to achieve change for the user.”

The practical use of outcome-based evaluation is still in its infancy for library systems, so Rubin believes quantitative measures should be combined with outcomes. John Carlo Bertot, professor and associate director of the Information Use Management and Policy Institute at the School of Information Studies, Florida State University, Tallahassee, agrees that a blended approach is best and recommends using some or all of the input, output, and outcome measures for good decision-making.

A family of measures

At the Fairfax County Public Library (FCPL), VA, we've adopted this approach in measuring our effectiveness to discover how we are making a difference in people's lives. We use a family of measures, which include (1) outputs; (2) efficiency measures, such as cost per visit; (3) service measures, such as visits per capita; and (4) outcomes, such as customer satisfaction. We define outcomes less strictly than Rubin does but apply her user-oriented outcome model for special, grant-funded projects, like a computer literacy initiative for seniors in an assisted- living facility. The nature of this project allowed us to gauge participants' computer literacy before and after the initiative.

Outcomes measurement is only the latest addition to an impressive set of data collection competencies for FCPL. We have been evaluating the quality of our service for more than 20 years and use a social entrepreneurship approach to public library management, as defined by J. Gregory Dees. In a Harvard Business Review article (“Enterprising Nonprofits,” Jan./Feb. 1998, p. 54), Dees suggests leaders of nonprofits should design strategies on a “social enterprise spectrum,” a model that combines both commercial and noncommercial aspects of a nonprofit organization. The business community bases strategic decisions on data that answers the question, “How are we doing?” and libraries should borrow from that practice.

We recognize that the public library's role as an institution for the public good as well as a public forum tempers any business model. Dan Walters, president of the Public Library Association and executive director of the Las Vegas–Clark County Library District, NV, believes gathering data is essential but should not overshadow the mission-oriented work of public libraries, which may not always have an economic benefit. There is no need to measure things that are self-evident and required in our role as an institution for the public good, Walters adds. Quoting Bob Dylan, Walters maintains there are times when “you don't need a weatherman to know which way the wind blows.”

We agree. Even if FCPL's demographics skewed toward males, aged 16–25, replacing our news periodicals with copies of Wrestling magazine would be irresponsible. Still, we might try to find funding for extra copies of Wrestling.

Tooling along

Unlike many public library systems that combine statistics efforts with other management functions, FCPL created a separate office for evaluation and planning in the early 1980s. Our initial efforts moved beyond standard output measures, such as circulation and door count statistics, to service-oriented numbers for a more complete view of the quality of our efforts.

Over the years, we developed various tools that measure effectiveness. Each year since 1982, we have conducted an in-branch Information Services Survey (ISS). Staff record the number of information, directional, and electronic resource instructional questions during a sample two-week period. The survey allows us to determine service quality measures, such as questions per capita, and efficiency measures, such as cost per capita. It also tells us who asks the questions—children or adults—and the level of in-house use of electronic resources.

A second survey FCPL conducts each spring is the ten-question Web Site User Survey, posted on the library's web site. It measures user satisfaction with the site, ease of site navigation, and what additional library e-services users might want. Along with the ISS and the number of Internet sign-ups and web site views, the Web Site User Survey helps measure the library's business activity.

Electronic transactions are a growth area for libraries, but collecting data on them can be difficult as technology changes, creating discontinuity in historical data. For example, FCPL now uses interactive voice response (IVR) technology to automate information like hours, directions, and even the library's URL. IVR eliminates the need for staff to respond to at least 9000 telephone questions every month. Of course, phone service is still an important aspect of what we do, but IVR is a good example of how technology lets us serve the public well yet affects data analysis over time.

Getting to potential users

Periodic in-branch and telephone-based user surveys are perhaps FCPL's most useful measure of customer reaction to library services, something we began in 1985. These surveys tell us who is coming to the library and why, but they are intrusive, so FCPL limits the number we conduct. In spring 2005, we completed our first telephone-based survey in eight years. It provided us with longitudinal information on how and if the changes in information services have affected library use. The 33-question survey determines the top library services used by customers, satisfaction rates, reasons noncustomers did not use the library, awareness of existing library services, new services customers might use, top web site uses, and how customers value the library in the community. We are still analyzing the results, but it is a rich source of data as we develop indicators in our budget and make management decisions. The main advantage of a survey like this is that it taps the people who don't use libraries. We were surprised to find that these individuals do value libraries, even though they may not use them, meaning libraries continue to have perceived value as a public good.

Always accountable

When the effects of the Government Performance and Results Act finally reached the local level in 1997, making performance measures a part of the Fairfax County budget process, FCPL was in a much better position than many other county agencies to devise measurable goals and objectives.

For example, for the FY06 budget, we submitted indicators based on four types of measures—output, efficiency, service quality, and outcome—to show how successfully we reached each goal for our various operational areas or “cost centers.” We used a model similar to Rubin's, in which we defined a library goal, selected realistic outcomes, identified indicators of change, and collected data to see if the goal had been realized.

As an example, the goal for the Library Operations “cost center” in the FY06 budget is “to provide public services that deliver information and materials to meet the informational, recreational, and educational needs to residents in a timely and easily accessible manner.” Specific objectives include achieving a resident contact rate of at least 36.4 contacts per capita in FY06 and responding to library users' information requests accurately and quickly by answering 71 percent of questions within 24 hours.

For output measures, we used the number of holds placed, total circulation, number of library visits, program attendees, total contacts, hours open, information questions addressed, in-house print use, in-house electronic use, and pages viewed on the library's web site.

For efficiency (or input) measures we calculated cost per citizen contact, contacts per hour of service, contacts per staff hour, questions asked per staff hour, and questions asked per hour of service. For service quality, we measured customer satisfaction and questions asked per capita. For outcomes, we used institutional measures, such as contacts per capita and the reference completion rate within 24 hours.

User-oriented

Collecting data from a variety of sources—our networked circulation system, web-based visits and products, staff and user surveys—has served FCPL well in making good management decisions and justifying ourselves to stakeholders, including our partners in the business community. The key is examining the data in new ways. For example, we developed a measure for items desired by customers at individual branches by comparing the percentage of books owned in a particular Dewey area—e.g., medical sciences—with the percentage of branch holds in that area. This allows us to fine-tune our collection and get the best value in a period when we have had our materials budget drastically reduced.

Eventually, we want to incorporate even more user-oriented methodology, but, for now, we have found that the mixture of data we gather can help us measure our success in achieving the outcomes we desire. We are on our way to proving the difference libraries make in the lives of those who use them.