FAQ on the LJ Index
-- Library Journal, 04/21/2009
Several questions have been raised, both online and offline, about the new LJ Index (LJI) of Public Library Service. This set of responses will be augmented at a later point, as the discussion continues.
As authors Keith Curry Lance and Ray Lyons explain, the LJI is “not cast in stone.” They encourage libraries to report their statistics to their state library agencies and ask the library community for feedback on what statistics are relevant to their communities and their funders.
While they mine the data from the Institute for Museum and Library Services (IMLS), they note that “for new indicators to be incorporated, IMLS and the state library agencies must agree to collect and report them.”
Given that 2007 data are expected this May, the next edition of the LJI will be later this year.
How were libraries included or not included?
See the Libraries Included in the LJ Index article for an explanation. If you can't find your library, and you'd like to know why, this list of excluded libraries may help you identify a particular reason (to search these documents click on a cell within the tables, press Control-F, and make sure that "Search all sheets" is checked).
Why did some states not have any star libraries?
While libraries from all states were included, in eight states no libraries got stars. This is due to how the Index scores for libraries in these states compared to all the libraries in their respective expenditure categories, and not due to the state they hail from.
How are scores calculated?
See the Fact Sheet for a detailed explanation of how scores are calculated.
Why do you use the statistical item "Users of Electronic Resources”?
That term was not used. The term “Users of Electronic Services” was used in the June 15, 2008 article, which described the concept for the LJI. While 2005 public library statistical data published by the Institute of Museum and Library Services cited “Uses of Electronic Resources,” the 2009 article is based upon 2006 data, which reports “Uses of public-use Internet computers” or, as cited in the article, “public Internet computer uses.”
Why aren’t inputs included?
The LJ Index is a simple index of public library service output, not library ''quality'' or ''goodness''—vague concepts for which available data are completely inadequate to measure conclusively, if at all. Combining input and output data in the same index as some suggest—and Hennen's American Public Library Ratings (HAPLR) does—begs this very question.
What's wrong with combining inputs and outputs?
HAPLR scores are compared by their creator to SAT scores. HAPLR scores are not like SAT scores in any conceptually meaningful way. Case in point: if SAT tests were scored the way HAPLR scores libraries, a student’s SAT score would be based on his or her personal demographics, family socioeconomic characteristics, study habits, school staffing and funding, etc. (i.e., the inputs), as well as how the student answered the questions (i.e., the outputs). This would be a preposterous practice—which is precisely why it does not happen. Inputs are invaluable in designing a probing analysis of why some output scores are higher and lower, but data about them should not play any role in determining an output score.
What about an index of library efficiency?
Efficiency—the ratio of output to input—is a legitimate criterion on which to rate libraries. But it is a different criterion than service output, which is the focus of the LJ Index. It is probably impossible to design an index with any measure of statistical validity that simultaneously measures both service output and efficiency. Thus, the LJ Index focuses on service output.
Can there be a single, all-encompassing measure of library quality?
That is akin to searching for the Holy Grail. The data we have—and are likely ever to have—will not provide it. The LJ Index scores libraries on a concept for which key, if not comprehensive, data are available.
Why aren’t reference transactions included?
Reference transactions are excluded for a simple reason: the available data on such transactions don’t correlate sufficiently with data on the four LJI output measures to justify inclusion. Anyone who has looked at library statistics recognizes that reference numbers can be questioned.
One ground is validity: do the numbers encompass only the intellectual product of trained librarians or do they include such things as directional information? Another ground is reliability: do all those reporting such transactions count in the same way?
Yes, all public library statistics can be questioned on those grounds, but concerns are greater for reference, because it is related much more weakly—consistently over time—to the four LJI indicators than they are to each other.
Consider the statistical relationships between the four LJI statistics to be like the relationships between various types of fruit. While such similar outputs can be placed appropriately in the same index, the variety of possible activities being subsumed under ‘reference’ might be likened to adding an onion to the fruit bowl.
The decision to exclude reference transactions does not reflect on the value of reference services, but rather on the quality of reference statistics. This remains an issue for the reference librarian community to address—preferably, on the basis of sound research.
Even though the four statistical indicators in the LJ Index are equally valued, could a library end up with a very high score thanks to a single indicator?
Yes, a very high per-capita statistic on one service measure can earn a library a higher score and a star rating when another library bests it more modestly on the other three service measures. This was addressed explicitly in LJ: “Because data is not reduced to ranks, a very high value on one indicator relative to the group average can earn a library a higher score than another that bests it on the other three indicators. (For more, see a post by Ray Lyons.) This sensitivity will make the LJ Index a valuable tool for identifying exemplary performance as well as dubious data.”
Neither of these ends is served by the limited HAPLR data made available in American Libraries or on the HAPLR web site. Indeed, by reducing ratio data to ranks, equating input and output measures, and utilizing imputed data (i.e., data not actually reported by libraries), HAPLR’s public manifestations provide few, if any, details to inform conversations about exemplary performance or data quality issues.
Moreover, the LJ Index is not weighted.
What exactly is weighting?
HAPLR creator Thomas Hennen claims that “The Index factors are not equally weighted. Fully 100% of the LJ Index weight could go to any of the four factors. That is curious, indeed.”
Actually, an extreme value reported on a statistic by a particular library does not constitute weighting that statistic. Weighting concerns an index creator’s decision to give more influence to one particular statistical indicator over another. LJI does not do this, while HAPLR does.
When a given library scores highly on a single indicator—as explained in the article—that single item may well overshadow the rest. But this is not “weighting.” It simply reflects the data. It is incorrect to conclude that, because the LJI items are equally weighted, there must therefore be a limit on how high each item can go, or that this limit is needed in order for the items to be considered equally weighted.
This misconception likely comes from applying the familiar measurement approach seen in school exams scoring. The typical case is a four-section exam where students are able to score up to 25 points on each. This limits the total possible score to 100, and is designed to produce scores that are analogous to percentages. It does not apply to library statistical comparisons that use real-world data. The LJI puts no arbitrary upper limits on the values of the indicators. Thus, it shows some libraries quite far from their group average, either authentically so or due to data errors/misreporting, or due to very low population figures which exaggerate per capita ratios.
Is it an equivalent decision to weight or to not weight the measures considered?
No. The choice to weight some measures more than others must be justified by the index creator. HAPLR weights originated with an informal listserv poll that—as far as we know—has not been reviewed or revised in at least a decade.
In LJI, the decision to give “equal opportunity” to qualitatively different service indicators does not constitute an equivalent active bias. Further, in the social science and statistical literatures, there is a growing body of skepticism toward weighting index variables.
Hennen dismisses methodological concerns about HAPLR by alleging that it is not a statistical index, but merely a rating system. Generically speaking, HAPLR is an index; abandonment of conceptually and statistically dubious weightings could improve it. In fact, HAPLR scores are distorted not just by weightings but also by the reuse of data like circulation figures in multiple measures.
