Digital Libraries: Doing Data Differently

Metadata is often created in a time-consuming process by catalogers, digital library technicians, and others. It is then underused in our systems and the systems of those with whom we share the information. But recent experiments by some library organizations indicate that we are only scratching the surface of what our data can do for us.

The American West Project at the California Digital Library (CDL) is aggregating metadata that describes digital objects related to the American West from a couple of dozen libraries, museums, and archives. When we used the Open Archives Initiative Protocol for Metadata Harvesting to gather these records, we discovered such significant metadata problems I was prompted to write a paper ("Bitter Harvest") as well as a similarly titled column (LJ 7/04, p. 32).

Now, a year later, we have made progress toward developing a set of procedures and software routines to normalize, transform, and enrich these records in new and interesting ways. For example, we discovered that the institutions from which we were obtaining records encoded dates using a wide variety of methods (for examples, see "CDL's OAI Harvesting Infrastructure").

We also discovered that useful dates were not always encoded in date fields within the record. Sometimes dates were embedded within a title field. Dates also appeared in descriptions, subjects, and other fields. We are writing software routines to normalize the dates in these records and match the normalized dates against a set of time periods that relate to the history of the American West. Because of all this processing, the users should have a much richer interaction with the site.

Normalizing is rather straightforward compared with trying to assign broad topic headings to a mass of heterogeneous records with subject headings from different vocabularies or with no headings at all. For this we turned to clustering software. Clustering software attempts to find associations among records that can then be exploited to assign one or more subject headings to appropriate clusters. We do not yet have a production capability for doing this, but early experiments have been encouraging. Meanwhile, the institutions from which we obtained these records are interested in the possibility of getting the enriched records back.

This desire of institutions to receive their enhanced records back is a familiar one to Diane Hillmann and her team at the National Science Digital Library (NSDL). They have long worked to improve the metadata they received from institutions participating in the NSDL and have recently taken that experience further by suggesting that an appropriate and useful role for metadata aggregators is to provide enhanced metadata for others to use (see "Improving Metadata Quality").

A basic finding of both the NSDL and CDL is that metadata enrichment should not be based solely on human-only nor software-only procedures but rather on a mix that uses the strengths of each to its full potential.

OCLC provides yet another example of how our metadata can do more for us. Lorcan Dempsey, OCLC's VP of research (see "Making Data Work Harder"), pointed out how a significant characteristic of both Google and Amazon is that they squeeze as much work out of their data as they can—all to create more useful and compelling services. He suggests we could learn a lot from them, as well as others that are also making data work harder. OCLC is itself "mining" the rich metadata store of WorldCat in interesting ways that may soon show up as useful new services (see "Works 4 You?").

Another example used frequently in this column is RedLightGreen.com, the bibliographic database tailored for the needs of undergraduate students by the Research Libraries Group. In RedLightGreen, subject headings from the records retrieved by a search are pulled out and put in a prominent location for users to discover easily. Why should this very useful information lie hidden in the metadata and only surface when a user requests to see the full record?

Our rich collections of metadata are underused. Meanwhile, we can employ automated techniques to make our metadata more uniform when it needs to be uniform and richer when it needs to be richer. We must get smarter about metadata in so many ways. Never have the skills of software-savvy catalogers and metadata-savvy software engineers been so essential to the future of our libraries and the users we serve.