Last Byte: Free Your Search with Open Source

While open source federated search tools are free of cost, this is the least of their several advantages. Built-in application programming interfaces (APIs), highly customizable interfaces, speedy return of search results, and the ability to participate in the development process are all potential benefits, though they also require staff time and effort to customize and configure successfully. In order to make the best decision, librarians should take the time to compare the features, functionality, and costs of open source products with their commercial cousins to choose the tool that best meets their needs.

In spring 2007, the University of Houston decided to take on this investigation. We had a lackluster experience with a commercial federated search tool in the past. We had difficulties getting customizations made in a timely fashion. Our product was also prone to exponentially increasing retrieval times as the number of search targets increased. Our reluctance to spend more on a similar tool, coupled with our desire to build discipline-specific federated search collections, led us to investigate open source options available to us. While we ultimately went with LibraryFind, developed by Oregon State University Libraries, we discovered two other open source tools as well: Index Data's Pazpar2 and dbWiz, developed by Canada's Simon Fraser University Library.

Pazpar2 is a middleware web service, which allows libraries to develop their own interface in the programming language of their choice. This requires significant development time, so, for libraries daunted by this path, Index Data also offers, for a fee, MasterKey, a hosted, fully customized and configured federated search tool. (It is also available from CARE Affiliates or LibLime.)

dbWiz is a MySQL and Perl-based federated search tool. It is part of a larger suite of tools called reSearcher, which Simon Fraser provides for managing electronic resources, and works with Simon Fraser's Godot OpenURL resolver. dbWiz is used by several libraries in Canada and is being actively developed to add new search targets.

LibraryFind is a MySQL, Ruby-on-Rails-based federated search tool. It can search Z39.50 databases, Open Archives Initiative (OAI)–capable databases, and OpenSearch-capable data resources. Unlike many federated search tools, LibraryFind has a built-in API, which allows developers to create their own interface or use LibraryFind search results in unique ways. The software is also capable of querying the API of an OpenURL resolver; determining whether or not full text is available; and creating a link directly to that full-text object.

While many people think the difficulty of implementing an open source federated search tool is in the installation, the real challenge is the configuration of search targets, or “connectors.” This means collecting connection information (host, username, and password) from vendors as well as mapping metadata gathered from each database to fields used by the tool. Getting this information correct can be a time-consuming process and requires a partnership between metadata and technical staff, since both skill sets are necessary.

In our case, this step has taken the longest. Nearly all of the databases we can set up for federated search have been input into LibraryFind, but many of the configurations have not been thoroughly tested. Thus, users don't always find full text for an article when it is available, nor realize we have a print copy of an article, or have the proper citation information for an article. Obviously, we attempt to correct these problems as they occur, and we ask subject librarians to examine the results and report any errors that are returned by newly built subject-oriented search collections.

Prior to the integration of a federated search tool into the University of Houston library's web site, our students regularly attempted to use web site search engines to try to find research articles and failed to retrieve meaningful results; our new search tool has reduced this tremendously. While LibraryFind (or any other federated tool) isn't perfect, going open source has given us the ability to tinker and make improvements and has allowed us the flexibility to respond to a continuously changing environment on our own terms.