Digital Libraries: Bigger, Cheaper, Everywhere

I'm fond of saying that hard disk storage is cheaper than dirt. Certainly there is a kernel of truth in such a statement (have you priced dirt lately?). Standard consumer disk storage can now be had for less than a dollar a gigabyte, and the cost continues to fall.

While the price drops, available storage sizes increase. Earlier this year LaCie introduced a product called Bigger Disk that provides a terabyte (1000 GB) of storage for $1000. We'll remember 2004 as the year that an individual could reasonably buy a terabyte of disk storage.

The technically savvy will quickly point out that disk storage for the consumer market is not the same thing as storage for large enterprises. What works for saving your MP3s may not work quite as well for a bank responsible for saving financial transactions.

In a previous column (LJ 11/15/01, p. 26ff.), I discussed RAID (redundant arrays of inexpensive disks) technology as a method for preserving mission-critical data. Depending on the level of RAID selected, more or less of your storage is given over to replicating or preserving the data you store—ergo, your terabyte disk would no longer hold a terabyte but perhaps a good deal less. Other differences between consumer and enterprise-class disks include such things as the type of hardware interface, spindle speed (revolutions per minute), and the quality of components.

Storage options are not limited to hard drives. Although writeable CDs have been used as a common storage device for files that don't change often (like MP3 files), they are already giving way to DVDs that can store much more data.

Further out on the horizon are more amazing storage technologies, such as holographic devices that use light instead of the magnetic charges of current hard drives. According to the Economist ("Light on the Horizon"), holographic storage devices should be on the market any day. Since data are stored in three dimensions instead of the two of magnetic storage, the potential efficiency is much greater. In theory, it should be possible to store a terabyte of data on a CD-sized disk.

Another dimension is that data storage is now ubiquitous. Virtually every electronic device comes with more or less data storage—and frequently with the ability to accept storage cards to increase storage. The cell phone I paid nothing for has storage, albeit not much. My PDA has some, but it accepts Sony Memory Stick removable storage media, which adds 128 MB of storage. Similarly, I have one gigabyte of storage in my digital camera (able to store two weeks' worth of vacation photos, or 480 pictures, without a single download).

MP3 players such as Apple's iPod often have large capacities (e.g., 40 GB) that can store just about anything. And, finally, we have devices that have truly made storage ubiquitous—the USB flash drive. These storage devices can store increasing amounts of data (e.g., two GB) in a form no bigger than your finger and portable enough to put on your keychain. These days you can carry more storage on your person than the typical personal computer had several years ago.

The main lesson of massive and inexpensive disk storage is that disk storage is no longer a credible barrier to anything. If your system administrator is crying the blues over the size of your database, or (in extreme cases) giving you grief over the size of your mailbox, fork over $20 and tell him or her to go buy more. Lack of storage isn't a reason to say no to anything you wish to do.

Another lesson is that library users have significant data storage on their persons, even if no backpack to carry such a device is apparent. Libraries should consider ways to allow users to carry away easily what they've been working on at library computers (e.g., easy access to a USB port). Academic libraries may want to offer a service that would download a library of reference books to a student's PDA. There are any number of library service enhancements that may be possible given pervasive and massive storage. One of the best ways to begin imagining these enhancements is to have these devices yourself.