The Internet Archive, an Interview with Brewster Kahle

Automated Digitisation of Printed Material for Everyone: The METADATA ENGINE Project, by Günter Mühlberger

print this article

We have gotten more usage than I thought we would. We get 20,000 different users each day. There are now parts of the Web that link to the past. What does it mean to have the past as part of the present? The Wayback Machine injects the past into the present. The Web is a self-documenting, self-cataloging machine. Unlike books, with the Web, the catalog and content are part of the same thing, but not completely.

We were first on the Yahoo Internet Life list of top 100 Web sites for 2001. Der Spiegel named us "Hit of the Year," and we received the 3rd Digital Archive Award in Kyoto, Japan. If you look at use by country, the greatest number of hits comes from Japan. We've hit a nerve. People care about their own history. They're psyched about it. On the Web, anyone can be a publisher, now there is a library for their work.

Also, the site has been used in more situations than expected. It has been interesting all around. For example, a group of Masters students at the Berkeley School of Information Management and Systems this spring presented the results of a series of interviews on why people use the Wayback Machine. They discovered that looking up personal Web pages is a major use. "Did I make it into history?" people wonder. After that, organizations look for their sites. I thought people would poke around, have some fun. It provides a reference service.

There are currently seven full-time employees and 14 interns at the Internet Archive. It would be an understatement to say that we did everything with that number. We contract with many organizations. Alexa Internet has thirty people, but not all of them are working on the Internet Archive. Compaq actively supports us. It's a ten million dollar per year operation to do the tasks we're currently doing.

Oh, yeah. The dataset we have is a queryable Web collection, which makes it one of the largest databases ever made. There are up to 200 queries per second. That combination, we believe, is unprecedented. Our claim to fame is our level of frugality. We don't have a lot of money.

We need to be willing to use innovative tools and techniques, to try the very newest ideas. The classic technology approach would be to buy a database and implement RAID (Redundant Array of Independent Disks). For 1 terabyte (TB) you'd be looking at $400,000 for EMC disk storage, $150,000 for a Sun box, and $100,000 for an Oracle database. Effectively, that would be almost a million dollars for 1 TB. Like Google and Hotmail, we use a large number of PCs in parallel. Our technology has to be state-of-the-art to keep up. We collect 10 TB per month. Our approach has to be cost-effective and flexible. For instance, we moved off tape to removable hard drives.

All our special collections have been done in partnerships. Someone else helps with the curatorship, selection and quality assessment. The 1996 Election collection became a kiosk in the Smithsonian. They thought it possible that the Web might be the next "bumper sticker" in campaigning. The Election collection grew into a big project with the Library of Congress for 2000. There were lots of questions—"could you?" "should you?" "are you allowed to do this?"

We see special collections growing through partnerships into national collections. It is inconceivable for countries not to record their digital heritage. A lot of history is born digital. This should not be like early television where there is not a record.

The Web is effectively infinite. If you start crawling, you'd never end. How big is our collection? Enormous—really big. We want to increase access—researchers' access—and have them help guide the collection. We have proven you can do something and it can be useful. We already have some of the deep Web—where the crawl can generate each page for capture. What we want now is complete collection of important things. We want more critiques of collections from researchers, historians and scholars—like "'it would have been great if…" These comments will inform the collections.

We check for robot exclusions and retroactively exclude content for those sites. There are some classes of people that have opted out. Many newspapers do not want to be included. Individuals are all over the map. Photographers feel disadvantaged; they don't feel that being part of it will help them. This represents a very small percentage. We do not document the removal of content, but we collect robot exclusions. At first, we didn't collect them. Ben Edelman, a researcher at Harvard Law School, is looking at the relationship between domain name ownership and robot exclusions.

There are more pages now, but we do not have a more recent number. The Internet Archive receives most of its Web pages from donations from Alexa Internet. They do a new crawl every two months. We crawl for special collections weekly or daily. If we discover a new page, we crawl it within 24 hours. We do a complete sweep every two months.

The crawler technology is constantly being rewritten. Every 12-18 months Alexa throws out the existing crawler and starts over from scratch. Revisions to the crawler reflect changes in the structure of the net, its size, the underlying technologies, etc. What we have now is the best available crawler.

All of the laundry from the past is shown to everybody in this collection! Some gaps were the Internet Archive's fault and some were not. Some missing images are due to robot exclusions. The technology has changed. From 1996-1998, the crawler crawled a full Web site or as many pages as it wanted in one day, so there'd be a clean copy. Other times, it might follow up later—many days later. Different crawl philosophies were used. The 1999 crawls do not contain a lot of images because we did not have enough bandwidth for text plus images. There were months when there was no crawling at all while the crawler was being rewritten.

Our terms of service are that users respect other peoples' privacy and copyright. You could interpret that as "don't copy from the Wayback Machine or the Unix archives collections without talking to the Internet Archive." We have robot exclusions on the Internet Archive collections. We do make case-by-case exceptions.

We haven't supported OAI (Open Archives Initiative) yet, but we want to. We work with users. We want to be as standards-based as possible as long as people would use it. We are willing to be an OAI data provider or service provider.

I guess not. Should we? We would like to comply with prevailing standards. We are currently reviewing the RLG/OCLC Trusted Digital Repositories white paper.

Once archived, we never change a page. The Web wasn't constructed to be archived. It's so interconnected. A book exists outside of time. Archiving Web sites is like putting together a bomb after it has exploded. We do as good a job as we know how. If anyone knows how to improve it, please let us know or help us change it. We can't force people to archive in a certain format. The Wayback Machine is the best way we know to look at the results.

File format changes haven't become a problem for us yet. Most old HTML is still supported. Sometimes we copy files forward. We store things in the exact way they were gotten. We are very careful at ingest to record all metadata. When using the Wayback Machine the links on a page are changed on the fly to point into the past. You need to watch the URL when you're surfing the Wayback Machine to answer the question—"When was this page archived?" We tried putting a frame around the page, but that is technically hard to do. Users, we are afraid, forget to look at the URL, but we don't know a better way.

We haven't approached a standards body at this time. It's a lot of work. I was in charge of an IETF group. I found that a couple of people working together works best. Put something together and throw it out there. If people pick it up, it's worth going through the headaches. Some others are doing Web archiving, but with their own tools. We have started active outreach to other projects to see if we can share tools, formats, and experiences.

Mirrors? Yes. We donated a complete copy of the collection to the new Library of Alexandria. I spent April 2002 in Alexandria, Egypt. It was a kick to see the machine it's on—it's front and center when you walk in. The best method for preservation is replication. Put copies in as many different contexts as possible—one in RAID, bury a copy. Put a copy in someone else's hands. They will care for it differently, and take care of some bugs. Usually books are destroyed because the new regime is not interested in the old. We learned a lesson from Alexandria; we want to place duplicates on other continents. Redundancy is antithetical to current culture. Libraries often strive to create unique collections that are proprietary. We are less proud of collected materials if there are other copies. We need to show positive examples to change the way things work.

Yes, we have a sustainable funding model. Everything that has been donated is safe. There is enough money to make sure that it's safe forever. Considering Moore's law—what we have is sustainable for the present, but access is more expensive. We need to blend with the research community. We need to put our hand out for support to further the work on access.

There has been almost no response to pledge unless you badger. The real money has come from private foundations, government support, and in-kind donations from corporations—Zoe Baird and the Markle Foundation, NSF, and LC. We need to expand the list by making this part of peoples' lives, to open new doors.

We are not bringing the cases. People are using the Internet Archive as an example of how they see the future. Peter Lyman suggested in his paper "Why Archive the Web?" that the Internet is the information resource of first resort for millions of readers. Some say the good stuff is not on the Net. I have a sinking feeling that if the young are learning from the Web, and the Web is not offering the best, that we are shortchanging our youth. This is criminal. It makes no sense. The expectation that materials for learning should be on the Web brings urgency to the issue. We—the establishment (face it, we are)—are screwing up. We do not need more meetings on metadata. We need to get our cultural heritage easily accessible on the Web. Newspapers are up there. Academic journals are becoming available. Books, music, videos, and TV are trying but they are suing their way into the next century—or the last one. That all this stuff ends up in the courts is dumb; that approach doesn't make sense. Raj Reddy promotes "universal access to human knowledge." Some for free and some for pay. If we take as our premise that all information should be at our fingertips, how do we get there? The public library system is a $25 billion industry. $5-6 billion goes to publishers. If students aren't consulting with the Library of Congress daily then let's get on with it. If we wait for 20 years, we could shortchange a human generation. Kids could grow up without the best works of the 21st century. Our libraries contain large amounts of pre-20th century materials which could all be online now because there are no rights issues. Raj Reddy says we are moving too slowly. We are trying to do the million books project. A million books for free, great. India and China are in, great. The US, not really—Cornell, Math—these are dribs and drabs. Kids are demanding a different way. RLG and the research libraries—these are rich institutions and they can be doing much more. The technology is not hard. Raj Reddy says it takes 2-4 hours to scan a book. Not the most interesting 2-4 hours, but then that book can be offered to the world. Gutenberg showed how it works. We need to do our job.

We want to grow our collections, but grow them in ways that they are useful to traditional library users—researchers, scholars, and the underserved. On the Web we can put tools and technology on top of collections—a search engine to answer harder questions. We can bring tools and information together in new ways that weren't possible before the Web. We think we have an archive, we want to build a digital library. We need partnerships to do this. We have collections but not a lot of finding aids. The top-level best thing for the community is universal access to human knowledge. It is within our grasp. We need to coordinate our efforts and just do it. I leap out of bed in the morning fueled by this possibility. We have the ability to touch people—like poor kids in Yemen. When the goal sinks in, every day is surprising. It straightens the road, it's not curvier. We can leave a legacy which will make our grandkids proud of us. This is not fanciful. We have technology for storing, for accessing. We have the political will to live in an open society. Libraries are now stepping up to the challenge.

print this article

• make digitisation more effective in terms of costs and resources needed
• automate the whole conversion workflow and especially metadata capture by applying layout and document analysis algorithms
• provide a standardized output that is compliant with emerging standards
• increase the added value of digitally reformatted material

Table 1. Metadata Creation During the Conversion Process

The conversion process begins with page images (e.g., TIFF files or other formats), that are scanned with the METAe engine or that are already available on a file system. At the same time, existing descriptive metadata from MARC records can be imported and integrated directly into the workflow.

The first step in metadata creation is to record administrative information such as the type of scanner, the file format, the date of acquisition, the person who has carried out the scanning, etc.

The next step is to create structural and descriptive metadata for the content of the converted document.

Structural metadata are recorded from a physical as well as from a logical point of view (4). From the physical point of view, we are concerned with such questions as: How are the bitmaps distributed over a given page image? Do they belong to textual or graphical zones? What coordinates do these elements have? At a more detailed level, we can ask: Where are the zones, lines, or even words, within the page image? What font size does a word have? Which alphabet (Latin, Greek, black letter, etc) is used? Once the conversion process is completed, this physical view will, in principle, allow a 1:1 reconstruction of a given document.

However, the structural metadata connected with the logical or intellectual dimension of a document is much more important than the physical description. The body text, with paragraphs, footnotes, margin notes, appendices and the like, forms the intellectual content of a book and needs to be recorded in detail. One of the main features of the METAe engine is its ability to create structural metadata automatically, based on a systematic analysis of the document and its layout.

An article in a journal may contain text, photographs, and drawings that are all part of the structural map of the document, and yet these components may also be valuable intellectual items in their own right and therefore should be described separately as well as in the context of the larger document.

For each document, all metadata elements, as well as their relation to each other, are recorded in the internal database of the METAe engine. This database produces a generic XML output file that can be configured to serve the particular needs of a digital library management system. Even so, the project team decided to support at least one preferred output schema, and has voted for the METS schema (5).

The standard output file of the METAe engine is therefore designed in the following way: The METS file is the surrounding bracket within which descriptive data is either referenced, e.g., to an existing MARC record, or labelled according to Dublin Core. Administrative metadata follows, in some respects, the specifications set up by DIG35 (6). The structural metadata, on the logical level, is formed according to the guidelines provided in the METS schema. The metadata describing the physical dimensions of the document is stored separately, using the so-called ALTO (Analysed Layout and Text Object) file, the structure of which has been developed by the project team (7).

Architecture

Workflow Component

The METAe engine consists of a workflow component and a database of rules. The workflow component and its related interfaces enable the user to carry out the whole digitisation process, including scanning, image-processing, physical and logical analysis, quality control, configuration, and administration. A graphical user interface (GUI) allows the user to verify and to correct all automatically processed metadata. The workflow can be configured in a flexible way, i.e., by doing one procedure after the other, or by checking selected pages and elements at crucial steps of the process. The engine will run on the Windows platform either on a single workstation or in a network environment.

The main user interface for interacting with the system consists of two parts, the first of which is a frame for displaying the physical and logical structure of a document, as indicated in Figure 1 below. The document can be browsed on the level of hierarchies, or on single elements. In the second frame, the physical representation of the logical element is presented. In the case of a chapter, all pages relating to the chapter are displayed. In the case of a picture, the related page is shown. In order to provide a better context, the elements are highlighted with different colors, e.g., yellow for running text, green for a picture.

Figure 1. Thumbnail of a screenshot of the METAe GUI
(click on image to enlarge)

Database of Rules

The second module is the core of the METAe engine. It is not visible to the user and consists of a database of rules designed to automate the digitisation process. In order to create effective rules, a "grammar of books and journals" has been set up. Our approach (8) is based on the assumption that documents are semiotic systems with a special syntax that can be modelled by applying rules derived from the layout of books. Even though the METADATA ENGINE project focuses on books and journals, it is obvious that the database of rules can be extended to other documents such as flyers, newspapers, manuscripts, magazines, posters, handbooks, encyclopaedias, or finding aids. At present, all basic rules are being implemented in the database. Our first results are highly encouraging. Data about the effectiveness and performance of the METAe engine will be available once the validation phase of the project has been completed.

Features

Cropping and Splitting of Pages

Although it might be a good decision to cut bound documents and to scan the leaves of a book one by one on a flatbed scanner, not all libraries will vote for this option. An alternative might be found in overhead scanners or in a completely automated scanning machine (9). These scanners will provide double page images that look more or less like the following:

Figure 2. Cropping of Single Pages by Utilizing the Print Space of Books

These double page images need to be split, the single pages cropped, and, optionally, they may be adjusted and deskewed. In the METAe engine the whole process will run automatically. Books are printed according to a clearly defined printing space. Only a limited number of special elements may appear in the surrounding margins. Therefore, the engine will first determine the coordinates of the print space used in the given document and then apply this zone to the actual page image. Next it will add a virtual margin around the printing space and cut the pages. If a document contains supplements that do not conform to the default print space, such as maps, tables, graphs, and pictures, this variation will be detected automatically.

Dynamic Binarization

The most important step in the digitisation process is to create the best image file possible, since it will constitute the basis for all further steps. Both the METAe layout analysis and the OCR engine rely on good image quality. In accordance with the guidelines recommended by the DLF (10), there are some circumstances for using 300-400 dpi grey-scale (8 bit) scanning instead of 600 dpi b/w. Grey-scale scanning may provide better results in the OCR and layout analysis process in the METAe engine, by applying a dynamic binarization feature that handles different parts of the page image at different thresholds. This improves the recognition rate for OCR remarkably. We also have to take into account the fact that, from the 1880s onwards, many documents contain halftones that cannot be digitized in a satisfying way unless grey-scale or color-mode scanning is conducted (11). In the example below, the big advantage of the METAe engine is that the whole book can be scanned in grey-scale in one single pass. The detection of the images and the dynamic binarization of the textual zones will be done afterwards automatically by the METAe engine.

Figure 3. Dynamic Binarization and Detection of Graphical Zones

Matching of Page Numbers and Image Files

In order to support the basic features of a digital library Web site, correct matching between image files and page numbers is imperative. However, page numbering is rather complicated, as in many instances there are pages within a book that are not counted, and others that are counted but do not show numbers, or that show roman numerals.

As mentioned above, it is the document as a whole, i.e., its overall syntactical structure, that is analysed by the METAe engine, allowing for a highly automated solution for matching images and page numbers. The engine will first find out where the page number is usually located on a page, then the whole row will be extracted, and after that the right sequence will be reconstructed. Pages that have been counted but do not display page numbers can have page numbers added automatically. Missing pages can be detected and marked with a placeholder, and also brought to the attention of the operator.

OCR Processing

The OCR engine is a distinct module within the METAe engine. Every kind of OCR engine available, or even more than one, might be used. Nevertheless, it is one of the objectives of the project to develop an OCR engine with improved recognition rates for historical documents. Typefaces used between the 16th and 19th centuries are, in many instances, considerably different from those in use nowadays. Since OCR engines have been trained for modern typefaces, this fact will lower the rate of correctly recognized characters for older documents. Moreover, the vast majority of all printed historical documents in central Europe were set in the German variant of the black letter font, Fraktur.

Figure 5. Page Image from Scientific Journal (1930)

Figure 5 shows a typical page from a scientific journal from 1930. There is a running title, a page number, graphs, caption lines, a footnote, and a signature mark. The output of a pure OCR engine is shown in figure 6.

Figure 6. Raw OCR Text of the Page Image of Figure 5

This electronic text is not readable and cannot be presented to an end-user since the raw OCR text has a flat structure and contains the complete text independently from its hierarchical level or logical value. Assuming that the METAe engine has correctly labelled all elements on this page image, it will be easy to design a Web application where only the intellectual item—such as the article shown above—is presented to the reader and where the other elements, such as the running title or page number, are either not shown or are presented in an adequate way, e.g., footnotes laying in the back of the text. This "cleansed" electronic text must not be mixed up with a real corrected OCR text, as is usually done with double keying procedures.

Figure 7. Cleansed instead of corrected OCR text
(click on image to enlarge)

print this article

An integral assumption in our research is that different record types have different preservation and authenticity requirements. Records ingested into the Testbed are analysed in terms of the five attributes posed by Rothenberg for digital records: Context, Content, Structure, Appearance, and Behaviour. Authenticity Requirements are developed for each record type in terms of these five attributes and act as the success criteria for an experiment's preservation approach. In addition to this, other preservation and archival issues arise from these experiments. Our research thus extends to wider considerations and includes objectives beyond the success of a technical strategy.

Objectives

The objectives of the project are to provide insights into:

• Technical solutions for the preservation of authentic electronic records
• The effectiveness of current and potential preservation approaches
• Authenticity features of digital records
• Cost factors for storing, preserving and managing digital records and associated metadata
• Management processes and activities required to capture, generate and maintain metadata that support the ingestion of records and preservation of long-term access to authentic electronic records

• What are the advantages and disadvantages of implementing each of the specified preservation approaches?
• What are the factors that affect the effectiveness or appropriateness of a particular preservation approach?
• What are the basic requirements for preservation functions?

• What are the options for preserving record attributes?
• What factors affect the preservation of record attributes?

Figure 1. Flow chart of Testbed Project Experiment Process

Chilling Effects: Monitoring the Legal Climate for Internet Activity This Web site was launched in February 2002 by the Electronic Frontier Foundation (EFF) and four law school legal clinics, representing Harvard University, Stanford University, UC Berkeley, and the University of San Francisco. The list of participating clinics is expected to grow. The site provides detailed information about Internet users’ legal rights in their online activities. Legal topics covered include DMCA (Digital Millennium Copyright Act) and other copyright issues, domain names and trademark law, anonymous speech issues, and defamation. For each topic, the site provides a basic legal overview, basic guidelines, FAQs, and related links. The most distinctive feature of this site is the “Cease & Desist” database. Chilling Effects Clearinghouse has encouraged Internet users to post cease-and-desist letters sent to restrict their online activities. Law students review the collected letters and annotate them with related resources. The letters are searchable by subject or keywords. Each search result includes a cease-and-desist letter, related FAQs and related links. This Web site allows librarians and archivists track online legal trends, justify their own online activities, and, most of all, to defend their legal rights on the Internet. In addition, this site provides a connection for groups and organizations that are threatened by the same kind of cease-and-desist letters. With its focus on intellectual property laws, this site is an invaluable information source.

print this FAQ

FAQ

Where are they now? Digitizing Microfilmed Newspapers

We often read about new projects and programs in RLG DigiNews, but what about past efforts? What results have been produced in the five years since RLG DigiNews began publishing?

Introduction

In this issue, we continue a feature started in our five-year anniversary issue (April 2002) and take another look at projects originally reported on during our first year of publication. In August 1997, an article in RLG DigiNews by Alan Howell discussed three International Newspaper Film Scanning Projects. The projects profiled were the scanning of the Burney Collection at the British Library, the Caribbean Newspaper Imaging Project of Cuban and Haitian newspapers at the University of Florida, and the scanning of mid-19th century periodicals and newspapers for the Australian Cooperative Digitization Project (ACDP).

As source material for digitization, newspapers are amongst the most challenging. Large page size, tiny type, halftone images, haphazard layout and poor bibliographic control are commonplace. Newspapers scanned from microfilm introduce additional difficulty in terms of image quality, processing, and sometimes cost. These factors complicate capture, delivery and access control and lead to a variety of technical and management obstacles.

Given the technological advances that have occurred in the past five years, we thought it would be instructive to revisit these projects to learn where they are today, what lessons have been learned, and how the state-of-the-art has changed. We were able to obtain updates on two of the three original projects, the Burney Collection scanning initiative and the Australian Cooperative Digitization Project. The Caribbean Newspaper Imaging Project (CNIP) is in the midst of a "technological renovation" (migrating from CD-ROM to Web) and was unable to meet our publication deadline. However, project reports from two earlier phases are available on CNIP's Web site (1).

Scanning the Burney Newspaper Collection at the British Library
Contact person: John Goldfinch, Early Printed Collections, The British Library

The Burney collection consists of 700 volumes of 17th-, 18th-, and 19th-century newspapers. It is especially prized for its coverage of 18th-century London newspapers, including many unique items. Owing to its age and condition, the originals are not available for study, and the popular collection is currently viewable only on microfilm.

The original digitization work on the Burney collection was actually a British Library (BL) experiment that started in 1992 with an effort to determine the utility of its Mekel microfilm scanner in digitizing the library's microfilm holdings. From 1993 to 1996, BL attempted to learn what digital technology could do for it, as part of its Initiatives for Access program. The Burney collection was chosen for testing in part because of the challenges it presented. The original documents varied substantially in density, both across and within pages, and much of the type is broken, leading to film described by Hazel Podmore in BL's write-up of its experiment as "not the best-quality the Library has ever produced (2)."

As expected, digitizing the Burney film proved difficult. According to John Goldfinch, the principle difficulties stemmed from the "expense and availability of high capacity storage for the files being created, together with the level of manual intervention required to deal with such things as deskewing the images, and coping with the highly variable print quality of the originals and deficiencies in the film."

In fact, Podmore noted that storage considerations led to most scanning being carried out at a less-than-optimal 200 dpi. Another technical obstacle, considered insurmountable at the time, was the inability of any available Optical Character Recognition (OCR) package to produce acceptable machine-readable text for searching and indexing. At the time, the difficulties were so significant the British Library concluded that it could not justify continuing digitization of the Burney collection.

BL feels that much has changed in the intervening years. Storage costs have dropped by about two orders of magnitude (about 100 times). John Goldfinch asserts that "recently demonstrated developments in OCR technology offer the exciting prospect that OCR is at last able to cope with the difficulties of early type." BL and other institutions have gained considerable experience in the scanning of microfilm of early printed material.

As a consequence, BL is revisiting the question of the Burney newspapers, and has received a grant from the National Science Foundation to begin creating a fully searchable on-line library of British 18th century newspapers. As it had originally hoped to do in the mid-1990s, BL now plans to produce a complete set of images along with an index of titles with issue dates and numbers, and to make the complete collection freely available to researchers over the Web. A precise release date is not yet available.

Australian Cooperative Digitization Project, aka the Ferguson Project
Contact person: Ross Coleman, Collections Coordinator, University of Sydney Library

The Australian Cooperative Digitization Project is a collaboration involving the University of Sydney Library, the State Library of New South Wales, the National Library of Australia and Monash University Library, amongst others. Ross Coleman and Colin Webb summed up the project's purpose as the "enhance[ment of] literary and historical research on nineteenth-century Australia by providing improved access to, and preservation of, scarce primary material confined to a few major library collections (3). The material selected for digitization, based on Ferguson's Bibliography of Australia, is confined to a critical six-year period in Australian history (1840-45). Though preceding the Australian gold rush, these materials represent a historical gold mine and provide a defining record of a distinct Australian colonial culture.

ACDP was carried out as a true hybrid project, insuring that preservation quality microfilm existed or would be created for every title, and that stringent quality control guidelines governed the creation of digital images from the film. Although conceived as an experiment with a strong mandate to develop policies and procedures that could be applied to other Australian digitization efforts, ACDP also carried significant production expectations. Sixty-seven periodical titles (including newspapers) and four novels were ultimately digitized.

In providing a retrospective to RLG DigiNews, as well as in previous summaries of the project (see http://www.nla.gov.au/ferg/about/ for several references) Ross Coleman and his colleagues have been unusually candid about the obstacles they encountered in bringing ACDP to fruition. Particularly noteworthy, and of value to anyone contemplating a digitization program of any size, is the excellent discussion in Webb and Coleman of the tug-of-war between workflow-enhancing automation procedures and the need to maintain sufficient quality for text legibility and successful processing via OCR (4).

For example, most of the existing film was not of sufficient quality for digitization, and even the refilmed material presented some barriers to fully automated digital capture. As is so often the case in library and archive projects, quality requirements ultimately ruled, but a heavy price was paid in terms of missed deadlines, tense vendor relations, loss of staffing continuity, and frayed nerves.

As might be expected, newspapers proved particularly troubling, especially because of their size. However, Ross Coleman identified additional obstacles stemming from "the variety within any one title, from foxing and discoloration, to the use of varying fonts and point sizes on the one page." Coleman also acknowledges that newspapers require quality OCR in order to truly justify their digitization, since they generally lack even rudimentary indexing. Unfortunately, marginal print quality and type size variation often thwart the creation of an accurate body of searchable text, even with current technology. (For example, ProQuest Historical Newspapers reports 80-90% OCR accuracy for the article text from its New York Times microfilm.)

Despite having successfully completed its initial objectives, ACDP exceeded its anticipated resource consumption to such a degree that conversion of additional 19th century periodical titles has been put on hold. Within the selection of periodicals, newspapers continue to be viewed as especially daunting targets for digital capture. Coleman reports "the fact that no more have been done, or even contemplated, highlights the fact thatat the timewe were not confident in the technology, or our procedures, or in the effectiveness in delivering such things over the Web in a usable manner."

So while ACDP has succeeded in greatly expanding access to a corpus documenting an important slice of Australian history, it has not, as yet, provided the basis for expanded conversion of other materials from that period.

Conclusion

In revisiting these two projects, we encountered somewhat different perspectives about the current viability of digitizing, OCRing, and providing Web access to microfilmed newspapers. One possible explanation for the differing opinions is the timing of the initiatives. ACDP started out as the Burney experiment (along with many other early digitization experiments) was wrapping up. Burney was conceived as more of an experiment, and was carried out at a time when the technology was clearly not up to the task. It was shelved until very recently, when the technology seemed like it might finally be able to tackle the challenge.

On the other hand, ACPD was conceived as a production enterprise, and was carried to completion despite knocking against technological barriers at several points. Having only recently completed the mounting of files, ACDP is still reticent about taking on additional conversion, given the technological obstacles it encountered.

It is noteworthy, however, that what most distinguishes Ross Coleman's perspective from that of John Goldfinchs has little to do with the technological underpinnings of the respective projects. Although both speak to the frustrations of digitizing challenging older materials, the most striking difference is in Coleman's emphasis on the obstacles created by management issues. Problems faced in the management arena remain underreported and under-discussed within digital imaging circles, compared to those in the technical realm. Even as some (though by no means all) the technological barriers to effective large-scale digitization of older printed materials begin to fall, we would be wise not to downplay the ongoing challenges represented by funding, staffing, vendor relations, planning, and the like.

Perhaps the ultimate lesson from the experiences described above is that there is still no such thing as a large-scale, cookie-cutter digitization project. Despite many successfully completed efforts and improved availability of training and documentation, the work remains technically complex, time-consuming, and expensive. Working from marginal source materials introduces additional complexities, and newspapers continue to push the limits of current digital capture, image processing, OCR, and Web delivery technologies.

Further reading

In addition to the references already given, here are some useful readings on recent newspaper digitization efforts:

The ProQuest Historical Newspapers project (backfiles of the Christian Science Monitor, the Wall Street Journal, the New York Times, the Washington Post and Canadian newspapers digitized by Cold North Wind ("practically every newspaper published in Canada from 1750 to 1950") with plans to add other national, regional and local publications). The home page provides links to a slide show about the project. An additional demo is also available.

OCLC Digital & Preservation Resources and