PRONOM—A Practical Online Compendium of File Formats

Jeffrey Darlington
The National Archives (UK)

There is a pressing need to establish reliable, sustained repositories of file format specifications, documentation, and related software. We recommend the establishment of such repositories for format-specific materials related to migration as a preservation strategy.”

Risk Management of Digital Information: A File Format Investigation,
Council on Library and Information Resources, Washington, D.C.,
June 2000

The problem faced by the National Archives and others who aim to preserve history by preserving records is that records are not always designed for longevity. They may be as ephemeral as messages written in the sand at low tide, or as permanent as the inscriptions on granite monuments. It is ironic that the primitive technology of ancient times has produced records lasting hundreds of years, while today’s advanced electronic world is creating records that may become unreadable in a few years’ time.

The correct interpretation of records has always required knowledge of the language in which they were written, and sometimes of other subjects too - medieval penmanship, for example. Fortunately enough of this knowledge has survived that we can make sense of most of the records that have come down to us. Modern technology has further complicated the problem of interpretation by making the viewing of records dependent on hardware and software environments whose own longevity is doubtful. Just as interpretation of the 1086 Domesday Book depends on the dictionaries and grammars for medieval Latin painstakingly compiled by long-dead scholars, interpretation of contemporary electronic records in the future will only be possible if the necessary methods and tools are compiled, documented, and preserved now.

When, in the evolution of computing, punched cards were superseded by magnetic tape, computer records became invisible and readable only by computers. Preservation programs were started up as early as 1962, and it was soon recognized that the incompatibility of tape formats would complicate the task of preservation. This realization led to the development of the ASCII character code, and other standards for media and recording formats that have served the preservation community well over the years. The new problems generated by the evolution of the personal computer and the word processing application were not so readily recognized. The new tools were seen at first as merely a method of creating paper documents that could be archived on paper. File formats were not standardized and soon proliferated alarmingly, every new software product having its own format.

Dimensions of Incompatibility

Besides the incompatibility between products, there is the time dimension. For each product, new versions of its file formats quickly superseded earlier ones. As more and more facilities were added, the formats became more complex. Sometimes the old format was a subset of the new, giving forward compatibility, but often the appearance of an old record was not rendered correctly by new versions of the software. These problems have multiplied as records have evolved from simple texts to complex assemblies of diverse elements, including embedded formulae, images, and charts. Web sites go a stage further with animations, video clips, and dynamic content. There is no paper analog that could be archived in these cases. And yet another dimension is that of dependencies. There are layers of application software that depend on operating systems that in turn depend on hardware.

Anyone responsible for managing and sustaining access to electronic records, even over relatively short timescales, must deal with these challenges to ensure that valuable records are not left stranded in formats that are no longer supported. One approach to a solution is to migrate records from obsolete formats into current ones, ideally into formats with published standards. Another is to preserve records in their original formats and keep copies of software products that can interpret those formats. To preserve the ability to run those products, copies of operating systems must also be kept. And in principle, old models of hardware must be either preserved (the museum approach) or emulated in software. The emulation approach is described by Rothenberg. Technical information about file formats and the software products that support them is a prerequisite for any digital preservation regime.

Introducing PRONOM

The National Archives is committed to preserving historic electronic records indefinitely, and has embarked on a program to make this feasible on a practical level. One strand in this program is PRONOM, our database of file formats, and its supporting library of software products. This collection of electronic materials is aimed initially at helping with the problem of software obsolescence, which seems the most urgent of the problems facing the preservers of electronic records. Hardware and media obsolescence are also to be addressed in future.

The first version of PRONOM was developed in March 2002 in parallel with the development of our Digital Archive system. It was designed to hold reliable technical information about the nature of the electronic records to be stored in the archive. For example, for Microsoft Word 97, PRONOM will tell you when it was launched, by whom, whether it is still on the market and whether it is still supported, what formats it writes, and what formats it reads. Interest expressed by other national archives led to the concept of distributing the database on CD, and PRONOM 2 was released in December 2002 to provide support for multilingual versions of the system.

The system was designed from the outset with a Web-based user interface and XML system interfaces, to conform to the UK e-Government Interoperability Framework. The latest version, PRONOM 3, is now being launched on the Web to make it available to the whole preservation community. We have simplified the user interface in the light of our own usability testing. The main search page allows the user to look up a file extension and see all the formats PRONOM recognizes with that extension, some extensions being shared by a number of products. It also allows the user to search for potential migration paths for a given format.

Content Development

In parallel with system development we set out to collect information about file formats from software vendors and from the Internet. This turned out to be more difficult than we expected. Vendors did not generally have systematic records of their old formats that were easily available. Many software vendors have been taken over by companies that have been bought up in their turn, and contacts for their products are difficult to find, especially if the product has been discontinued. A number of enthusiasts have Web sites containing collections of file format information, but their coverage is incomplete, particularly for vendor and product details. The Representation and Rendering Project (PDF) at the University of Leeds provides a useful survey of these sources.

We also began to build up our library of software products. This was an easier task, and it turned up a useful new source of information. The boxes in which distribution CDs are packed often display information about operating system dependencies and compatibility with other products that is not published elsewhere.

Since that time, considerable effort has been devoted to the collection of PRONOM content. National Archives staff have undertaken intensive research and liaison with major software developers in order to create an initial core data set of software product information. Microsoft has been particularly helpful. For the initial data load the focus is on the most commonly used office products for PC operating systems from PC-DOS onwards. We intend to load information on about 450 products over the next few months. The content development work is ongoing, and we have at least some information on over 3,000 file formats yet to be verified. We encourage software developers and others to be proactive in providing information - there is an online submission form on the PRONOM Web site.

Preservation Strategies

Information on file formats read and written by a product supports the migration of data by suggesting migration paths. However, this data needs interpretation and validation to be practically useful. Ideally, migrating a record into a new format is done at an early stage in its life and by the original author, who can check that the essentials of his or her work have been preserved. Later migrations are more risky, especially if more than one generation of format has intervened and the original author is no longer available. Unfortunately these are the conditions under which an archive may have to work. For example, there survive large collections of documents created by the early word processor WordStar that need to be preserved. Other word processing products may claim to read WordStar files, but how can we be certain that the full contents of the migrated document are captured?

We need a measure of how much the information content of a record is altered as a result of a particular migration process. Information content includes formatting, and functionality that is integral to the record rather than to the creating software (the difference between a hyperlink in a Word document and the fact that Word includes a spell check function). The design of PRONOM allows us to keep a measure of the “content invariance” of a migration path. Our intention is to define an objective and rigorous methodology for testing migration paths to measure content invariance, and to record this within PRONOM for each format that a particular product can read.

Migration paths for WordStar data are complicated because early DOS versions of WordStar used seven-bit ASCII characters, the eighth bit being used as a line wrap marker. When viewed by later products, these characters are wrongly interpreted as eight-bit ASCII equivalents, and to achieve a successful migration it is necessary to strip out the marker bits from the WordStar files. Since line wrapping is handled differently in later products, the loss of the eighth bit normally makes no difference, and at worst causes the text to be adjusted to different margins. This example shows the part that detailed technical knowledge plays in implementing a workable migration strategy, and also the importance of keeping the original bit-streams.

The alternative to migration is to have the original bit-streams interpreted by preserved software from the software library. To support this strategy, it is necessary to preserve old operating systems and the ability to run them. Fortunately the dominance of the PC platform and Microsoft operating systems makes this approach practical for many collections of records, and the compatibility between successive PC models means that a PC emulator is not always needed. However, some older operating systems are dependent on a 16-bit architecture, and future PCs may eventually outgrow the current 32-bit architecture; emulation will be needed to overcome these problems.

Technologies are constantly evolving, and archivists must be aware of these changes and the implications for the electronic records they are preserving. As old software products cease to be supported and become obsolete, preservation activity will be needed for the file formats that depend on those products, whether by content invariant migration or by preserving the software. A “technology watch” process to identify triggers for preservation actions is a component of the PRONOM program.

There would be many advantages in migrating records to an XML-based standard format. The development of practical tools for this task depends on detailed technical descriptions of how each format actually works. This information is beyond the present scope of PRONOM, but we plan to collect it for the next major release. A subset of this information would be useful to develop tools for the recognition of file formats, a function included in our Digital Archive and at present provided by commercial viewer software. We expect to include this recognition function in a later version of PRONOM.

The Web-enabled PRONOM 3 completed this month, marks the latest stage in the evolution of the system. At the same time, it is a starting point for the development of PRONOM as a major shared online resource for the international digital preservation community. The National Archives has plans for major enhancements over the next few years, including the development of a number of specific tools to support digital preservation activities. Further information about our future plans is available on the National Archives Web site. We hope that through our continuing research and contributions from the Web community, the content of PRONOM will expand to give a comprehensive coverage of file formats that will support worldwide preservation initiatives.

Acknowledgments
Thanks to Adrian Brown, Jo Pettitt, and Rob Taylor of the National Archives Digital Preservation team.

Publishing Information

RLG DigiNews (ISSN 1093-5371) is a Web-based newsletter conceived by the RLG preservation community and developed to serve a broad readership around the world. It is produced by staff in the Department of Research, Cornell University Library, in consultation with RLG and is published six times a year at www.rlg.org.

Materials in RLG DigiNews are subject to copyright and other proprietary rights. Permission is hereby given to use material found here for research purposes or private study. When citing RLG DigiNews, include the article title and author referenced plus "RLG DigiNews." Any uses other than for research or private study require written permission from RLG and/or the author of the article. To receive this, and prior to using RLG DigiNews contents in any presentations or materials you share with others, please contact Jennifer Hartzell , RLG Corporate Communications.

Please send comments and questions about this or other issues to the RLG DigiNews editors.

Co-Editors: Anne R. Kenney and Nancy Y. McGovern; Associate Editor: Robin Dale (RLG); Technical Researcher: Richard Entlich; Contributor: Erica Olsen; Copy Editor: Martha Crowe; Production Coordinator: Carla DeMello; Assistant: Valerie Jacoski.

All links in this issue were confirmed accurate as of October 15, 2003.