1. Introduction

Digital publishing is causing publishers, research institutions, and libraries to develop new policies, new infrastructures and techniques, and new business models as well. A major problem is that, at the same rate at which our world is becoming digital, digital information is threatened. New types of hardware, computer applications, and file formats supersede each other, making our recorded digital information inaccessible in the long term. The Koninklijke Bibliotheek (KB) has, jointly with IBM, developed and implemented an OAIS-based deposit system: the e-Depot. Moreover, the KB signed archiving agreements with major scientific publishers for permanent storage of their digital materials. An important issue in digital archiving is long-term access: how can we guarantee permanent access to digital publications while software and hardware are constantly changing? This issue strongly relates to the object’s life cycle management, as ineffective life cycle management might compromise availability of the digital object in the long run.

In this paper, we discuss life cycle management issues as they relate to two prominent digital preservation techniques and associated costs: migration and emulation.  We argue that applying the emulation strategy may be more efficient in terms of life cycle management (and thus costs) than the migration strategy. We introduce the KB e-Depot, in which the main workflow is described, discuss the two main digital preservation strategies, and relate these strategies to life cycle management and cost issues.

In 1999, the KB specified the system requirements for a full-scale deposit system, which were based on the ISO 14721 standard for digital archives: the Open Archival Information System.¹ As a result of a European tender procedure in 2000, the KB contracted the development of the deposit system to IBM in the Netherlands. In December 2002 the system was delivered to the KB. IBM constructed the system using as many off-the-shelf components as possible, such as WebSphere, DB2, Tivoli Storage Manager, and Content Manager, and branded it under the name Digital Information Archiving System (DIAS). Using DIAS, the KB maintains the deposit service called the e-Depot. (See Oltmans & Van Wijngaarden, 2004² for a complete description and Steenbakkers, 2003³ for more details about the history of the KB e-Depot.)

The KB has developed a workflow for archiving electronic publications and has implemented the other parts and interfaces of the infrastructure in which DIAS is embedded. This infrastructure consists of a variety of functions for:

• validating and pre-processing electronic publications,
• generating and resolving unique identifiers,
• searching and retrieving publications, and 
• identifying, authenticating, and authorizing users.

The process of loading consists of pre-processing and ingesting the digital content. Electronic publications are stored in the e-Depot on offline media such as CD-ROMs (also referred to as “installables”, cf. Oltmans & Van Wijngaarden 2004² for more details) and online, most often electronic articles. Online journal articles are either sent to the KB on tapes or DVDs, or they are fetched by means of FTP. In both cases, publications ready for ingest end up in an electronic post office. At this stage the content of the submitted publication is validated in regard to its authenticity and well-formedness, based upon earlier agreed upon specifications. If the material does not match the checksum (or if other errors occur), the content is passed to a database for error recovery. If the content appears to be valid, content and metadata are combined to form Submission Information Packages (SIPs). These SIPs are then processed by DIAS. See Figure 1 for a complete overview of the data flow.

Figure 1: General e-Depot Data Flow

DIAS ingests both the content and the metadata, converting the publisher’s bibliographical descriptions into the KB internal format and adding a National Bibliographic Number (NBN). This number functions as the unique identifier of every digital item stored in the system. The content itself is stored in the e-Depot, while the metadata is stored in the KB catalogue. Technical metadata is stored and maintained by using the Preservation Manager, developed in collaboration with IBM.⁴ End-users may query the online catalogue and retrieve the full text of the publications. In the case where access restrictions are imposed by the publisher, retrieval may occur only after a process of identification, authentication, and authorization (IAA). The e-Depot itself cannot be accessed directly, but passes relevant publications to the end-user after verification.

Six major publishers have signed distinct archiving agreements with the KB for long-term digital archiving of their electronic publications:

• Elsevier Science
• Springer
• BioMed Central
• Blackwell Publishers
• Oxford University Press
• Taylor & Francis

At this moment the digital publications of these publishers are being loaded into the e-Depot, involving more than 2,500 journals, containing over 8 million articles. For publications that are processed both in digital and printed form, the KB has decided to process only the digital manifestation of the publication. New acquisition methods are studied in order to obtain the electronic publications, such as extensions of the OAI harvesting protocol.^{5, 6} With respect to version management of electronic publications, the e-Depot is able to deal with updates and retractions. Updates of electronic publications are sent to the KB with different time stamps compared to the original submissions. These authentic publications are not discarded from the system, but the original metadata is temporarily withdrawn, so that only the updated material will be found in the central catalogue. This way, the KB does preserve the complete record of science, but allows publishers to ask for the temporary withdrawal of specific articles. Once an article is in the system, it will never be discarded or deleted.³

New types of hardware, computer applications, and file formats are continually being developed, making digital information from the past inaccessible. Even if the hardware or the carrier-media do not deteriorate, the technology to access the information will inevitably become obsolete. Preservation or permanent availability of digital information is one of the processes dramatically affected by the evolution towards an all digital world.

In general there are two main digital preservation approaches. The first one focuses on the digital object itself and aims at changing the object in such a way that software and hardware developments will not affect its availability. By changing or updating the format of an object, it is made available on new software and hardware. The digital object will be adjusted to changes in the environment, which makes it possible to render objects by using current systems.

The second approach does not focus on the digital object, but on the environment in which the object is rendered. It aims at (re)creating an environment in which the digital item can be rendered in its authentic form. The first approach (changing the object) is known as migration or conversion. The second approach (changing the environment) is known as emulation. Both models are considered for implementation at the KB, and each will be discussed here in brief.

With migration, file formats will be converted into new formats as soon as the original formats run the risk of becoming obsolete. For example, if technology scans indicate that PDF version 1.1 will soon be inaccessible, all files in the digital archive of format PDF 1.1 will have to be converted into, for example, PDF format version 1.4. This way, the digital publications will be prepared for rendering for another period of time, until the format PDF version 1.4 runs the risk of becoming obsolete itself. At that time another migration procedure will need to be carried out.

An advantage of migration as a digital preservation strategy is that electronic publications will always be available in the form that is generally accepted, e.g., PDF, and current hardware and software will be able to render these formats with little difficulty. Older documents that are properly migrated will be available for some time in the present and the near future, and their electronic content can be used for copy and reuse. A major drawback might be that while converting documents from one form to another, some aspects of the document’s layout or–even worse–data might get lost. If preserving the original “look and feel” of the document is important, or when one is dealing with with dynamic objects, then migration is probably not the best solution. Moreover, migration is necessary for every single document in the collection, and should preferably be carried out each time a serious update of the file format is available. It may be straightforward to convert from version A to version B, but converting from version A to version C or D might be a complicated matter (see Caplan, 2004⁷ for an elaboration on this issue). Moreover, with migration it may be impossible to perform a conversion if the file format and the migration tool are no longer active. When applying the migration strategy we have to constantly study conversion programs and execute them when possible, so as to prevent digital information from getting lost.

Emulation, on the other hand, preserves the authentic document and provides the user with a tool that enables “old” software and “old” viewer programs to render this original document. An emulation tool generates an authentic view by launching the original viewer in the context of the original platform. The emulation tool makes the original viewer and the original platform work in future environments.

An advantage of emulation is that the original “look and feel” of the publication can be preserved. As with preserving books, the authentic instantiation will be there to be rendered, in contrast to migration in which possible other instances are used rather than the original. However, a serious drawback is the complexity of developing and maintaining emulation tools. In the future, we will have to maintain several emulation tools, and it cannot be proven that these will always work on future computer platforms. Maintenance of emulation tools can be reduced considerably if a virtual layer is introduced. This means that emulation tools are developed to run on a virtual machine, of which the upper side—the interface to the emulation tools—remains the same through time. One only has to adapt the bottom side of the “virtual machine” once in a while. This way, emulation tools will remain unaffected.

In the next section, emulation and migration techniques will be compared in terms of life cycle management and associated costs. In general, we will see that cost arguments will be in favor of emulation techniques. However, this does not imply a strong preference for emulation over migration. There are arguments both for preserving the original “look and feel,” as well as for converting documents to new standards.

The main reason for preserving the authentic form is that the KB digital archive serves as a safe place for original materials from publishers. The KB promises to keep the original bit stream of the received document. In the future, emulation tools will be needed in order to render these publications in the same way as they were published originally. Secondly, authenticity of a publication may be of importance for end users who want to access publications and experience the original “look and feel.” For these reasons, emulation tools are needed.⁸

On the other hand, there is a specific need for converting documents into the most current standard as well. For future end users who want to have access to publications according to the standards and functionalities of that time, migration might be needed in order to copy and reuse data. In short, we do not favor one strategy over another. In fact, both are studied and considered for implementation at the KB.

Any particular digital preservation strategy strongly determines the life cycle management of digital publications and thus the associated costs. In order to specify the long-term costs of a digital archive, it is necessary to understand the implications of choosing a particular preservation strategy. The resulting costs may in turn determine (or limit) the choice for certain preservation strategies. Unfortunately most business models and cost-estimates available in the literature only address the general preservation issues and say little to nothing about the costs of specific strategies (although Dürr, 2001⁹ specified some numbers about the costs of migration). The following comparison is based on available information from the literature in combination with our own insights and the experiences at the KB.

Where the conversion of objects to other formats constitutes a considerable cost factor in migration, these costs can be saved when applying emulation. In turn, emulation requires more initial investments, which makes it inappropriate for short-term preservation. For a proper cost comparison, the costs of each strategy should be specified in relation to the term for preservation. Emulation costs can be classified by the tasks that should be executed in order to realize access in the future. Consider the following task overview:

• One-time costs: Development of an emulation device
• Recurring costs: Developing emulators for the components of the original hardware platform
• At access time: Running the emulator and the appropriate software environment.

The costs of developing an emulation device can be derived from the costs associated with the creation of a demonstration version of the Universal Virtual Computer (UVC), which was implemented in 2004 by IBM as a result of a request of the KB. Including research and design, it took 32 weeks of 40 hours a week to accomplish this task. If we assume an hour rate of $120 the one-time costs of developing an emulation device are approximately $150,000. The actual implementation costs (excluding research and design) are estimated at $20,000 (for a total of 160 man-hours). The first prototype of this emulation tool based on the Universal Virtual Computer concept is now available. For more information about this project, we refer to Lorie, 2002¹⁰ and Van Wijngaarden & Oltmans, 2004.¹¹ More information and a “proof-of-concept” demonstration of the UVC can be obtained from http://www.alphaworks.ibm.com/tech/uvc.

The development of emulation tools requires serious research and development, and requires technical expertise to implement the concepts. Once it has been implemented it should also be maintained. Moreover, one must develop an emulation tool each time objects from a new platform are accepted into the archive, which also requires investments in both research and implementation. We assume maintenance costs will be relatively low (between $2,000 and $3,000 per year). The development of an emulation tool will be more costly. However, one must realize that the costs of developing emulation devices can easily be shared by digital preservation repositories all over the world, since once an emulator is available, it can be used to access any digital object that used to run on the emulated platform. In contrast to migration, where the task must be executed for each object separately, an emulator can be used to access a whole range of digital objects. In short, emulation tools can be shared among institutions, which makes it possible to share the costs of research and development investments as well.

Unfortunately the costs of accessing a digital object using the emulation approach are not known. In order to make a proper cost estimate one should account for the time necessary to configure an emulation of the original hardware using the appropriate emulation modules. Then one should add the time necessary to install the software environment that is associated with the specific digital object. This time will be less if the process is automated.

Migration on the other hand is relatively cheap in the sense that many conversion tools are available, and executing a conversion program is a relatively straightforward task. However, by definition, migration applies to the entire collection repetitively: each and every single object in the digital archive has to be converted, again and again. This means that the bigger the archive gets, the more expensive migration will be. This is in contrast with emulation: emulation tools apply to the collection as a whole and do not affect the format of individual digital objects.

In order to specify this difference exactly, we will use the cost model formula as presented by Shenton, 2003.¹² The costs of preserving a non-digital monograph over time are taken as an example. Shenton specifies them as follows:

(1) K(t) = s + a + c + p1 + h1 + p(t) + h(t)

Where K(t) is the total cost of holding an item for a period of t years, where s=selection, a=accessioning, c=cataloguing, p1=initial preservation, h1=initial handling, p(t)=longer-term preservation, h(t)=storage.

By applying this formula, the long-term costs of preserving a monograph can easily be calculated. Moreover, this formula makes it possible to calculate the downstream costs if for instance, an additional $100,000 would be available on acquiring monographs.

It is of significant interest that such a formula is available for electronic publications as well. Obviously, not all the variables in Shenton’s formula are applicable to digital objects. A fundamental part of every formula that applies to electronic materials would consist of:

(2) K(t,a) = s(a) + i(a) + h(t,a)

Where K(t,a) is the total cost of holding a objects for a period of t years, where s=selection, i=ingest, and h=storage.

The selection process is quite obvious: it consists of acquiring the objects and preparing them for further processing. The ingest process consists of the automatic processing of the digital objects by some sort of software program. It should, among other tasks, convert the associated metadata into a usable format and store the digital objects on some sort of storage system. The storage costs itself are for purchasing storage media, applying media refreshment, and maintaining some sort of database management system. There is a direct relation between the overall costs and both the number of items and the number of years they are preserved. More items will cost more, and storage for a longer term will cost more as well.

A part of the formula above is fundamental for two other formulae that we propose. However, these will not include the costs for selection and ingest for two reasons. First of all, the costs for selection and ingest will be the same for both emulation and migration; in other words, they will not influence the relative difference. Secondly, it is quite difficult to estimate these costs, as they depend on the archiving agreements with the publishers (selection) and the type of software that is in place (ingest). Both will differ considerably, depending on the circumstances. Therefore we focus on storage costs and the dedicated costs for both migration and emulation.

The first formula, for migration, is as follows:

(3) K(t,a) = h(t,a) + m(t,a)

Where K(t,a) is the total cost of holding a objects for a period of t years, where h=storage costs, and m=migration costs.

A new variable is introduced that expresses the costs of migrating an object. The costs of migrating digital objects is dependent on the time t (the longer we preserve the objects, the more often we have to convert them) and on the number of objects a (the more objects in the archive, the more conversion actions have to be executed).

The formula for calculating the emulation costs is as follows:

(4) K(t,a) = h(t,a) + E + e(t)

Where K(t,a) is the total cost of holding a objects for a period of t years, where h=storage costs, E=costs of setting up the emulation virtual machine, and e(t)=costs of emulation over time.

Two new variables are introduced: the one time costs for developing an emulation device and tools are expressed by E, while yearly maintenance of the emulator are expressed by e. Maintenance costs and costs for the development of emulation tools are independent of the number of objects: the emulation device and other emulation tools apply to the entire collection, and no special action is needed when rendering an object in the digital archive. However the emulation tools need to be maintained over time, which makes the maintenance costs dependent on the number of years. (In this cost comparison it is assumed that migration is not suitable for the preservation of dynamic digital objects. Therefore this cost comparison only focuses at the costs of preserving static digital objects.)

Having the first, primitive, formulae in place, we can now associate specific variables with the cost components. The costs for storage and migration are based on figures from the literature (for instance, see Fox, 2002¹³ for an elaboration on storage costs), but may vary. A complicating matter in this respect is that both figures usually express costs per Megabyte or Gigabyte. Since insufficient information about the costs of migration and emulation was available in the literature, a number of assumptions have been made:

• The number of objects (a) in the repository increases every year.
• Storage costs are estimated at $0.05 per object per year.
• The required storage space for both strategies is about equal (the additional storage space required for the preservation of the software environment in case of emulation is considered negligible).
• The average costs of migrating an object into a newer format (m) are $0.10 per object per year.
• The one-time costs for creating an emulation device, including research and design, (E) are $200,000. (It is assumed that this will be more complex than the UVC mentioned before.) 
• The average costs for maintenance of the emulation device  and development of emulation tools for new sorts of objects (e) are $30,000 per year.
• The costs of running the emulator and the appropriate software environment at access time are not included.

Considering these figures, the costs of preserving 1,000,000 objects for a period of 50 years can now be calculated, both when applying migration and emulation as the leading digital preservation technique. The graph below demonstrates these costs:

Figure 2: Costs for migration (blue line) and emulation (red dotted line) for maintaining an archive of 1,000,000 digital objects over a period of 50 years. The initial investments of setting up an emulation tool yields high costs in the first five years. But soon after that, the migration costs are higher than the emulation costs and the difference increases every year. In 50 years, the migration costs are 79% higher than the emulation costs.

The difference between emulation and migration is clear, and can be explained by the difference in costs coverage of the two techniques. As the size of the collection directly affects the migration costs, it will be clear that the bigger the archive gets, the higher the migration costs will be. This effect is demonstrated in the second graph. It covers the same period of years, but the size of the collection now is 5,000,000 instead of 1,000,000.

Figure 3: Costs for migration (blue line) and emulation (red dotted line) for maintaining an archive of 5,000,000 digital objects over a period of 50 years. Compared to the graph in Figure 2, the size of the collection is five times as big and the migration costs are now more than twice as high as the costs for emulation.

As shown, the figures vary with the values that are assigned to the variables. The period that it takes before emulation is less expensive than migration is dependent on the following factors:

• If the number of objects to be preserved is increased, less time will be needed for emulation to be more economical than migration.
• If the average migration costs per year per digital object are higher, it will take less time before emulation is cheaper than migration.
• If the costs of developing an emulation device are increased, it will take more time before emulation is less expensive then migration.
• If the maintenance and development costs for emulation per year are higher, more time will be needed before emulation is more economical than migration.

As said, a number of assumptions were made with respect to the values of the variables. In order to demonstrate the effects of other assumptions, we developed a spread sheet so as to assess the consequences of different values. The spread sheet is freely available for test purposes, and researchers and digital archive managers are invited to share their opinions and experiences with respect to cost issues while using the spread sheet. (Click here to download spreadsheet.)

5. Conclusion

In this paper we discussed life cycle issues in the context of long-term preservation of digital objects. At the National library of the Netherlands a fully operational digital archive is in place, and this archive, the e-Depot , provides the context for studies in a number of digital preservation techniques. Both emulation and migration are discussed, as the KB needs to provide access to the original publication as sent by the publisher, while at the same time the KB wants end users to access converted materials according to the most recent standards and functionalities.

Emulation and migration are inherently different in terms of life cycle management, which causes a serious difference in costs. While migration applies to all objects in the collection repetitively, emulation applies to the entire collection as a whole. This makes emulation most cost-effective in cases of large collections, despite the relatively high initial costs for developing an emulation device. When considering the fact that only small fragments of digital archives need to be rendered in the long run, it may turn out that from a financial perspective emulation techniques will be more appropriate for maintaining larger archives.

In this overview, we deliberately neglected a number of important issues. First of all, we did not consider the fact that the migration costs per object may be less if the number of objects to be converted gets considerably high (economies of scale). What is more, we calculated the costs of migration in terms of the number of objects, while it would also make sense to calculate the costs in terms of Gigabytes. The problem here is that preservation costs are connected to the number of objects and it is not clear how many objects are contained in a Gigabyte. This points out that the issue of costs in digital archiving needs more study and should be based on practical experience. The results presented here are a first step for determining life cycle issues in digital archiving, and may serve advanced studies that reach for a complete understanding of cost models in long-term preservation. Participants in this research field are invited to challenge or further develop the cost formulas, preferably by using the attached spread sheet.

The authors wish to thank Hilde van Wijngaarden (KB) and Raymond J. van Diessen (IBM) for valuable comments on earlier drafts of this paper.

Notes: 

¹OAIS 2002. Consultative Committee for Space Data Systems, Reference Model for an Open Archival Information System (OAIS), Blue Book, 2002.

²Oltmans, E. & Van Wijngaarden, H. (2004). Digital Preservation in Practice: The e-Depot at the Koninklijke Bibliotheek. In: VINE – The Journal of Information and Knowledge Management Systems, Vol. 34 (2004), No. 1.

³Steenbakkers, J.F. (2003). Permanent Archiving of Electronic Publications. In: Serials, Vol. 16 (1), March 2003.

⁴Oltmans, E., Van Diessen, R.J. & Van Wijngaarden, H. (2004). Preservation Functionality in a Digital Archive In: Proceedings of the Joint Conference on Digital Libraries, Tucson, Arizona, June 7-11, 2004.

⁵Lagoze, C., Van de Sompel, H., Nelson, M., Warner, S. (2002). The Open Archives Initiative Protocol for Metadata Harvesting, Version 2.0, 2002.

⁶Jerez, H.N., Liu, X., Hoschtenbach, P., and Van de Sompel, H. (2004). The Multi-faceted Use of the OAI-PMH in the LANL Repository. In: Proceedings of the Joint Conference on Digital Libraries, Tucson, Arizona, June 7-11, 2004.

⁷Caplan, Priscilla (2004). Building a digital preservation archive: Tales from the front. In: VINE – The Journal of Information and Knowledge Management Systems, Vol. 34 (2004), No. 1.

⁸Van Diessen, R.J. & Van der Werf-Davelaar, T. (2002). Authenticity in a Digital Environment. IBM/KB Long-Term Preservation Study Report Series #2, IBM Netherlands, Amsterdam. Available through http://www.kb.nl/e-depot.

⁹Dürr, E., van der Meer, K. (2001). Emulation and conversion – Organisational and architectural overview. Way of working, costs, methods. Available through http://www.library.tudelft.nl/e-archive/Documenten/Resultaten/roquade2.pdf.

¹⁰Lorie, R. (2002). The UVC, a Method for Preserving Digital Documents: Proof of Concept. IBM/KB Long-Term Preservation Study Report Series #4, IBM Netherlands, Amsterdam. Available through http://www.kb.nl/e-depot.

¹¹Van Wijngaarden, H. & Oltmans, E. (2004). Digital Preservation and Permanent Access: The UVC for Images. In: Proceedings of the Imaging Science & Technology Archiving Conference, San Antonio, USA, April 23^rd 2004.

¹²Shenton, Helen (2003). Life Cycle Collection Management. In: Liber Quarterly – The Journal of European Research Libraries, Vol. 13 (3/4).

¹³Fox, Peter (2002). Archiving of electronic publications – Some thoughts on cost. In: Learned Publishing, Vol. 15, No. 1, January 2002.

Electronic records pose many challenges for archivists, but these arise from a single underlying issue: access to a digital object is entirely dependent on technology. A file in a given format requires software to decode and display it; that software in turn requires a specific combination of hardware, operating systems, and other software to run. Equally, the storage media on which the file is stored requires its own combination of hardware, software, and operating system in order to be accessed. Understanding this complex network of technical dependencies lies at the heart of any archiving programme for electronic records.

The technology on which electronic records are so utterly dependent is constantly evolving: existing technologies are redeveloped in new versions, or become obsolete, and entirely new technologies emerge to replace them. This can happen at a very rapid rate, with new versions of software products being released on an annual basis. The challenge for the archivist is not only to understand the nature of these technical dependencies, but also to continually monitor changes that threaten the continued accessibility of electronic records. This monitoring process is frequently referred to as “technology watch.”

In order to meet these challenges, the National Archives of the UK (TNA) has developed a technical registry system called PRONOM. At the heart of PRONOM is a database containing detailed information about the various types of technical components which may be required to access or process an electronic record. These components include file formats, software products, operating systems, hardware components, and storage media.

Fig 1: The PRONOM information model

PRONOM has been developed as a resource to support both TNA’s own digital archive and for the wider international community. For this reason, it has been made freely available on the Web, where users can search its content, generate reports, and even submit new information for inclusion.

The genesis of PRONOM has already been described in a previous article by Jeffrey Darlington. This article describes significant new developments that have taken place since 2003 and some of TNA’s future plans in this area.

Enhancing the Technical Registry

The new version of PRONOM (PRONOM 4) will include a substantial revision and enhancement of the database. Amongst other things, this will allow us to record detailed technical information about specific file formats, including links to the formal specifications where possible.

Fig 2: Modeling file formats in PRONOM

For the first time, users will be able to search for specific file formats and retrieve a wide range of information on everything from supported byte orders and compression methods to copyright and patent restrictions. At launch, we expect to have detailed information on over 100 formats, and this will be expanded rapidly thereafter. The underlying data model has also been substantially reworked, providing much greater flexibility for future development and facilitating interoperability with other possible future registry services, such as the planned Global Digital Format Registry. These enhancements will allow us to expand the scope of PRONOM beyond its original focus on software tools and into the full technical registry service required to support long-term preservation.

The value of a registry such as PRONOM is directly related to the quality and breadth of the information it contains. We, therefore, place great importance on maintaining the authority and reliability of that information and on working with many different communities of interest to develop new content. In recent years, TNA has developed very productive relationships with a number of major software developers. These relationships should enable us to significantly enhance PRONOM’s content and also, we hope, signal a notable step forward in the recognition of long-term preservation issues within the developer community.

In addition, our on-line submission form, which allows anyone to submit new information for possible inclusion in the database, has proved very successful, particularly for improving coverage of more specialised software products.

Automatic File Format Identification

The second major new feature to be introduced in PRONOM 4 will be the first in a planned series of standalone tools designed to use information stored in PRONOM to provide specific preservation services. The first tool, DROID (Digital Record Object IDentification), will perform automated batch identification of file formats, using internal and external signatures to identify and report the specific file format versions of digital files. DROID will first attempt to match a file against a list of internal signatures—specific patterns of bytes that can be used to identify a format. These signatures are expressed as sequences of hexadecimal values and can also incorporate wildcard operators, providing a very flexible and expressive syntax. A match against an internal signature will result in a positive identification of the format. As a secondary method, DROID will also attempt to match any external signatures, which are currently limited to file extensions, although support for other types, such as Macintosh data forks may be added in the future. However, any identification based purely on an external signature will be accorded a much lower priority.

Fig 3: An internal signature for JPEG File Interchange Format 1.02

The signatures will be stored in an XML signature file, generated from information recorded in the PRONOM technical registry. Initially, we plan to provide over 130 internal signatures and over 600 external signatures. New and updated signatures will be regularly added to PRONOM, and users will be able to configure DROID to automatically download updated signature files via Web services. Full documentation of the signature syntax and XML schemas will be made available on the PRONOM website.

DROID is designed to support batch processing of large numbers of files. It will allow files and folders to be selected from a file system for identification and saved as a list in XML or CSV format. After the identification process has been run, the results can be output in XML, CSV, or printer-friendly (HTML) formats, for further processing or import into the appropriate preservation metadata scheme.

DROID will be made freely-available to download from the TNA website and, being written entirely in Java, will be fully platform-independent. It will provide both a graphical user interface and a command-line interface and will also include a documented, public API for ease of integration with other systems. This will enable use by other digital archives. The potential for integration is already being demonstrated through a JISC-funded project with Southampton University, which is building the tool into a new ingest module for the Eprints digital repository system, used by over 130 archives worldwide.

PRONOM Unique Identifiers

In parallel to these developments, TNA will be implementing an extensible scheme of PRONOM Unique Identifiers (PUIDs), which will provide persistent, unique, and unambiguous identifiers for file formats. Such identifiers are fundamental to the exchange and management of electronic objects, by allowing human or automated user agents to unambiguously identify, and share the identification of, the encoding format of an object. This is a virtue both of the inherent uniqueness of the identifier and of its binding to a definitive description of the format in a file format registry, such as PRONOM. No existing, universally-applicable system provides for this. UNIX “magic numbers” and Macintosh data-forks do provide some of this functionality, but the same is not true within Microsoft DOS or Windows environments. The three-character file extension is neither standardised nor unique and is interpreted differently by different environments. Equally, the IANA MIME-type scheme does not provide sufficient granularity or coverage to satisfy the requirements for unique identifiers. The PUID scheme has been developed for the single purpose of providing such identifiers.

The new scheme of PUIDs has been adopted as the recommended encoding scheme for describing file formats in the latest version of the e-Government Metadata Standard. This will mean that, for the first time, a consistent, persistent and highly-granular scheme for describing file formats will be in use across the UK government. Further information on the PUID scheme and a list of currently-assigned PUIDs will be available through PRONOM 4.

In the future, the scheme may be extended to include other technical components described by PRONOM, such as operating systems and codecs.

The Seamless Flow Programme

TNA has recently initiated a major programme called Seamless Flow, which will integrate and automate processes for managing electronic records throughout their lifecycle, from creation, appraisal, selection, and transfer from government departments, to preservation and dissemination by TNA. The future development of PRONOM will be an integral part of this programme, creating an automated technology watch service that will enable us to understand the digital objects we are storing, identify when some form of preservation intervention is required, determine the nature of that intervention, and automatically invoke the appropriate service.

Fig 4: Components of the technology watch service

PRONOM already provides us with the technical registry needed to support these activities, although further enhancements will doubtless be required. The remainder of this article summarises the additional services that we will now be building on top of this.

Technical characterisation underpins all subsequent preservation activities: if we don’t understand the precise technical characteristics of a digital object, we cannot hope to preserve it or make it accessible. The most obvious form of characterisation is to identify the format in which a computer file is encoded, an issue which TNA is addressing through the DROID tool described above. However, under this theme we also plan to address issues such as format validation, automatic extraction of technical metadata, and the identification of dependencies between digital objects. We intend to develop a suite of tools to provide these services, which will be automatically deployed in accordance with the identifications provided by DROID. Wherever possible, existing tools, such as the JHOVE format validator, developed by Harvard University, and the National Library of New Zealand’s metadata extractor, will be used.

The preservation planning service will form the decision-making heart of our preservation system. It will enable us to identify and monitor technological changes and their potential impacts on electronic records stored by TNA and UK Government departments, as well as to develop preservation strategies to mitigate the impact of these technological changes. TNA’s long-term preservation strategy is based on object migration, coupled with retention of all records in their original formats. The system will therefore focus on the development of migration pathways for the automatic conversion of electronic records to new formats as required for preservation or presentation purposes.

PRONOM already records information about the product support lifecycle for the software tools required to create or render electronic records and in PRONOM 4 this support information will be extended to cover other technical components such as file formats. This information will play a vital role in making decisions about when to migrate. In addition, existing information about the formats that particular software can read and write can form the basis for identifying potential migration pathways. However, many other factors will also be significant in the preservation planning process. TNA intends to develop a holistic risk assessment methodology for electronic records that will enable us to identify risk factors at an early stage, predict their impact, and plan appropriate mitigation strategies.

A number of other projects are also beginning to develop risk assessment techniques for digital objects. OCLC’s INFORM project has developed a methodology for quantifying the risk factors affecting digital formats and measuring their potential impact on preservation decisions. The PANIC project, at the University of Queensland’s Distributed Systems Technology Centre, is developing a semi-automated preservation service for scientific data that will allow monitoring of archival collections, support decision making about preservation actions, and then invoke the appropriate preservation service (such as a format conversion service), using the semantic Web and Web services. Cornell University’s Virtual Remote Control project is developing risk assessment and remote monitoring services that are specifically aimed at web-based content and builds on previous seminal work in this field.¹ Projects such as these will also inform TNA’s approach to preservation planning.

The final stage in the process is to actually perform the selected migration process. In an operational environment, automation is the only viable approach to this, and our work will therefore focus on the development of a framework for controlling automated migration services. Specific tools that have been tested and approved to provide these services will then be deployed within this framework as required. As always, we will use existing conversion tools wherever possible. The migration service will export electronic records previously identified as requiring migration from our digital repository, automatically migrate them using the selected migration pathway, and accession the migrated records into the digital repository as new manifestations.

The documentation of these preservation actions will perform a vital role in establishing the continuing authenticity of electronic records, and TNA is developing a comprehensive data model to enable multiple manifestations of records and the migration pathways that link them to be documented and managed within our digital repository. This model also takes into account the issue of dependencies between digital objects. This is of particular importance, given that many electronic records are compound objects, composed of many inter-related files, such as websites, or office documents with linked or embedded content. It is essential to understand the nature of these dependencies in order to predict the full impact of a preservation action such as migration. To give a simple example, the migration of images in a Web page from GIF to PNG will also necessitate updating of the HTML image references. Thus, a migration pathway may actually comprise a complex set of format conversion processes, emendation processes, and management of the associations between objects.

The new version of PRONOM, including the DROID tool, is due to be released in June 2005–full details will be made available on the PRONOM website in due course.

Notes:
¹Lawrence, Gregory W., Kehoe, William R., Rieger, Oya Y., Walters, William H., and Kenney, Anne R. Risk Management of Digital Information: a File Format Investigation. Washington DC: Council on Library and Information Resources, 2000 (http://www.clir.org/pubs/abstract/pub93abst.html).

Librarylaw.com is maintained by Mary Minow, a librarian cum JD who works now as a consultant. The site broadly covers many legal issues as they pertain to libraries including:

• Copyright and Libraries (including Licensing)
• Disability Law and Libraries
• Government Document Depository Program
• Intellectual Freedom (First Amendment)
• Policymaking and Libraries
• Privacy and Libraries (including the PATRIOT Act)

In particular, there is a section devoted to Web Pages and Digitization Projects that includes links to relevant articles, including an excellent introduction to the topic by Minow entitled “Library Digitization Projects and Copyright.”

What are “orphan works,” why are they important, and why I have I heard so much about them recently?

This FAQ is answered by Peter Hirtle, Technology Strategist and Intellectual Property Officer for Cornell University Library.

The Orphan Works Problem

“Orphan works” are works whose current copyright owner cannot be located. There are a number of reasons why it may be very difficult if not impossible to locate a current copyright owner. One of the most obvious is when one cannot identify the author of the original work. The work in question could, for example, be an anonymous manuscript or photographs from a shoebox. The author is the initial copyright owner; if you do not know the name of the author, it is impossible to trace current copyright ownership.

Even with a name, it can often be difficult to determine the current copyright owner. Copyrights can be transferred to others. An author, for example, might transfer copyright to a publisher. If she retains copyright, upon her death the copyright would become part of her estate and be passed according to her will to her inheritors, who might subsequently transfer the copyright to others. A third possibility is that the copyright belonged to a firm that was subsequently sold or went out of business. Tracking the ownership of copyrights in such a situation can become very complicated and may be fruitless, even if one follows all of the suggestions in such guides as “Locating U.S. Copyright Holders.” Although the Copyright Office has had a voluntary system for recording transfers of copyright, there has never been a requirement that such transfers be recorded.

The number of orphan works has certainly increased due to changes in U.S. and international copyright laws. Prior to 1989, authors had to take affirmative steps to secure Federal copyright protection. They might have had to include a copyright notice stating who the copyright owner was, register the copyright with the Copyright Office, renew a copyright after a limited period, or manufacture a published copyrighted item in the United States. Many authors (as many as 90% according to one study) did not bother with such formalities, and their work entered the public domain, either at the point of publication or after a brief period of twenty-eight years.

In 1989, however, with the passage of the Berne Convention Implementation Act of 1988, the last of the formalities associated with copyright protection was removed. Original expression is now copyrighted as soon as it is fixed in a tangible medium, with no need for renewal. Works that once would have entered the public domain remain copyrighted. For some (if not many) of these, the current copyright owner cannot be found. Furthermore, the duration of copyright has increased by an order of magnitude from the time of the Founding Fathers. The initial copyright term of fourteen years (with the possibility of a fourteen year renewal period if the author was still living) has expanded to become a term equivalent to the life of the author plus seventy years. Longer copyright terms greatly increase the likelihood that the identity of the current copyright owners will be lost.

The inability to locate the current copyright owners of older works has thwarted the Constitutional intent of copyright: “the progress of science and useful arts.” When current creators and users cannot locate a copyright owner, they cannot negotiate over the use of the older work. Potential users, therefore, are reluctant to incorporate orphan works in new creative efforts or in projects (such as library digitization efforts) that would make the older works available to the public. Users, publishers, libraries, archives, and other institutions worry that reproducing and/or distributing copyrighted works without the permission of the current copyright owner may leave them open to the draconian monetary, statutory, and criminal penalties found in current copyright law.¹ Although a thorough search can sometimes turn up the current copyright owner, such a search more often is inconclusive and expensive, rapidly consuming funds that could be used for other purposes. The result is that orphan works often are not used—even when there is no one who would object to the use.

The Copyright Office Investigates

Recognizing that there might be a problem associated with orphan works, the Copyright Office issued a Notice of Inquiry seeking comments on the orphan works problem. Over 700 responses were received by the 25 March deadline; they are available for reading on the Copyright Office website.

Many of the submissions document how concerns over the orphan status of copyrighted works have led libraries and archives to exclude them from digitization projects. The submission from the Library Copyright Alliance is particularly telling in this regard, as are the submissions from the University of California San Diego Libraries and the Cornell University Library. Many of the hundreds of submissions from individuals also document how their efforts to use orphan works have been thwarted by their inability to locate the current copyright owner. (To be fair, there are also submissions from representatives of copyright owners who feel that any new regulations might lead to an increase in infringements, to the detriment of active copyright owners.)

Although the vast majority of the submissions document problems working with orphan works, some suggest solutions as well. I have yet to find a submission that endorses the Canadian approach, which requires extensive research into the copyright status of an orphan work, followed by submission of the results of the research to a government board, which then decides if the orphan work can be used and how much of a license fee should be paid into escrow pending the discovery of the copyright owner. Many submissions instead argue for a solution similar to the one proposed by the Glushko-Samuelson Intellectual Property Law Clinic. Their approach would require that potential users of orphan works conduct a “reasonable” investigation into the copyright status. Users who conduct such a search would be immune from the particularly onerous threat of statutory damages, attorney’s fees, and injunctive relief. The administrative costs should be kept to a minimum. Other proposals, most notably those from the Center for the Public Domain and the Society of American Archivists, suggest that a potential user should publicize her intent to exploit a presumably orphan work via a website maintained by the Copyright Office.

Next Steps

The Copyright Office’s process is only partially complete. While the deadline for the initial comments was 25 March 2005, there is an open period for replies to the responses received to date. The Copyright Office will accept reply comments until 5:00 EDT on 9 May 2005; instructions on how to submit comments are available from the Copyright Office. More evidence of orphan works’ negative impact on society would be useful, but reactions to the proposed solutions would be especially helpful. The Glushko-Samuelson Intellectual Property Law Clinic’s proposal, for example, resists spelling out in concrete terms what constitutes a “reasonable effort” to locate a copyright owner. Would this approach really provide the assurances that “gatekeepers” (publishers, library administrators, institutional counsel, etc.) need in order to permit a project involving orphan works to proceed?

This is only the start of a solution to the orphan works problem. Based on the final responses, the Copyright Office will decide if new legislation or regulations are needed. There will be more comments, more hearings, and more discussion before the orphan works issue is resolved. Nevertheless, the Copyright Office is to be commended for initiating the discussion. It is now up to all of us who have been affected by the orphan works problem to make our opinions known.

¹ Penalties for copyright infringement can include the actual damages suffered; statutory damages of up to $150,000 per infringement; the costs and attorney’s fees associated with the case; impoundment of the infringing works; and imprisonment for up to five years. See 17 U.S.C. § 503-506.

The Mind's Eye: Theory and Practice of Digital Imaging in Cultural Heritage Institutions
April 22, 2005
Berkeley, California

Sponsored by the Los Angeles Chapter of the American Society for Information Science and Technology (LACASIS), “The Mind's Eye: Theory and Practice of Digital Imaging in Cultural Heritage Institutions” will be held at the Berkeley Art Museum and Pacific Film Archive. The one day event will feature presentations, discussion groups, and tours.

DASER-2
April 29-May 1, 2005
Adelphi, Maryland

The second Digital Archives for Science & Engineering Resources Summit will examine new issues and challenges surrounding Science-Technology-Medicine (STM) digital libraries, institutional repositories, and open access publishing. Topics to note include the impact of OA on the future of STM libraries; institutional repository models; publisher-library collaboration strategies; and user needs and patterns.

World Wide Web Conference
May 10-14, 2005
Chiba, Japan

This fourteenth annual conference will include workshops focused on the Semantic Web, XML programming, user-focused search and crawling, and others. Tim Berners-Lee (W3C), Eric Brewer (Berkeley), Lorrie Cranor (CMU), and Rob Glaser (RealNetworks) will give keynote presentations.

Digital Repositories: Interoperability and Common Services
May 11-13, 2005
Heraklion, Crete

This DELOS-sponsored workshop will focus on digital repositories: infrastructure, workflows, policy and organizational issues, metadata capture and creation, open access, and other topics.

Fedora Users' Conference
May 13-14, 2005
New Brunswick, New Jersey

This event is organized for current and prospective users of Fedora (Flexible Extensible Digital Object Repository Architecture). Program details and registration are now available online. Attendees can learn about FEDORA development updates and share implementation experiences.

DELOS Summer School 2005
June 5-11, 2005
Sophia Antipolis, France

Co-sponsored by the Digital Curation Centre (DCC) and ERPANET, the DELOS Network has developed a summer school on Digital Preservation in Digital Libraries. Program details have been released and registration is now open for the week long school targeted at students and professionals in archives, libraries, commercial organizations, and data creating and managing environments.

The Digital Humanities / Humanities Computing Summer Institute
June 11-14, 2005
British Columbia, Canada

The Digital Humanities / Humanities Computing Summer Institute presents one week of coursework, seminars, and lectures and brings together students, faculty, staff, and professionals from different areas of the Arts and Humanities with members of the digital library, library, and archival studies community. This fourth in the Institute’s series offers topics such as Text Encoding Fundamentals and their Application, Digitization Fundamentals and their Application, and Encoding: Metadata, Style-sheets, Text Transformations, and Databases.

2005 International Conference on Digital Archive Technologies (ICDAT2005)
July 16-17, 2005
Taipei, Taiwan

Targeted to a diverse audience interested in technologies, tools, and best practices for advanced digital archive systems and preservation, ICDAT 2005 will highlight technology solutions, case studies, and usability evaluations.

Digital Preservation Management: Short-Term Solutions to Long-Term Problems
July 17–22, 2005
Ithaca, New York

Registration will open on May 31, 2005 for Cornell University Library’s summer offering of its digital preservation management workshop.

Preserving Photographs in a Digital World: Balancing Traditional Preservation with Digital Access
August 20-25, 2005
Rochester, New York
 
This full week seminar will cover issues of both traditional photographic collection management and digital imaging. Lectures will cover topics such as color management, quality control, image-capture, storage, display, and output strategies, database design and application, and preservation and archival issues. This seminar series is sponsored by George Eastman House and Image Permanence Institute, Rochester Institute of Technology.

Ensuring Long-term Preservation and Adding Value to Scientific and Technical Data
November 21-23, 2005
Edinburgh, UK

The organizers for the third Ensuring Long-term Preservation and Adding Value to Scientific and Technical Data conference, PV 2005, have issued a call for papers with a focus on e-Science and digital libraries. The conference addresses issues of long-term preservation and adding value to scientific data.

The 2005 Digital Preservation Award Call for Entries
The Digital Preservation Coalition (DPC) is soliciting entries for the Digital Preservation Award for projects that demonstrate advancement in digital preservation that will benefit the UK. Entries will be accepted through May 31.

IU Archives of Traditional Music awarded NEH Grant to Digitally Preserve Endangered Sound Recordings
Indiana University and Harvard University have entered into a collaborative research and development project to create best practices and test emerging standards in the digital preservation of critically endangered sound recordings. The 18 month long project began in February.

eSPIDA Website Launched and Report Released
A report and session papers from eSPIDA's (An Effective Strategic Model for the Preservation and Disposal of Institutional Assets) February workshop are now available on the project’s new website. The workshop marked the launch eSPIDA, an new JISC funded project that is taking a holistic approach “to take digital preservation on to the next phase – sustainable institutional implementation.”

Database of the Argentinean Parliamentary Archives Developed with UNESCO’s CDS/ISIS
An online database of records of the parliamentary history of Argentina has been developed with UNESCO’s WINSIS software. It currently covers 1864-1914 but will be extended through 1976 and includes information such as origin of the related paper file, author, type of project, and summary of the document’s topic.

Managing Digital Assets: Speaker Presentations Online
The reading list and speaker presentations from the February 4-6, 2005 workshop Managing Digital Assets; A Primer for Library and Information Technology Administrators are now available on the CLIR (Council on Library and Information Resources) website.

NDIIPP Holds First Partnership Meeting
The first meeting for the formal NDIIPP partners was held at the Library of Congress in January 2005. The NDIIP (National Digital Information Infrastructure and Preservation Program) partnership is a cooperative initiative with partnership awards made to eight institutions “with the common goal of building a national digital-preservation infrastructure. The partners will collect and preserve significant information that is ‘born digital’ (created in digital form) and at risk of loss. They also will form an enduring collaborative network for sharing ‘best practices’ and addressing common issues.”

Happy Birthday Internet Public Library!
The Internet Public Library turned 10 years old in March. To celebrate, a timeline and tenth anniversary blog (and a birthday card) have been added to the resident resources on the website.

AU$1.5 Billion of Australian Culture Goes Online
The State Library of Australia has begun to make the contents of its “DNA of Australian Culture” digitized collection available online. The site will include such items as journals, photos, and interactive journeys to guide users through the content.

The Grey Journal Launches
The Grey Literature Network Service (GreyNet) has launched a new publication, The Grey Journal (TGJ). The journal will be published in three thematic editions each year and will offer an international perspective to grey literature and the types of documents that contain grey literature.

National Digital Newspaper Program Grants Awarded
Six institutions received more than $1.9 million in grants in the National Digital Newspaper Program (NDNP) to digitize early twentieth-century newspapers in order to create a Web accessible historical resource.

RLG DigiNews (ISSN 1093-5371) is a Web-based newsletter conceived by the RLG preservation community and developed to serve a broad readership around the world. It is produced by staff in the Department of Research, Cornell University Library, in consultation with RLG and is published six times a year at www.rlg.org.

Materials in RLG DigiNews are subject to copyright and other proprietary rights. Permission is hereby given to use material found here for research purposes or private study. When citing RLG DigiNews, include the article title and author referenced plus "RLG DigiNews." Any uses other than for research or private study require written permission from RLG and/or the author of the article. To receive this, and prior to using RLG DigiNews contents in any presentations or materials you share with others, please contact Jennifer Hartzell (jlh@notes.rlg.org), RLG Corporate Communications.

Please send comments and questions about this or other issues to the RLG DigiNews editors.

Co-Editors: Anne R. Kenney and Nancy Y. McGovern; Associate Editor: Robin Dale (RLG); Technical Researcher: Richard Entlich; Contributor: Ellie Buckley; Copy Editor: Ellie Buckley; Production: Jenn Colt-Demaree, Carla DeMello; Advisor: Peter Hirtle.

All links in this issue were confirmed accurate as of April 13, 2005.