Introduction

In 2003, the U.S. National Endowment for the Humanities (NEH) awarded Cornell University Library a grant to develop a digital preservation management training program, consisting of intensive weeklong workshops and the creation of an online tutorial. In 2005, NEH provided a second grant to continue the program through 2006. Seven workshops have been held, reaching 164 individuals from 110 institutions. Prior to each workshop, participants were asked to complete an online survey designed to assess their institution’s readiness to develop and maintain a digital preservation program. This article reports on the aggregate findings from this survey and offers comparisons where possible to RLG’s 1998 survey on the status of digital archiving in its member institutions. A comparison of findings from these surveys provides some means for assessing change in institutional response to digital preservation over an eight-year span.

Much has occurred in the interim to raise the visibility of digital preservation concerns at the institutional level:

• dominance of digital production for almost all forms of communication including scholarly content and institutional records
• ubiquity of the Internet and broadband communication—and the attendant vulnerability due to security concerns and the ephemeral nature of Web-based content
• increasing shift to e-only access
• massive digitization efforts to bring analog materials onto the Internet
• development of key research, standards, open source software, and legal requirements to support digital preservation
• mind-numbing changes in the hardware, software, media, and formats used to create and deliver information

Have these factors affected institutional responses to digital preservation needs? By no means definitive, the results from the Cornell survey do offer a picture of the growing trend by institutions to address these needs, as well as point to some of the areas requiring additional work.

The Cornell Survey

The survey of institutional readiness was designed to prepare participants for the Digital Preservation Management workshop by encouraging them to consider their organization’s digital preservation efforts in terms of scope, priorities, resources, and overall readiness. The survey itself was divided into three main sections, reflecting our approach to digital preservation training: organizational infrastructure, technological infrastructure, and resources framework.

Not all participants completed the survey, but a significant number of them did. Over the past seven workshops, we received surveys from individuals representing 100 distinct institutions, for a response rate of 90% overall of institutions that sent staff to one of the workshops. Fourteen institutions sent participants to different workshops. We included these survey responses as separate data points in the results to reflect the level of progress that had been made in the interim. The tally of institutions included in the survey results, therefore, is 114, greater than the total number of distinct institutions. The number of institutions counted in this report represents 82 respondents to the 2003-2004 survey and 32 to 2005 survey.

Participants came from across the United States as well as eight other countries. Half came from academic institutions and over a quarter represent government institutions (international, national, state and local), including five national libraries. Table 1 provides an institutional breakdown. “Other” includes consortial, corporate, foundation, public school, and religious entities.

Table 1. Survey respondents by institutional type.

Survey caveats:

Although many institutions are represented in this survey, there are some caveats to consider in reviewing the findings. First, although the participants represented those charged with some level of digital preservation responsibility at their respective institutions, individual assessments do not necessarily represent official responses. For instance, we offered participants the option to reply “don’t know” (or leave blank) to most of the questions. Presumably a formal institutional response would not include that response. It is also worth mentioning again that participants completed the surveys before coming to Cornell for the workshop, and it’s quite possible that some of their responses would have changed if they had responded after the workshop. Second, for the first three workshops, all participants were asked to complete the survey, including those coming from the same institution. Because the results showed some disparity in responses from participants from the same institution, we limited the survey submissions to one response per institution, beginning with the fourth workshop, held in July 2004. We asked participants from the same institutions to prepare a consensus response. Third, the survey instrument was not intended to capture all information reflecting an institution’s commitment to digital preservation, only indicators of efforts in key areas. Fourth, we used the same survey form for the first five workshops held in 2003-2004 (August and October 2003; May, July, and November 2004). In 2005 we slightly modified the survey form before each of the workshops (May and July) to capture additional information in response to new developments and to provide greater clarity to some of the questions asked. The current version and the 2003-2004 version are posted on the workshop website. Fifth, institutions that sent participants to the workshop were presumably fairly motivated by their concerns over digital representation. The results therefore may present an information bias and should not be construed as a representative sample of any institutional type as a whole.

Survey Findings

Participants submitted their institutional readiness surveys online. Cornell entered the responses into a spreadsheet and analyzed them before each workshop. Some data recoding was necessary in the case of invalid responses and in order to combine multiple responses from the same institution. Responses from the five 2003-2004 workshops (82 institutions) were aggregated and collectively compared. Similarly, responses from the two 2005 workshops (32 institutions) were aggregated for comparison. Due to changes in the survey form itself, it is difficult to compare several questions from the 2003-2004 data to the 2005 data, and some information is only available for 2005. This discrepancy accounts for some variation in years spanned for the reported findings below. It should also be noted that not all of the survey questions are presented in this article.

Organizational Infrastructure

Insufficient attention has been paid to the institutional context in which digital preservation programs must be developed. One measure of organizational readiness is the development, adoption, and implementation of policies that address digital preservation commitments and decisions. Too often, an organization undertakes responsibility for digital stewardship without first ensuring that the necessary policies and controls are in place or that the institution itself views digital preservation as a core mandate. In the 2003-2004 survey, only 26% of them reported that their organizational mission statement explicitly committed them to the long-term preservation of valuable digital materials that they had acquired or created. In 2005, the question was rephrased to ask whether institutional mission statements could be interpreted as supporting long-term preservation: 63% of the 32 institutions reported in the affirmative, 28% said no, and 9% didn’t know. The first challenge for those responding “no” will be to ensure that a strong institutional case for digital preservation can be made: without it the chances for maintaining a program over time will be low.

Asked whether the institutions had any policies and practices that covered long-term access to digital content (as opposed to those covering digitization), 52% indicated that they did, 36% did not, and the remaining did not know. This finding is consistent with that of the RLG Survey of 1998, when half of the institutions with digital holdings reported they had policies for managing them. However, in the RLG survey, these policies included guidelines for acquisition and digital conversion activities. It would be inaccurate, therefore, to conclude that little progress has been made in the development of policies specifically related to digital preservation. Indeed a Digital Library Federation survey in 2001 of its member institutions reported that only 33% of those responding (7 out of 21) had a formal preservation policy. A 2002 preservation survey, by the Council on Library and Information Resources of libraries at leading liberal arts colleges, land grant institutions, and mid-sized universities, indicated that only 6% of respondents (4 out of 67) had developed a preservation plan for digital resources.

Developing policies is a good first step, but they must be vetted and approved at the senior management level, and then implemented for a digital preservation program to develop effectively. Figure 1 indicates that only about one third of the surveyed institutions have completed all three steps.

Figure 1. Comparing the Availability of Policies to the Percent Vetted and Implemented, 2003-2005

In the 2005 survey, we included questions on the development of specific policies covering the following:

• stakeholder roles and responsibilities
• selection, de-selection, and acquisition
• quality creation requirements and procedures
• deposit guidelines
• transfer requirements
• preservation strategies

Figure 2 presents the responses to questions on these topics from the 13 institutions in the 2005 survey that indicated they had policies in place. Over half of these have developed policies relating to stakeholders, selection, and quality requirements. Fifty-four percent have defined digital preservation strategies, compared to 39% reporting guidelines for preservation action in the RLG survey. However, over 40% responded that they did not have formal policies covering deposit guidelines or transfers requirements. These responses indicate key areas needing attention at the institutional level.

Figure 2. Comprehensiveness of Digital Preservation Policies in 13 Institutions, 2005

Technology Infrastructure

Organizations tend to rely on or create digital content first and address long-term access issues later. The technology section of the institutional readiness survey addressed digital content, data management practices, storage, preservation actions, and depository development.

Figure 3 reflects the diversity and pervasiveness of content types managed by the institutions surveyed. Of the 11 digital object types mentioned in both surveys, only one, GIS files, was held by under half of the institutions (45%). Over ninety percent of institutions held websites, digital images, and PDFs; over 85% held word processing files and databases/spreadsheets; over 70% held audiovisual digital content. This represents a significant increase in the prevalence of content types over those reported in the RLG survey. In that survey only 36 of 54 institutions reported digital holdings. Of those, 55.6% reported word processing files, 50% reported audio files, 38.9% video, and 38.9% spreadsheets. Both our survey and the RLG study also looked at the number of formats that an institution needs to maintain. RLG found that the majority of institutions maintained at least six different formats. In the Cornell survey, on average, each institution affirmed responsibility for maintaining 9 of the 11 formats mentioned.

Figure 3. Common Digital Object Types Managed by Institutions, 2003-2005

Digital storage is a significant factor in digital preservation programs. We asked institutions to report the kinds of storage they were using, and Figure 4 reflects the breakdown. Nearly 90% of all institutions reported using optical/magneto-optical disks and 85% reported using online storage. In 2005, we asked participants to distinguish between access copies, master files, and backup in terms of storage options. Figure 5 provides those results. We expected to find that access copies were stored online. What surprised us was the percentage of masters and even backup copies that were maintained online. Similarly, we were intrigued to note that 63% of institutions in 2005 still maintain access copies on a removable storage device (CD, DVD, etc.).

Figure 4. Storage Media Used, 2003-2005

Figure 5. Storage Options Used by Function, 2005

The next set of questions in the survey covered various aspects of the file management program that would support digital preservation, such as the use of high quality media, backups, security, and the like. Responses indicated that most institutions practice good storage management processes in terms of redundancy, media used, and storage location. Figure 6 compares the responses from the 2003-2004 to those in 2005, indicating a slight trend toward better file management practices. For instance, in 2005, 75% reported that they provided an environmentally controlled location for digital content compared to 43% in 2003-2004, a 32 percentage point increase. Fifty-nine percent had disaster recovery plans in 2005 compared with 28% in 2003-2004, a 31 percentage point increase. Neither group, however, has done much to develop a media testing or refreshing/migration program. This is clearly an issue that institutions should address as part of their digital preservation program.
 

Figure 6. File Management Storage Practices, 2003-2005

One of the greatest threats to continuing access to digital content is the rate of obsolescence of file formats, storage media, and the supporting hardware/software to access and use digital objects. In the 1998 RLG study, 41.7% of the institutions reported that they lacked the operational/technical capacity to mount, read, and access some digital materials in their holdings. In 2005, 44% of institutions participating in the Cornell survey reported that same difficulty, although the actual figure may be higher: 28% of respondents left the question blank or indicated they didn’t know.

Responding effectively to obsolescence is an absolute imperative for a digital preservation program. But how many institutions have already risen to that challenge? In the Cornell survey, over 50% of all institutions (2003-2005) had already undertaken actions to extend the life of threatened digital content. Figure 7 breaks this down in terms of specifics for the 2005 respondents: the most common action reported was keeping pace with the changes in storage media.

Figure 7. Actions Taken to Extend the Life of Digital Content, 2005

Digital repository development is a relatively new phenomenon, but those that specifically address digital preservation are of very recent vintage. Two key documents have spurred this activity. The first is the publication of the final version of the Reference Model for an Open Archival Information System (OAIS) in January 2002. Development of OAIS began in 1995 and OAIS became an international standard, ISO 14721, in 2003. The second is the Trusted Digital Repositories: Attributes and Responsibilities, an RLG-OCLC report, published in May 2002. Prior to 2002, the digital preservation community lacked foundation documents like these to serve as the basis for institutional programs and to enable the effective exchange of information and developments between institutions.

We asked institutions to report whether they had established any kind of depository arrangements for managing their digital collections and, if so, how. Over one-third of all institutions reported that they had. Nearly forty percent indicated that their institution was committed to the development or use of one that was OAIS compliant, although another 39% did not know or left the question blank. Figure 8 indicates the type of arrangements for depository development these institutions have chosen. Most are developing in-house solutions—creating their own software or relying on such developments as DSpace and Fedora—rather than contracting with third party services or making consortial arrangements.

Figure 8. Choices made by institutions establishing depository arrangements, 2003-2005

Resources Framework

Once the need to establish a digital preservation program is recognized and there is the will to do so, institutions must be ready to build and sustain the program. This requires the ongoing commitment of resources: financial, human, technical and other.

The survey form used in 2003-2004 asked respondents whether their institutions currently had set aside funding or made an ongoing commitment to the long-term maintenance of digital collections. In 2005, the question was rephrased to ask whether there was funding dedicated for the long-term maintenance of digital collections. A little over one-third of the institutions reported that they did. Many more wrote of relying on one-time monies or grant funds to support the program. An unsettling number of respondents did not know whether there was any ongoing support for the program.

Figure 9. Sustainable Funding for Digital Preservation, 2005

In the 2005 survey we included questions on human resource commitment to digital preservation. First we asked whether there were staff members specifically charged with digital preservation responsibility. 59% responded that there were, 38% replied there were not, and the rest did not know. In addition to staff, organizational and technical expertise is needed to build and sustain digital preservation programs. Impressively, the majority of institutions were confident that they possessed both. This is in sharp contrast to the findings of the 1998 RLG Survey where the lack of staff expertise was a common problem: under 25% of institutions holding digital content ranked their staff as expert. Organizational expertise seems to be in shorter supply than technical expertise, as highlighted in Figure 10. Expertise can be gained through training, and 44% of institutions reported that training was adequately supported. It can also be supplied by outside experts: 34% of institutions participating in the 2005 workshops indicated that they were currently using outside experts.

Dedicated staff and the requisite level of expertise are critical human resources. Neither will turn a burgeoning initiative into an ongoing effort without strong administrative support at the top. Close to half of the institutions reported that their senior management views digital preservation as a key priority. Figure 10 highlights the human resource commitment of institutions participating in the 2005 survey.

Figure 10. Human Resource Commitment, 2005

Overall, survey respondents seemed optimistic about the resources dedicated to their technological infrastructure. Fifty-nine percent felt that their organization had sufficient hardware and software to build and/or sustain a digital preservation program, with requisite upgrades and enhancements overtime. Only 12% did not feel that the infrastructure was adequate (Figure 11).

Figure 11. Adequacy of Current Technological Infrastructure, 2003-2005

Conclusion

There is increasing evidence that cultural institutions are taking seriously the need to safeguard digital heritage materials—in large measure because they can no longer avoid the problem. The correlation between the acquisition of digital content and digital preservation practice was noted in the RLG survey. The prevalence of digital content and concerns over its continuing accessibility was documented in the Cornell survey as well, as were the actions taken to protect these assets. We’ve noted the development of policies, the growing awareness among senior managers, the commitment of resources, the adequacy of technical infrastructure, and direct practical experience with some preservation activities. Probably the biggest difference between the RLG survey and the Cornell survey was the shift in focus from technology concerns to organizational ones. In 1998, RLG member institutions ranked technology obsolescence as the greatest threat to the loss of digital materials. In 2005, respondents to the Cornell survey ranked it as the fourth in a list of five major concerns. Nearly twice as many of them cited insufficient policies and plans as the greater threat. Figure 12 lists the threats to digital content as ranked by twenty-two institutions.

Figure 12. Threats to Digital Materials, 2005

Cornell will continue to collect survey data through 2006. And, as announced in this issue, the Digital Preservation Coalition has launched the UK Digital Preservation Needs Assessment survey. Surveys such as these will help build a picture of the current state of digital preservation activity and identify the most pressing needs. This baseline information will help institutions and consortia alike in planning concrete steps to meet the greatest preservation challenge facing society today. At the end of each workshop, we asked participants what would increase their confidence levels in addressing digital preservation. The most common response was practical experience and hearing from others about what works and what doesn't.

ii.Seven institutions submitted two individual responses to the survey and one institution submitted three responses. When different responses were received for questions that could be answered as “yes,” “no,” or “don’t know,” Cornell counted the “yes” or “no” responses rather than the “don’t know” responses. In the cases that two participants responded with a “yes” and “no” respectively, the affirmative response was kept.

iii. In 2005, the question was rephrased to ask whether the institution had written policies and procedures that addressed specific aspects of long-term access.

http://www.webarchive.org.uk/

Leveraging group resources to meet a common set of requirements, the UK Web Archiving Consortium was formed in June 2004 and has embarked on a 2-year Web archiving project to “investigate the issues of selection and the technical challenges involved. During the course of the project it is estimated that around 6000 websites will be collected and made accessible through this archive.”

The public website, launched in May 2005, presents a clean interface and user friendly navigation. Well–documented project details are provided, as well as access to the collection. Harvested pages from over 490 sites can be searched, or selected sites may be browsed via category, subcategories, or name navigation.

The institutions that comprise the consortium are:

• The British Library
• The National Archives
• The Wellcome Library
• The National Library of Scotland
• The National Library of Wales/ Llyfrgell Genedlaethol Cymru
• The Joint Information Systems Committee (JISC)
  
  

The project has been recognized on the “shortlist” for the Digital Preservation Award, sponsored by the Digital Preservation Coalition.

Are there any recent studies that can help predict how long removable media such as tape cartridges and optical discs will last?

Editor's Note: This FAQ is answered by Dr. Vivek Navale. Dr. Navale is the Information Technology Chief for the US National Archives and Records and Administration (NARA) where he manages a program involving preservation and access of electronic records. The longevity of digital storage media has long been a subject of interest to librarians and archivists. In recent years, there has been no single, comprehensive source of information on media longevity, but a number of agencies have been conducting studies on specific media types. In this FAQ, Dr. Navale presents an overview of three studies that have looked at the life expectancy of high density magnetic tape, CD-ROMs and recordable CDs and DVDs.

Unfortunately, physical deterioration of storage media is only one of the challenges faced in the long-term management of digital information. Although there are scientifically accepted techniques for predicting how long media will remain physically viable, we still have no way to predict when technological change or market forces will cause a particular storage medium to become obsolete. Although this FAQ deals only with physical longevity issues, prudent decision-making about the use of any digital storage medium also needs to take obsolescence into consideration.

Introduction

Measurements that allow prediction of the life expectancy (hereafter LE) of electronic storage media are important for archival preservation and access of electronic records. In 1994, the National Media Laboratory (NML) investigated the stability and life expectancies (LEs) of commonly used magnetic tapes available at that time. The NML study provided detailed information on the effect of environmental conditions on physical and chemical characteristics of different types of magnetic tapes (e.g. IBM 3480 data cartridges, 8mm data cassettes, D1 and D2 digital video cassettes, and VHS analog videocassettes) obtained from various manufacturers. Changes in magnetization and levels of chemical decomposition products from induced stress (elevated temperature and humidity) were analyzed over intervals from weeks to months. Models were developed to provide comparisons of LEs for the various tape products [1].

During the same decade, the National Institute of Standards and Technology (NIST) tested and analyzed 12" WORM (write once read many) optical disc media, and determined LE values. The NIST study monitored changes in error rates, noise ratios, and reflectance under different temperature and relative humidity conditions. An important conclusion of this work was that several factors, including the quality measurement used, the environmental conditions for testing, and the mathematical model used for analyzing the data can affect the LE values [2].

Since the 1990s, there has been a 10 to 1000 times increase in storage density of magnetic tapes and optical discs. Increase in capacity has resulted from use of newer materials and associated technologies. For example, the chromium dioxide recording material in IBM 3480 tapes was replaced by the metal particulate (MP) type materials used in DLTIV and several newer generation higher density tapes.

Many federal agencies (e.g. NOAA, NASA) and commercial institutions are storing and accessing electronic information on higher density media. In 2003, NARA (the US National Archives and Records Administration) revised the Code of Federal Regulations (CFR) to enable agencies to use DLTIV tapes and CDs (compact discs) for transferring permanent electronic records to it. CDs and DVDs are extensively used in government agencies. For example the Library of Congress (LC) has a huge repository of CDs and the Census Bureau and the US Government Printing Office have started creating and maintaining DVD collections. In light of changes in technology and the extensive use of higher density media in the government, the 1990 era studies became inadequate and additional studies were needed that incorporated LE predictions for newer media types.

From 2002-04, NARA, LC, and NIST conducted technical studies on storage media LE. These studies resulted from specific needs and objectives of the individual agencies. Collectively the investigations provide contemporary knowledge and understanding of media longevity and LEs of commonly used removable higher density magnetic tapes and optical discs.

The US National Archives Study on Higher Density Magnetic Tapes

In 2002, NARA directed a systematic study comprised of six technical areas significant for archival preservation of electronic records on higher density magnetic tapes. The study areas included examining the magnetic properties and microstructure of the recording media, physical and recording characteristics, binder chemistry, and error correction/tape drive performance. All testing occurred at Arkival Technology Corporation as part of the NARA contract requirements, and the evaluation of certain higher density magnetic tapes was carried out under accelerated aging conditions. A detailed description of the study requirements and findings are provided in the final report to NARA [3].

Several measures of magnetization were made before and after different combinations of induced temperature and relative humidity (RH) stress conditions. Samples of each of the magnetic tape types (DLTIV, SDLT, IBM 3590) under the specified environmental conditions were taken every 3-4 weeks for a period of 100 to 3500 hours. Every tape type showed a loss in magnetization when they were under induced stress conditions of higher temperature and relative humidity.

Compared to the 1994 NML study of older generation magnetic tapes where significant decomposition of the tape binder was reported, the NARA study on higher density magnetic media did not find such indications. Therefore NARA modeled its LE determinations on the magnetization property instead.

LE predictions for SDLT, DLTIV and IBM 3590 tapes are displayed in Figure 1 at a constant relative humidity (RH) of 30%. At lower values in temperature, for example 20 °C (68 °F at 30% humidity), the predicted LEs for the tapes examined are in the range of 100+ years.

Figure 1 [a]

Figure 1 illustrates the variability in LEs when temperature increases from 20 to 100 °C. At temperatures in the range of 50-60 °C, the predicted LEs are as low as one year. It is clear that relative decline in LE values are significant when the temperature increases from room temperature conditions to 50 °C (122 °F) .

A combined high stress condition of higher temperature and higher RH is illustrated in Figure 2. The overall trend for all the tapes examined is similar, revealing a decrease in the LE values with an increase in RH conditions. The data shows that LEs of the tapes at 50 °C and at 50% RH are in the range of 1-3 years.

Figure 2 [a]

Error Rate Study and LEs

The standard method for determining LEs has been based on accelerated aging techniques, which expose media to higher than usual temperatures and humidity conditions for short periods as a stand-in for more moderate environmental conditions over much longer periods. It is likely that higher stress conditions can accelerate changes in physical or chemical media properties in a shorter time than changes that would occur over longer time intervals in typical storage environments. Data derived from observed changes in physical or chemical properties can be used for LE prediction. In the NARA study, changes in the magnetization property were used to predict LEs.

Errors during recording can result from more than one physical or chemical parameter. Many of the read and write errors during recording by tape drive systems are corrected by the Error Correction Code (ECC) system built into the tape drive. Differences in ECC implementations in tape drive systems with specific media can result in different error rates that can be used as a means for determining the likelihood of media failure.

The NARA study determined Block Error Rate (or BLER, defined as the number of defective blocks reported to number of bytes read or written) under different temperature and humidity conditions. Changes in BLER over time for specific media examined in the study are shown in Figure 3a. Within a short time frame (months), a consistent pattern of increase in errors emerges. A detailed discussion of read and write errors observed during the study, the BLER values, and the means for estimating the threshold value is provided in the NARA study report [3].

Figure 3a [a]

Comparison of LE predictions based on ECC and magnetization changes without use of ECC are shown in Figure 3b. For all media examined, the LE based on BLER values was higher than that based on the magnetization property. This indicates that error correction code can correct many of the read-write errors that may result from deterioration in certain physical and chemical characteristics.

It is important that LE predictions should be viewed more in terms of being able to predict average trends for different media types rather than as an absolute measure of end-of-life values. A LE prediction of 100 years for a specific medium will serve little purpose if an error condition violates the ECC capabilities of a system within a year.

Figure 3b [a]

The NARA study also found that tape media defects can result in failure of the initial write operation, resulting in "hard errors" that are uncorrectable by ECC. However, the more common failure occurs when tape is being read and data is being recovered. Failure of this type allows useable reads of the non-failing portion of the tape until the failure point is reached. The tape then generates a hard failure that can result in loss of electronic data. Failures of this type were observed when tapes were subjected to accelerated aging conditions. Therefore dependence on ECC should not be total. The practice of creating backup tapes significantly reduces the risk of data loss from media failure in an archive facility. Also, the backup tapes should be from different lots than the ones used for the master tapes, to reduce the risk of data loss due to manufacturing defects affecting an entire run of tapes.

The Library of Congress Study on CD-ROMs

In 2004, the Library of Congress (LC) completed a pilot study that evaluated the life expectancy of prerecorded compact discs (CD-ROMs) [4]. The impetus for the LC study was to understand the factors that influence the LE of CD-ROMs. Recordable CDs (CD-R), rewritable CDs (CD-RW), and DVDs were not part of the LC study.

One hundred and sixty prerecorded compact discs were randomly sampled from duplicates present within LC's holdings. The CDs were subjected to environmental stress conditions (temperatures of 60, 70 and 80 C, relative humidity from 55-85%) over a time period of 500-1000 hours.

The rate of deterioration of each specimen was determined by measuring block error rate (BLER) according to techniques specified in ANSI/NAPM IT9.21-1996 [5]. The estimated time to reach end of life for each disc subjected to a particular stress condition was compared to four standard distribution patterns considered the most relevant to this kind of study.

One of these distribution patterns (called lognormal) provided the best fit to the experimental data and resulted in the end of life estimates for the CDs examined (at specified temperature and RH conditions) shown in Table 1 [4].

Table 1: Summary of Lognormal Distribution Figures

The end of life estimate at 80 °C and 70% RH was considered an anomaly and was discarded. Data from table 1 and a linear equation that relates the rate of a chemical reaction to temperature and humidity were used to obtain revised end of life estimates. Figure 4a shows the probability of failure as a function of time at 25 °C and 50% RH. It predicts a mean life time of 1592 years for CD-ROMs stored under those conditions.

Figure 4a [b]

It is important to note that a certain percentage of CD-ROMs maintained at 25 °C/50% RH are expected to fail much sooner than maximum predicted values. Figure 4b provides an indication of the expected percentage failure rate during the first 200 years. The probability for CD-ROMs to fail when maintained at 25 °C and 50% RH during first ten years is estimated to be less than 2% and may increase to 5% over a 20-year period. These estimates apply only to CD-ROMs, not to any form of writable or re-writable CDs or DVDs.

Figure 4b [b]

The NIST study on CD-Rs and DVD-Rs

In 2004, the National Institute of Standards and Technology (NIST) published results of accelerated aging tests conducted on commercially available recordable media (CD-Rs and DVD-Rs) [6]. Changes in Block Error Rates for CD-Rs and parity inner errors (PIE, referring to the number of rows in a data block that contain errors) for DVD-Rs were used as a measure of the quality of the specific commercial media types analyzed during this work. The study also examined the effect of direct exposure to light, including some UV exposure. Samples were exposed to extreme conditions (temperature 60-90 °C and 70-90% RH) with BLER and PIEs measured as a function of increasing exposure time (450-850 hours). BLER maximum has been stated at 220 errors per second in the CD-R specification [6, 7].

Typically, CD-Rs are composed of five components (polycarbonate, the organic dye, the reflective layer, acrylic lacquer, and the label) that can affect the stability of the media. The organic dye is the most fragile of the components, and the composition of the dye can significantly influence the longevity of the media. The commonly used dyes in CD-Rs are phthalocyanine, cyanine, metal stabilized cyanine, and azo. During the NIST work random samples of commercial CD-Rs that included the major dye types were analyzed. Table 2 provides a list of coating and dyes for the various samples numbered S1-S7 used in the NIST study.

Table 2 [c]

Effect of light exposure on test samples

The seven sample sets demonstrated significant variation in BLER values over time when continuously exposed to direct light, as shown in Figure 5a. Phthalocyanine based samples S2 and S4 had the lowest error rates and remained below the end of life threshold for the entire testing period, whereas the azo dye based samples (S1 and S3) showed sharp increases in BLER values before 400 hours of light exposure was reached, the shortest of any of the samples. The two phthalocyanine samples using only silver in the reflective layer (S6 and S7) performed well in direct light exposure until approximately 600 hours, while the silver with metal stabilized cyanine (S5) showed relatively low BLER values until an exposure time of 800 hours.

Figure 5a BLER changes in CD-R test samples after continuous light exposure by using a metal halide lamp [c]

Effect of higher temperature and humidity on test samples -

Again, the samples showed considerable variability in BLER values after exposure to high temperature and humidity conditions (90 °C and 90% RH). Sample set S4, which had the lowest BLER values in the extended light exposure test, also had the lowest BLER values in this test. On the other hand, sample S2, which had the second lowest BLER value in the light exposure test was in a group of samples (also including sets S1, S3 and S5) that spiked high BLER values after less than 100 hours of exposure to high temperature and humidity. Samples S6 and S7 had identical dye and reflective layers with a common manufacturing source, evident by similar overall BLER trends—low values until 100 hours followed by an increase to 1000 by about 400 hours of exposure (Figure 5b).

Figure 5b BLER changes in CD-R test samples after continuous exposure to higher temperature (90° C) and humidity (90% RH) [c]

The NIST study was not able to determine stability of the DVD-Rs based on dye types because information on the dye types of the commercial DVD-Rs could not be obtained. Considerable variability can exist in the quality and LEs of DVD-Rs between manufacturers, even if same dye type is used, because of difference in dye formulations, manufacturing processes and quality control procedures of the DVD-Rs.

Unlike CDs, DVD failure is determined by measuring parity inner errors (PIE) and parity outer errors (POE). A certain number of PIEs can be corrected by the DVD drive's error detection and correction system. However, the POEs are uncorrectable errors by the drives and are not permitted by the DVD specification standards. The NIST study measured PIEs and POEs as a function of time under continuous light exposure and elevated temperature and relative humidity conditions (similar to the experimental conditions used when studying CD-Rs).

The PIEs of DVD-Rs samples varied significantly. As seen in Figure 6a, the PIE for sample D2 remained almost constant even after 800 hours of exposure to direct exposure whereas D1 and D3 showed significant increase in PIEs.

Figure 6a PIE changes in DVD-R test samples after continuous light exposure with metal halide lamp [c]

Even under extreme temperature conditions (90 C, 90% RH), D2 showed little variability in PIE over a 400 hours exposure time period. Both samples D1 and D3 reached PIE of approximately 1000 under identical exposure conditions (as shown in Figure 6b).

Figure 6b - PIE changes in DVD-R test samples after continuous exposure to higher temperature (90 ° C) and humidity (90% RH). [c]

The NIST study profiled two other parameters, jitter analysis for both DVDs and CDs, and E-32 errors analysis for CD-Rs. Variability among sample on exposure to direct light source was observed over similar time periods. The plots and definitions for jitter and E-32 errors are provided in the report [6].

Also, an attempt was made to correlate a high BLER rate to the onset of uncorrectable errors (E32) in CDs, and likewise a high PIE rate to onset of uncorrectable POEs in DVDs.

Using similar accelerated aging methods as discussed above, earlier studies on several CD-Rs with BLER values of 20 per second generated LEs for the CD-Rs tested in the range of 15-30 years [9]. The study also reported observing significant variability in the quality of the media from various source manufacturers. Clearly BLER is a parameter for determining media defects on CDs and can be used a measure for quality testing of the CDs used for maintaining digital information.

Summary and Conclusions

The NARA study showed that measuring changes in the magnetization property under accelerated aging conditions provided LE values for different magnetic tape types that can be easily compared. The DLTIV tapes showed the least change in magnetization values resulting in higher LE values, whereas IBM 3590 and SDLT tapes had lower LE values. Doubling the temperature stress from 20 °C to 40 °C decreased the LEs of magnetic tapes by a hundred times. Extremely low LE values of one year or less was predicted when stress conditions were high (50 °C and 50% relative humidity).

The NARA study found only minor indications of binder decomposition from higher density magnetic tapes whereas the earlier NML study had shown significant indications of chemical decomposition of the magnetic tape binder with older generation tapes. These observations indicate that the composition of binders has improved over the past decade.

Newer tape storage systems also utilize better error correction codes that compensate for read and write errors, including the errors resulting from changes in magnetization. By measuring block error rate it is possible to account for the tape drive's ability to respond to changes in the tape media quality (such as loss of magnetization). For all the higher density tapes examined, the LE values predicted by using block error rate was higher than the LE values predicted by a model that used only change in the magnetization parameter. Therefore meaningful determination of end-of-life of media should monitor changes in block error rate under induced stress conditions over time.

It is evident from the study that catastrophic failures such as first write failures in tapes can occur during operations possibly caused by defective media. Therefore LE predictions serve only as a measure of general trends for media types. An LE prediction of 100 years for a specific media will serve little purpose if the error correction capabilities of the system are unable to correct the error in less than one year. Monitoring error rates over time can be useful in determining the end of life of the magnetic tapes, allowing timely data migration of electronic information onto newer media.

In the case of the CD-ROMs from the LC tests, LE predictions were expressed in terms of probability plots, to estimate the percent of media survival over time under a range of stressful environmental conditions. The models developed indicated that 2% of the CD-ROMs may reach the end-of-life values by 10 years, and the percent of CD-ROMs to fail would increase to 13% by fifty years if the media is stored under normal office environment conditions.

Both the NIST study and previous reports on CD-Rs verified that discs containing phthalocyanine dyes and metal stabilized (gold or silver alloy as a reflective layer) phthalocyanine are more stable than azo and cyanine dyes. For DVD recordable media, it was noted that manufacturers can make modifications to the cyanine dye to improve its stability or to lower manufacturing costs, resulting in similar dye types having considerably different qualities.

In conclusion, measuring BLER values for CDs and magnetic tapes, and PIEs for DVDs under various environmental conditions can provide a method to index quality of media for archival purposes.

REFERENCES

1. John W.C. Van Bogart, "Media Stability Studies," National Media Lab Technical Report RE-0017, pp. 1-86, 1994 .

2. F.L. Podio, "Development of a Testing Methodology to Predict Optical Disk Life Expectancy Values," NIST Special Publication 500-200, 1991

3. R. D. Weiss et.al., "Environmental Stability Study and Life Expectancies of Magnetic Media for Use with IBM 3590 and Quantum Digital Linear Tape Systems," Final Report to the National Archives and Records Administration, Delivered by Arkival Technology Corporation under contract requirement NAMA-01-F-0061, 2002 , pp. 1-94.

4. W.P. Murray and C. Shahani, unpublished Library of Congress draft report on the life expectancies of CD-ROMs, in the possession of the author, 2004.
 

5. Life Expectancy of Information Stored in Compact Disc systems, Method for estimating based on temperature and relative humidity, ANSI/NAPM IT92.1- 1996 .

6. Slattery, O., Lu, R., Zheng, J., Byers, F., Tang, X. "Stability Comparison of Recordable Optical Discs- A study of error rates in harsh conditions," Journal of Research of the National Institute of Standards and Technology, 109, 517-524, 2004 .

7. A. Svensson, CD-CATs SA3 Users Manual, AudioDev Inc., West Des Moines, Idaho, USA, 2000 .

8. DaTarius Inc. (previously CD Associates, Inc) - The DVD1000P Analyzer Manual, Irvine CA, 1998 .

9. J. Trock, Permanence of CD-R media, In Image and Sound Archiving and access: The challenges of the 3rd Millennium, Proceedings of the Joint Technical Symposium Paris 2000 , CNC, France.

10. V. Navale, "Longevity of high-density magnetic media," Advanced Recording Technology Forum (THIC) meeting, 2002.

NOTES

a. Adapted from V. Navale, "Longevity of high-density magnetic media," Advanced Recording Technology Forum (THIC) meeting, 2002. Used with permission

b. Adapted from W.P. Murray and C. Shahani, unpublished Library of Congress draft report on the life expectancies of CD-ROMs. Used with permission.

c. Reproduced from Slattery, O., Lu, R., Zheng, J., Byers, F., Tang, X. "Stability Comparison of Recordable Optical Discs- A study of error rates in harsh conditions." Journal of Research of the National Institute of Standards and Technology, 109, 517-524, 2004 . Used with permission.

TAPE Workshop on Management of Audiovisual Collections
September 28 - October 4, 2005
Amsterdam, the Netherlands

This 5-day workshop will target librarians, archivists, and curators in charge of audiovisual collections. The introductory level courses will explore handling, storage, transfer and conversion methods, technical requirements, and management issues, such as needs assessment, setting priorities, planning, budgeting and outsourcing, and project management.

First International Digital Curation Conference
September 29 - 30, 2005
Bath, UK

The UK Digital Curation Centre (DCC) will sponsor the First International Digital Curation Conference. The conference will hightlight DCC activities and sessions on topics including Socio-Legal Issues, Format Registries, Storage Media, Training & Staff Development, and Certification.

LITA 2005 National Forum
September 29 - October 2, 2005
San Jose, California

The theme of this year’s Library & Information Technology Association Forum will be “The Ubiquitous Web: Personalization, Portability, and On-line Collaboration.” This small conference format will offer networking opportunities, speaker and poster sessions, and vendor exhibitions.

Digital Preservation Training Programme
October 10 - 14, 2005
Coventry, England

As part of the JISC-funded Digital Preservation and Asset Management Programme, this week-long residential training program will address practical training and support for a broad range of staff involved in managing digital information. Topics will be geared towards, but not limited to, Higher Education and Further Education institutions. The cost of this pilot session will be free, but participants will be responsible for the costs of their travel and accommodations.

Long-term Curation and Preservation of Medical Databases
October 13 - 14, 2005
Lisbon, Portugal

This ERPANET / Digital Curation Centre-sponsored workshop will bring together information professionals, medical practitioners, commercial developers, students, and researchers to consider preservation of medical databases. Participants will have opportunities to network with colleagues, learn about current developments, discuss their own experiences, and to identify risks and challenges to the long-term preservation of medical databases. The format of this cross-disciplinary meeting will include presentations, panel discussions, and breakout sessions.

CERN Workshop on Innovations in Scholarly Communication (OAI4)
October 20 - 22, 2005
Geneva, Switzerland

This fourth workshop in the series, which began as the Open Archives Initiative Workshop in 2001, will present a forum for technical issues associated with scholarly communication. Sessions will include tutorials, technical presentations, library and publishing community presentations, and breakout groups.

Managing Digital Assets: Strategic Issues for Research Libraries
October 28, 2005
Washington, DC

This forum, sponsored by ARL, CNI, CLIR, and DLF, will be held at the Mayflower Hotel immediately following the ARL Membership Meeting.  It will consist of an overview by Donald J. Waters of The Andrew W. Mellon Foundation followed by three program sessions on policy and tool developments for managing digital assets.  The discussion focuses on issues of concern to decision makers at research institutions.

ASIS&T 2005 Annual Meeting
October 28 - November 2, 2005
Charlotte, North Carolina

The theme of the 2005 annual meeting of the American Society for Information Science & Technology will be “Sparking Synergies: Bringing Research and Practice Together.” The second International Conference on Knowledge Management (ICKM2005) will be held at the same venue just prior to the ASIS&T Annual Meeting, October 27 - 28, 2005.

Persistence of Memory: Stewardship of Digital Assets
November 1-2, 2005
Boston, Massachusetts

A pre-conference to the Museum Computer Network's 2005 Conference (see below), this Northeast Document Conservation Center (NEDCC) event will provide a comprehensive overview of managing and preserving digital assets for the long term.

DIGITS FUGIT! Preserving Knowledge into the Future
November 3-5, 2005
Boston, Massachusetts

The theme of this year's Museum Computer Network (MCN) Annual Conference is the challenge digital preservation presents to the museum community and beyond. The keynote speaker will be Alexander Rose, Executive Director of the Long Now Foundation.

Archival Perspectives in Digital Preservation at SAA
November 3 - 4, 2005
Richmond, Virginia

This Society of American Archivists’ (SAA) seminar will draw on a growing technical literature to define digital preservation requirements and explore concepts such as integrity, authenticity, and trust in digital preservation. Specific programs, including the work of OCLC/RLG, InterPARES, and selected European initiatives, will be reviewed.

Metadata and Semantics Research (MTSR'05)
November 21 - 30, 2005
Online

The first On-Line Conference on Metadata and Semantics Research (MTSR'05) will aim to gather researchers and practitioners “interested in metadata, its representation, its semantics and its diverse applications to Information Systems.” The on-line format of the conference will encourage demonstration of systems and prototypes and allow for presentation of schemas, ontologies, learning objects, and sample metadata records.

DASER: Digital Archives for Science & Engineering Resources
December 2 - 4, 2005
College Park, Maryland

The second DASER meeting will feature speakers examining new issues and challenges related to digital archives and Science-Technology-Medicine publishing. Specific topics will include: impact of OA on the future of STM libraries, institutional repository models, publisher-library collaboration strategies, and user needs and patterns related to digital libraries.

DPC Digital Preservation Award Shortlist Announced

The Digital Preservation Award, sponsored by the Digital Preservation Coalition and part of the Conversation Award program, recognizes new initiatives in digital preservation. The award will be bestowed in November 2005 to one of the following finalists:

• Choosing the Optimal Digital Preservation Strategy, Vienna University of Technology
• Digital Preservation Testbed, National Archives of the Netherlands
• PREMIS (Preservation Metadata: Implementation Strategies), PREMIS Working Group
• Reverse Standards Conversion, British Broadcasting Corporation
• UK Web Archiving Consortium, The Consortium

GPO Releases Digitization Specification for Review and Comment

Following internal and external reviews, the U.S. Government Printing Office's (GPO) has released its Digitization Specification version 3.0 with requests for review and comment. The specification will be used to support GPO's plan to convert legacy U.S. government documents into master files suitable for preservation and for making derivative access files. A digitization pilot project will be undertaken to validate the specification.

The International Calendar of Information Science Conferences

The Special Interest Group on International Information Issues (SIG/III),
the European (ASIST/EC) and the New England (NEASIST) chapters of ASIS&T
(American Society for Information Science & Technology) have launched a searchable calendar of relevant conferences. Submissions are invited and moderated. The calendar is also available via RSS/XML.

State of the Nation Survey for UK’s Digital Heritage

The Digital Preservation Coalition (DPC) has launched the UK Digital Preservation Needs Assessment survey. The survey has been designed to reveal the extent of the risk of loss or degradation to digital material held in public and private sectors and to explore existing digital preservation activity in the UK.

Redesigned Geospatial Electronic Records Website

The Geospatial Electronic Records website contains resources on managing and preserving geospatial data and related electronic records. The website includes a Guide to Managing Geospatial Electronic Records; a Data Model for Managing and Preserving Geospatial Electronic Records; categorized references to related resources; project presentations; and materials from “Extending the Boundaries: A Workshop on Managing and Preserving Geospatial Electronic Records.”

Getty Funds Women Artists Archives National Directory (WAAND)

WAAND will contain information on primary source materials of and about contemporary women visual artists active in the U.S. The intended audience includes scholars, artists, curators, students, and collecting institutions. Participating repositories will be asked to complete an online survey form that is due to be released in September.

RLG DigiNews (ISSN 1093-5371) is a Web-based newsletter conceived by the RLG preservation community and developed to serve a broad readership around the world. It is produced by staff in the Department of Research, Cornell University Library, in consultation with RLG and is published six times a year at www.rlg.org.

Materials in RLG DigiNews are subject to copyright and other proprietary rights. Permission is hereby given to use material found here for research purposes or private study. When citing RLG DigiNews, include the article title and author referenced plus "RLG DigiNews." Any uses other than for research or private study require written permission from RLG and/or the author of the article. To receive this, and prior to using RLG DigiNews contents in any presentations or materials you share with others, please contact Jennifer Hartzell (jlh@notes.rlg.org), RLG Corporate Communications.

Please send comments and questions about this or other issues to the RLG DigiNews editors.

Co-Editors: Anne R. Kenney and Nancy Y. McGovern; Associate Editor: Robin Dale (RLG); FAQ Editor: Richard Entlich; Contributor & Copy Editor: Ellie Buckley; Production: Jenn Colt-Demaree, Carla DeMello; Advisor: Peter Hirtle.

All links in this issue were confirmed accurate as of August 15, 2005.