Debunking of Specsmanship

Don Williams
Eastman Kodak Company
don.williams@kodak.com

Introduction

And so it is with the claims of the capture performance of digital imaging devices. One is beckoned by a cacophony of vendor specifications. Loud and confusing (and unregulated), they are, ironically, seductive. The greater the numerical extreme, the greater the allure. For the inexperienced, evaluating these assertions in a consistent scientific sense is futile. Indeed, even the experienced are disadvantaged without the appropriate tools and guidelines. Let’s face it—for the most part we use these marketing specifications, along with brand name and price, as imaging performance guides.

For serious amateurs and professionals with demanding projects or clients, relying on this information as an imaging performance indicator is precarious, especially in the context of the constraints of high-productivity workflow. The difference between sampling frequency (dpi) and true resolution is confusing. Improved "optical" resolution claims remain suspect. Bit depth alone is far from a sufficient criterion for specifying dynamic range, and the existence of artifacts and noise are dismissed with a shrug. Unlike the world of analog imaging, where one could confidently rely on the history-rich reputation of a few manufacturers for performance integrity, today’s digital imaging landscape offers far fewer assurances.

The efforts of ISO/TC42 toward imaging performance standardization are slowly, but surely, changing this free-for-all. Through the participation of scientists and device manufacturers, a unified architecture of objective signal- and noise-based metrics is evolving to help remove device performance ambiguity and permit robust cross-device comparison. Adapted from proven approaches over a half-century of analog imaging experience, these metrics can be used as figures of merit in their own right or may be extended as weighted inputs into higher-order models of image quality. Some are still the subject of ongoing research. They are not perfect, but nonetheless offer the best compromise between technical rigor and ease of use.

Of course, the simple issuance of a standard will not ensure its adoption. For this, education, enablement, and improvement efforts are necessary. These have not been forgotten by the TC42 committee members who complement the standard itself by way of classes, technical papers, free software, benchmark testing, and target creation. These are perhaps more important than the documentation itself because they provide practical exercising of the standard by interested users who in turn provide feedback that allows improvements to the standard’s practice.

The progress, status, and content of the following TC42 imaging performance standards will be discussed in the indicated groupings. Associated with each standard is an ISO status that ranks, in order, its progression toward full ISO adoption.

Terminology

Frequently forgotten among the techniques and practices outlined in technical standards is the definition, or terminology, section. Although individual standards typically carry their own terminology section, ISO 12231 is a collective document that draws from a number of TC42 electronic-imaging working groups (WG 18, JWG 20, JWG 23). As such, it provides a broad perspective on electronic-imaging terms. Occasionally, definitions from working groups involved with traditional imaging are also included for completeness.

Opto-Electronic Conversion Function—OECF

At the foundation of nearly all of the ISO/TC42 performance standards is the opto-electronic conversion function (OECF). Similar to a film’s characteristic curve that plots the transfer of exposure into optical film density, the OECF defines the relationship between exposure, or reflectance, and digital count value of a capture device. By itself the OECF appears low tech, but it allows one to evaluate the effective gamma applied to an image, as well as any unusual tonal manipulations. Its real power, though, lies in acting as a tonal Rosetta Stone for remapping count values back to a common, physical image evaluation space so that meaningful cross-device performance evaluation can be done. It is also the hub for all of TC42's performance standards, which is why it is cited and used so frequently in all the performance standards.

The only dedicated standard to OECF is for digital cameras, ISO 14524. Nevertheless, the use of OECF for film and print scanners is described and required as defined in the standards' annexes peculiar to these devices. Though the OECF is intimately tied to other performance metrics, its calculation was always made from separate image captures rather than from metrics of prime interest. This led to inconsistent results between captured frames because of auto-contrast or scene-balance algorithms associated with capture devices. For this reason, gray patches for OECF calculation are now being integrated into targets for all the other performance metrics.

Although performance metrics tend to be considered from a benchmarking perspective, monitoring the OECF on a periodic basis for QC purposes is the greatest benefit. A good example of this can be found in a paper by Johnston.[1] Unless one is confident that auto-contrast features are not being invoked, capture devices cannot be counted on to have a unique OECF. This is why small gray-scale patches are frequently placed alongside documents at the time of digitization and remain part of digitized images. If designed correctly or included as metadata, they will provide an unambiguous tone path to the source document for faithful future rendering.

Resolution

This observation was made with respect to traditional analog imaging more than a decade before digital imaging began to become popular. Now, sampling and interpolation associated with digital imaging has made the term "resolution" even more ambiguous.

The advertising of device resolution in terms of finished image-file size is perhaps the most misleading of all. Through interpolation an infinite amount of "empty" resolution can be synthetically created that has no physical bearing on spatial detail detection (i.e., real resolution). Short of removing the detector from the camera and physically counting the sensor sites (ugh!), there is no way for the casual user to know the difference. Fortunately, through education, litigation, and standards this practice is becoming less common.

Though simple pixel count (e.g., Mpixels) and sampling frequency (e.g., dpi) are always cited and easy to understand, Mother Nature frowns at such laziness. She requires that optics, motion, image processing, and electronics contributions also be considered as factors influencing a device's true resolution. Then, and only then, is realistic spatial resolution determined. For this, the measurement of spatial frequency response (SFR) or modulation transfer function (MTF) of a device is required. These measurements unify the spatial resolution standards for electronic capture devices under TC42 and are described for cameras (ISO 12233), reflection scanners (ISO 16067-1), and film scanners (ISO/CD 16067-2). Each of these standards adopts a common slanted-edge-gradient MTF analysis technique especially suited for digital capture devices. Its accuracy has been benchmarked [2] with both synthetic and real image data. Its chief advantages are ease of use, durability, and analytical insight.

The suitability of MTF as an objective tool to characterize spatial imaging performance is well documented and has been used as an image-quality prediction tool for more than fifty years.[3] By characterizing contrast loss with respect to spatial frequency, one of its many uses can be to objectively establish the limiting resolution of a device. This is done by determining the spatial frequency associated with a given MTF value, typically 0.1. This frequency is then translated into limiting resolution for a given set of scan conditions and compared to the manufacturer’s claim to determine compliance. An example of this for a reflection scanner at three different sampling frequencies is shown in figure 1.

Fig. 1

Notice that the MTFs for each sampling frequency (250, 300, and 500 dpi) are essentially identical. The individual curves of figure 1 are difficult to identify because they literally overlay. This indicates no real resolution advantage at 300 and 500 dpi compared to the 250 dpi scan. This is indisputable. The 0.1 modulation level corresponds to 4 cycles/mm. Translating this to an effective resolution (dpi = (cycles/mm) * 50.8 ~ 200 dpi ), one finds that this scanner is really no better than a 200 dpi scanner, no matter what the advertising claims or sampling frequency. This analysis was performed with tools provided through the TC42/WG18 standards group and is one of many examples where they have been used to objectively clarify resolution performance.

Parenthetically, informative references to ISO 16067-1 detail methods to extract sub-pixel color channel registration errors from the ephemeris MTF data.[4] This artifact is often a problem with linear array scanners and is quantified in the analysis tools provided through the I3A Web site. Color misregistration as large as 1.5 pixels was calculated in the scanner of figure 1 with the same tools used for MTF calculation.

In the past, MTF measurement was confined to laboratory settings and never matured as a particularly field-friendly method for objectively determining resolution—a requirement for widespread adoption and credibility as a standard. This hurdle to acceptance has now been largely removed. Through the efforts of TC42 members, free automated software, debugging, affordable high-bandwidth targets, technique documentation, and educational workshops have been provided. The remaining challenges lie in the manufacturing and design of robust targets for film scanners and in improvements to target design for cameras.

Noise and Dynamic Range

Part of the seduction of digital imaging is the myth that it is noise free. By proclaiming a lack of film grain, this claim implicitly suggests that this is so. To demystify this, two ISO/TC42 standards are in development that define noise and dynamic range measurements. ISO/FDIS 15739 is intended for digital still cameras, and ISO/CD 21550 for film/print scanners. The camera standard (15739) is primarily intended to measure noise, but it also makes recommendations on dynamic range. Similarly, the film/print scanner standard (21550) is primarily intended for dynamic range measurement where noise characterization is required. Both standards use identical techniques for characterizing dynamic range and noise and are described next.

For the uninitiated, assessing dynamic range in the context of noise may not be obvious. After all, most claims for dynamic range are typically tied to device bit depth alone: the higher the number, the better. For instance, 12 bits/color (4096 levels/color) would indicate a precision of 1 part in 4096, or a maximum optical density of 3.6.[5] These simple calculations of dynamic range may be suitable for tutorials on concept capability but are far from sufficient for real imaging performance. To understand why, a qualitative definition of dynamic range as applied to imaging applications is needed.

I propose the following:Dynamic range—the extent of light over which a digital capture device can reliably detect signals—reported as either a normalized ratio (xxx:1) or in equivalent optical density units.

The operative words in this definition are reliably detect. Detection is a function of signal strength (think contrast), so the stronger the better, in this case. The reliability, or probability, of that detection is a function of the noise associated with that signal, so the lower the better. This logic suggests that maximizing the signal-to-noise ratio (SNR) is appropriate for increasing the dynamic range of a device. This was not lost on the members of TC42/WG18. Thus, SNR is integral to measurements of dynamic range under the cited standards. They marry signal with the probability of detecting that signal; that is, noise. So far, so good. We now know what to measure. Knowing how to measure it is more complex.

Both standards have taken the high road and adopted an incremental SNR approach to the metrology mechanics. Of all the ways to measure signal, incremental signal is probably the most informative for realistic imaging use. An example of this for an 8-bit reflection scanner is given in the top of figure 2. Its utility lies in quantifying how well a given intensity, Io, can be distinguished from another intensity of an arbitrarily small difference, I. Unlike the simple counting of bits, which is a capability measure, this usage of dynamic range is a performance measure as dictated by everyday needs. In the context of noise it will answer questions like, "Can this capture device distinguish between an optical density of 1.00 and 1.10?"

The other portion of dynamic range measurement is device noise characterization. This is determined through a "noise cracking" technique.[6] This step is extremely important for scanners because noise due to the target often accounts for the majority of the total noise. This target noise must be discounted so that the scanner itself is not discredited. The center graph of figure 2 illustrates the noise function.

Taking the ratio of the incremental signal to noise at each OECF patch yields the incremental SNR function. An example of this for a reflection print scanner is illustrated it the bottom of figure 2. Dynamic range is then determined from the incremental SNR by noting the density at which a prescribed SNR value is met. For instance, using a typical value of six, the scanner of figure 2 would have a dynamic range of roughly 1.5 or 32:1. This measure of dynamic range is significantly lower than the noiseless and flare-free capability measure of 2.4 that a simple bit count yields.

Conclusion

Ultimately, the success of any standard is measured by its level of adoption. And adoption is achieved through regulation, education, and enablement. The regulation metrology for digital imaging performance is well on its way through the vetting and review process provided by ISO/TC42. This paper details the content and status of these standards and includes a view of the scientific rationale for each. With an aim of combining technical rigor and utility, a sound architecture of signal and noise metrology techniques has been established. Though not perfect, the goal is to have them evolve so that one day they may nearly be so.

Education and enablement can accomplish this. Automated, easy-to-use software; economical targets; publications; and presentations through I3A and committee members have provided a good start toward this goal. The use of these tools and resources is beginning to allow users to accept, refute, or at least question manufacturers' "siren calls" in a scientifically sound and unified manner.

Acknowledgments
Gratitude to Jack Holm, Peter Burns, Sabine Süsstrunk, Ken Parulski, Sean Kelly, and Bruce Pillman for insightful talks on the theoretical and practical aspects of the above standards.

Footnotes
[1] Daniel L. Johnston, "A Simplified Method of Digital Image Tonal Capture for Archival Projects," IS&T PICS Conf., p. 210-213 (2002). [back]
[2] Don Williams, "Benchmarking of the ISO 12233 Slanted-Edge Spatial Frequency Response Plug-in," IS&T PICS Conf., p. 133-136 (1998). [back]
[3] T.H. James, "The Theory of the Photographic Process," p. 604-606 (1977). [back]
[4] Peter Burns and Don Williams, "Using Slanted-Edge Analysis for Color Registration Measurement," IS&T PICS Conf., p. 51-53 (1999). [back]
[5] Anne R. Kenney and Oya Y. Rieger, "Moving Theory into Practice, Digital Imaging for Libraries and Archives," Research Library Group, p. 40-41 (2000). [back]
[6] Peter Burns and Don Williams, "Distilling Noise Sources for Digital Capture Devices," IS&T PICS Conf., p. 132-136 (2001). [back]

Publishing Information

RLG DigiNews (ISSN 1093-5371) is a newsletter conceived by the members of the Research Libraries Group's PRESERV community. Funded in part by the Council on Library and Information Resources (CLIR) 1998-2000, it is available internationally via the RLG PRESERV Web site. It will be published six times in 2003. Materials contained in RLG DigiNews are subject to copyright and other proprietary rights. Permission is hereby given for the material in RLG DigiNews to be used for research purposes or private study. RLG asks that you observe the following conditions: Please cite the individual author and RLG DigiNews (please cite URL of the article) when using the material; please contact Jennifer Hartzell, RLG Corporate Communications, when citing RLG DigiNews.

Any use other than for research or private study of these materials requires prior written authorization from RLG, Inc. and/or the author of the article.

RLG DigiNews is produced for the Research Libraries Group, Inc. (RLG) by the staff of the Department of Research, Cornell University Library. Co-Editors, Anne R. Kenney and Nancy Y. McGovern; Production Editor, Martha Crowe; Associate Editor, Robin Dale (RLG); Technical Researchers, Richard Entlich and Peter Botticelli; Technical Coordinator, Carla DeMello; Technical Assistant, Valerie Jacoski.

All links in this issue were confirmed accurate as of February 15, 2003.

Please send your comments and questions to RLG Diginews Editorial Staff.