CuBISM Computationally-Based Imaging of Structure in Materials

Computational Imaging of Composite Materials:

Micro to Macroscale Interactions

The aim of this portion of the project is to collect property information from complex material mixtures—painted surfaces—through computational imaging methods. Material properties at the nano- and microscopic scales are inferred from optical measurements of macroscopic appearance.

Expertise and Overall Approach

For highly scattering media, such as paint, analytical light transport models are generally used. These models assume a homogeneous or, at best, a multi layered medium, which is a rough simplification of the real, often complex, material. Since the obtained spectra contain information about the whole probed volume, we must accurately capture and model light transport through these complex morphologies. Our key technical novelty is to provide solutions to both the forward and inverse problems by restricting solution space for the absorption and scattering coefficients as well as the fluorescence response by using material dictionaries. By doing so we are able to obtain the following:

• Accurate estimations of the bulk material properties in non-homogeneous stratified layers of densely scattering materials (e.g., paintings, opacified glasses, ceramics, and corrosion layers) directly from optical measurements of macroscopic appearance.
• Improved imaging techniques and image processing algorithms to estimate bulk material properties over a wide field of view (e.g., from 10 – 100 cm2 surface areas), extending into the microscopic volume of non-homogeneous scattering layers (e.g., 200-1000 mm depths).
• An improved ability to predict and remedy material deterioration over time.

 

Topic 5: Multimodal Imaging to Produce Material Dictionaries

Northwestern Student: Bingjie (Jenny) Xu, Computer Science

Primary Advisors: Marc Walton (NU), Jack Tumblin (NU), Maurice Aalders (University of Amsterdam)

Northwestern Postdoctral Scholar: Pengxiao Hao, Materials Science and Engineering

Primary Advisors: Marc Walton (NU), Jack Tumblin (NU), Maurice Aalders (University of Am

Key Outcome: Development of Material Dictionaries and Databases Relevant to Art Conservation, which will tie into the Analysis-by-Synthesis framework described in topics 6 and 7.

The aim here is to develop techniques that enable useful property information to be extracted from complex material mixtures using a so-called “analysis-by-synthesis” approach  as described in further detail below in Topics 6 and 7. Data from painted surfaces will be collected using several compatible modalities at different length scales to provide both compositional as well as molecular information. To start, instead of working on actual works of art, imaging will be done on “model paint systems” of common artist pigments, pigments mixtures, and pigmented layers to arrive at the “virtual paint” described in the introduction of this proposal. These paint systems will be informed by the basic information contained in such volumes as the Pigment Compendium, the Artists’ Pigment series, the pigments in the Forbes collection at the Art Institute of Chicago — one of the most comprehensive collection of artist pigments available—to make imaged-based libraries of material data, and will use the wealth of information already contained in material databases, such as the Fiber Optic Reflectance Spectra database housed at The “Nello Carrara” Institute of Applied Physics (IFAC), Florence. Images for this topic will be collected using the imaging facilities both at Northwestern as well as in Europe in collaboration with the University Amsterdam and at Synchrotron SOLEIL. Paint samples will be interrogated by VIS/NIR spectral measurements to extract absorption and scattering characteristics of the pigments. The data collected from these cameras will provide information on how light is reflected and absorbed from these surfaces across visible through near infrared wavelengths. In addition to these spectral measurements, specular and diffuse components of reflectance will be estimated using surface-oriented reflectance techniques. Complimentary to the reflectance measurements, a macro X-ray fluorescence imaging system at Northwestern will be used to extract information on the elemental composition of the materials. This technique is capable of producing spatial maps of elemental data and is suited to discriminating hidden layers of information when an underpainting or underlayer has a significantly different composition to the overlying layers.

Current Focus: Optical Coherence Tomography (OCT) has become attractive in the field of cultural heritage due to its ability to probe the hidden layers of artworks in a non-invasive and non-destructive way. While OCT systems with the central wavelengths from 900 nm to 1300 nm are commercially available for biomedical uses, a central wavelength at 2200 nm is better suited to detect pigment layers, due to the larger penetration depth. Liang et. al. have demonstrated that a custom-built 2.2 um Fourier domain OCT (FD-OCT) system significantly improved the imaging depth through oil-paint pigment layers such as titanium white, cobalt blue, and yellow ochre. However, these FD-OCT systems incorporated a customized high-power laser source and a costly broadband spectroscopic camera. To make the system more accessible to a broader user community in the field of cultural heritage, we are currently focusing on building an inexpensive alternative assembled from commercially-available parts; a 2-um time domain OCT (TD-OCT) system with comparable resolution (axial resolution of 4.85 um), signal quality, and penetration depth. The system reduces cost and equipment complexity substantially but does require modest increases in measurement time.

In addition to pigments, we are also studying holographic films using TD-OCT and related techniques such as X-ray fluorescence and transmission electron microscopy. We hope to use OCT to obtain the layered structure of the holographic films to identify their manufacturing and processing techniques. By combining with other characterization methods, we aim to understand the chemical and physical properties of holographic films for documentation and conservation purposes.

To demonstrate that long wavelength OCT can improve penetration depth, Figure 6 compares the imaging results of our 2 um TD-OCT with a commercial 900 nm FD-OCT for a layer of indigo-linseed-oil on a glass slide. While the indigo layer exhibited strong scattering at 900 nm, as indicated by the absence of the indigo/glass interface, this interface was clear at 2 um (top of Fig. 6a). In addition, the 2-um result revealed the bottom of the glass slide, confirming indigo layer penetration sufficient to measure its underlying layers. The OCT image matched the physical thickness of the indigo layer (Fig. 6c), further validating the result of our 2 μ m OCT. In addition to indigo, the 2 μ m laser also proved better penetration ability than the 900 nm laser for several other pigments such as yellow ochre (iron oxyhydroxide) and alizarin crimson (1,2-dihydroxyantraquinone). Another result with a three-layer sample containing yellow ochre, smalt and chalk is shown in Figure 7. The left one is the sample we are using, and we do a B-scan in the red area. The top right figure is a schematic diagram of the scanning result. The middle right figure is the scanning result of 900 nm commercial OCT and the bottom right figure is from our lab-built 2 μ m OCT. Notice that the bottom surface of the first layer in 900 nm result is indistinct. However, in 2 μ m result, we can see the clear interface and get actual optical depth of each layer.

Figure 6: A layer of indigo-linseed-oil on a glass slide scanned by a) lab-built 2 μm TD-OCT and b) commercial FD-OCT with a central wavelength of 900 nm. c) The physical thickness of the paint film measured by a profilometer.

Figure 7: Three-layer sample of yellow ochre, smalt and chalk on a glass slide scanned by a lab-built 2 um TD-OCT and a commercial FD-OCT with a central wavelength of 900 nm.

We have also successfully obtained the layered structure of a hologram using OCT (Fig. 8a); the thickness of silver bromide-gelatin-emulsion layer is ~6 μ m. Scanning electron microscopy (Fig. 8b) and energy-dispersive X-ray spectroscopy (Fig. 8c) have confirmed this result, suggesting OCT as a reliable non-invasive technique for holograms. In addition, we have been able to characterize the shape and the distribution of silver bromide nanoparticles in the gelatin emulsion using scanning transmission electron microscopy (Fig. 8d); it may provide valuable information to retrieve the processing chemistry of the hologram.

Figure 8: Characterization of a holographic film using a) scanning electron microscopy, b) energy-dispersive X-ray spectroscopy, c) 900 nm OCT, and d) scanning transmission electron microscopy.

 

Topic 6: Forward Image Modeling

Northwestern Student: Lionel Fiske, Engineering Sciences and Applied Mathematics

Primary Research Advisors: Oliver Cossairt (NU), Maurice Aalders (U. Amsterdam)

Key Outcome: Development of novel rendering algorithms that incorporate multiple scattering and absorption and enable inference of spectra data with the use of OCT data

A set of multimodal optical measurements will be fused together to explore a core problem in the imaging of materials – how light is absorbed and scattered in complex media, be it skin,  ceramics, glass, plastics etc. In this project, we focus on physical modeling of macroscopic (electro-magnetic/optical) appearance to enable non-destructive chemical analysis of complex, art-related materials. A key component of our analysis by synthesis framework is a forward model that can faithfully render the images captured by our cameras.  We will use a Monte-Carlo Radiative Transport (MCRT) rendering framework as a forward model for our multimodal measurements. The renderer will take into account the complex paths of individual photons as they are scattered through the layers of pigments, as well as propagation through the imaging optics, in order to faithfully simulate images captured by our OCT and hyperspectral imaging instruments. In addition, our forward modeling will incorporate the effects of fluorescence, so that we can accurately model the images captured by our XRF-mapping instrument. Our inference framework will utilize this forward model to predict macroscopic appearance based on estimates of material parameters. Initial estimates of material parameters will be derived from Optical Coherence Tomography and hyperspectral imaging measurements. These estimates will be progressively refined using the optimization procedures developed in Topic 7.

Current Focus: Earlier work demonstrated the potential of a modeled, dictionary-based reflectance approach for pigment classification. This technique identifies pigments by searching for pigment combinations which best reproduce the measured reflectance curve. Further, this method can be used in addition to XRF helping to address some of the ambiguities in the XRF measurement such as pigments with similar elemental signatures. Thus far, this approach has only been used in single paint layer systems; however, artists often layer paints to create various optical effects. In this situation the reflectance cannot be assumed to come from a single layer. Current work is addressing this gap by generalizing this modeled reflectance method to multi-layered systems. In particular, Lionel is considering two-layered systems in which a thin layer of paint sits on a very thick layer of paint.

To solve the layered unmixing problem, inverse modeling techniques have been coupled with modern deep network approaches, using the conceptual framework outlined in Figure 9. The transport of light through a paint layer is modeled using the Kubelka-Munk equations. In this model, each layer has a wavelength dependent scattering and absorption coefficient giving the layer its appearance. These parameters can be deduced by taking a series of reflectance measurements. We have measured a small set of paints to demonstrate the approach and are currently measuring a larger set of these coefficients to allow us to simulate many real potential paint configurations. Using these simulations, a deep neural network is trained to identify paints in layered systems. Once the pigments are identified, it becomes straightforward to find the correct relative concentrations of the pigments to match the desired output. We have tested this approach on simulated data and been able to achieve very admirable accuracies (90% of pigments correct to within 10% of actual relative concentration).Figure 9: This figure demonstrates the proposed algorithm for layered pigment identification. The reflectance and thickness data are input into a neural network which returns pigment candidates. Then using a non-linear least squares regression approach the pigments are then identified.

 

Topic 7: Inverse Image Modeling

Northwestern Department: Henry Chopp, Electrical Engineering and Computer Science

Primary Advisors: Aggelos Katsaggelos (NU), Maurice Aalders (U. Amsterdam)

Key Outcome: Development of dictionary learning approaches for solving inverse problems in material identification and spectral unmixing

In this topic, we extend previous work on data fusion to increase the resolution of certain modalities  (for example, increasing the spatial resolution of x-ray fluorescence to that of RGB images).  Furthermore, we will build nonlinear atoms and apply dictionary based sparse representation by using Kubelka-Munk theory to untangle the problem of nonlinear spectral mixing of pigment particles in complex mixtures. Other learning techniques, such as deep learning and analytical techniques will also be investigated.  We are are also developing material identification algorithms for paintings, using the database of macroscopic optical measurements of common artist pigments created in Topic 5. The results of this task will provide the initial estimates of the scattering and absorption parameters for each segment provided by the segmentation of the OCT images.

Current Focus: We have developed algorithms to perform sparse sampling of paintings using XRF as the modality. Using an XRF scanning probe takes a significant amount of time to complete a full raster scan. Our primary objective is to subsample the painting in an effort to reduce the scan time, then recover the missing samples using an inpainting technique. Currently, our objective is to develop a novel subsampling technique that continuously scans a painting along a trajectory. This will further decrease the scan time in comparison with the techniques we have developed in the past year.

While we are still in the process of developing a technique for continuous subsampling along a path, we have made significant progress in the subsampling and reconstruction domains. For subsampling, we have developed a novel convolutional neural network that generates a sampling mask using an RGB image as the input. This network can both binarize and control the sparseness of the sampling mask it generates. Figure 10 shows the network architecture we developed. After applying many different inpainting algorithms to many different sampling mask generation algorithms, we found that our network’s sampling mask outperformed other state-of-the-art subsampling algorithms across all inpainting techniques.

Figure 10: The architecture for the sampling mask generation network.

For the XRF reconstruction, we developed another new algorithm to greatly improve the quality of the reconstructed image. We posed the problem in an optimization framework whereby we use dictionary learning to recover the image. We assume we have both a subsampled XRF image that is composed of a surface and underneath component, as well as an RGB image (just the surface component) registered with the former. In the decomposition of the three signals, the same abundancy matrix is used for the surface components. The RGB image can also be used to smooth the reconstructed surface-level XRF signal. This is the basis of the solution framework we developed. Figure 11 shows the full pipeline from sampling to recovering the XRF image.

Figure 11: Top left: Jan Davidsz. De Heem’s Bloemen en Insecten, the painting we would like to analyze. Top right: Feeding the RGB image into our convolutional neural network, we select 20% of the samples to collect. Bottom right: Based on the mask, we collect the XRF samples. d) Using our optimization algorithm, we recover the missing values.