【ENGINEERING】Deep learning in optical metrology: a review

In 2016, the Google-owned artificial intelligence (AI) company DeepMind shocked the world by defeating Lee Sedol four matches to one with its AlphaGo AI system, alerting the world to deep learning, a new breed of machine learning that promised to be smarter and more creative than before 1. Since then, we have witnessed its rapid progress and extensive applications in solving many tasks in computer vision, computational imaging, and computer-aided diagnosis with unprecedented performance. Meanwhile, tech giants Google, Facebook, Microsoft, Apple, and Amazon have ignited the “art” of data manipulation and developed easy-to-use, open-source deep learning frameworks. These deep learning frameworks allow us to build complex and large-scale deep learning models using a collection of pre-built and optimized components in a more clear, concise, and user-friendly way, without getting into too many details of underlying algorithms. Deep learning has left the halls of academia very quickly and is ready to reshape an array of companies across multiple industries.

Fig. 1 | The rise of deep learning. In 2016, the AlphaGo defeated Lee Sedol four matches to one, alerting the world to deep learning. Meanwhile, tech giants ignited the “art” of data manipulation and developed easy-to-use, open-source deep learning frameworks.

On the other hand, optical metrology is the science and technology of making measurements with use of light as standards or information carriers. Although optical metrology is a rapidly growing area, it is not a new discipline. The development of physical sciences has been driven from the very beginning by optical metrology techniques. In return, optical metrology has been revolutionized by the invention of laser, charged coupled device (CCD), and computer, developing into a broad and interdisciplinary field relating to diverse disciplines such as photomechanics, optical engineering, computer vision, and computational imaging. In light of the great success of deep learning in these related fields, researchers in optical metrology were unable to hold back their curiosities with regards to adopting this technology to further push the limits of optical metrology and provide new solutions in order to meet the upcoming challenges in the perpetual pursuit of higher accuracy, sensitivity, repeatability, efficiency, speed, and robustness. In this context, we incidentally become the “first to eat crab” research group (SCILab: www.scilaboratory.com) of introducing deep learning to optical metrology.

“The first to eat crab”

For many phase measuring optical metrology techniques, including optical interferometry, digital holography, electronic speckle pattern interferometry, Moiré profilometry, and fringe projection profilometry, the physical quantities to be measured (such as the surface shape, displacement, strain, roughness, defects, etc.) are directly or indirectly encoded in the phase information of the fringes formed by means of interference or projection. Consequently, phase demodulation, which analyzes the quasi-periodic fringe pattern for the wrapped phase extraction, is the most critical step because the measurement accuracy of these optical metrology techniques depends directly on the phase demodulation accuracy of recorded fringe patterns. How to extract the phase information with the highest accuracy, fastest speed, and full automation remains a research hotspot in the field of optical metrology.

Traditional fringe pattern analysis, or phase demodulation techniques can be broadly classified into two categories: spatial and temporal techniques. Temporal phase demodulation techniques (representatively, the phase-shifting technique 2) detect high-resolution pixel wise phase distribution from the temporal variation of fringe signals at the cost of time sequential data acquisitions. Spatial phase demodulation methods, such as Fourier transform (FT) 3 and windowed Fourier transform (WFT) 4, are capable of estimating the phase distribution from a single fringe pattern, but fringe discontinuities and rich details of testing surfaces prevent them from high-accuracy phase measurement of complex surfaces. In this context, the goal of our first attempt is to develop a deep-learning-based fringe pattern analysis technique that is capable of combining the single-frame strength of spatial phase demodulation techniques with the high measurement accuracy of temporal phase demodulation techniques.

Initially, we tried to accomplish this goal by designing a deep convolutional neural network (CNN) with an end-to-end architecture, which directly links the input fringe image to the output phase map. However, it has been found that such an end-to-end learning scheme had difficulties in reproducing abrupt 2π phase jumps in the wrapped phase map, making the training process fails to converge. After many twists and turns, we have finally devised a pragmatic and practical solution. Inspired by the physical model of conventional fringe analysis techniques, where the wrapped phase is calculated from the arctangent function, we attempted to predict the sine and cosine components of the fringe pattern from one input fringe image [Fig. 2a]. Encouragingly, such a strategy worked extremely well after appropriate network training, and can provide high-accuracy phase predictions close to those of the 12-step phase-shifting approach [Fig. 2b]. We further applied this learning-based fringe analysis technique to a high-speed fringe projection profilometry system, achieving an unprecedented 3D imaging frame rate up to 20,000 Hz [Fig. 2c]. This work was published in Advanced Photonics as a cover paper in 2019 5, which now has become the most cited paper of Advanced Photonics since its inception.

“A logical hierarchy”

At the beginning of 2020, the sudden COVID19 raged and spread around the world. The New Year, an originally lively traditional Chinese festival, became unusually silent. During the long hours of leisure, I have noticed that researchers in optical metrology started actively participating in the explosively growing field of deep learning, as evidenced by the ever-increasing number of publications and exponentially growing citations to our earlier research. Within a few short years, deep-learning-

based techniques have been gaining increasing attention and demonstrating promising performance in various optical metrology tasks, such as phase demodulation, phase unwrapping, system calibration, and error compensation. However, those research works are scattered rather than systematic. “Whether machine learning will be the driving force in optical metrology not only provides superior solutions to the growing new challenges but also tolerates imperfect measurement conditions with least efforts?” The answer to this key question deserves deeper thought and exploration. “Under these circumstances, a comprehensive review that covers principles, implementations, advantages, applications, and challenges in utilizing deep learning for optical metrology tasks will be extremely useful.” The idea for a review article entitled “Deep learning in optical metrology: a review” was therefore born.

On Mar 22, 2020, I said to my Ph.D. student Jiaming Qian, the primary coauthor of this review article: “Let us together take an extensive and thorough examination of previously published relevant literature and create a compelling synthesis of gathered references.” Hundreds of deep learning papers on classical interferometry, digital holography, fringe projection profilometry, etc., were then surveyed and categorized [Fig. 3]. In May, under the shadow of the epidemic, some senior Ph.D. students were allowed to return to the university on the premise of ensuring safety, including Jiaming. I enjoy using figures and tables to summarize research progress and suggest future research trajectories. So I drew a very sketchy optical metrology vein diagram for Jiaming: “Optical metrology covers a wide range of methods and applications today. It would be impractical for this review to discuss all the relevant technologies and trends.” My advice to Jiaming is to accept the fact that a review is different from a textbook: it should be more focused, and it’s OK to skip some topics so that it does not distract the readers.

With this sketch in hand, Jiaming began to meticulously “carve” and “polish” it [Fig. 4]. Little did anyone predict that 19 months later a sketch would expand into a 54-page paper with double-column layout. Indeed, when it comes to optical metrology, ones subconsciously associate them with ‘fringe’ (phase measuring metrology) and ‘speckle’ (speckle metrology). We therefore restrict our focus to phase/correlation measurement techniques, such as interferometry, holography, fringe projection, and digital image correlation (DIC). Image processing plays an essential role in optical metrology, which is very similar to those of computer vision and computational imaging for the purpose of converting the observed measurements (intensity image in most cases) into the object quantities taking into account the physical model describing the image formation process. In most cases, image processing in optical metrology is not a one-step procedure, but a logical hierarchy consisting of three main steps, pre-processing, analysis and post-processing. Such a logical hierarchy provides a systematic framework throughout this review to classify and summarize the various optical metrology techniques and tasks that are otherwise fragmented. It also helps to decide what should and should not be included in the review.

“A panoramic comparative picture”

After this, we proceed with the writing. We need to fill each layer of the hierarchical steps, which is essentially writing a mini-review of various types of algorithms distributed in different layers. I remind my students to imagine themselves as ‘artists of science’ and encourage them to develop how they write and present information. “Adding more words isn’t always the best way, we have to get something concisely from a broad reading.”

Because of the significant changes that deep learning brings to the concept of optical metrology technology, almost all elementary tasks of digital image processing in optical metrology have been reformed by deep learning. This encouraged us to further summarize these existing researches leveraging deep learning in optical metrology according to a similar logical architecture [Fig. 5]. We went through multiple iterations to make sure that we had scanned the literature sufficiently and provided a clear, concise, appropriate, and informative review of conventional and new “learning-based” optical metrology techniques. Gradually, a clear and beautiful view unfolded before our eyes: the new deep learning-enabled optical metrology algorithms [Fig. 5] and their traditional counterparts [Fig. 4] echo each other, providing us with a panoramic comparative picture.

As a smooth transition between the old and the new, we gave a brief introduction to deep learning and summarized its threefold advantages from extensive literature: from “physics-model-driven” to “data-driven”, from “divide-and-conquer” to “end-to-end learning”, and from “solving ill-posed inverse problems” to “learning pseudo-inverse mapping”. In general, deep learning is particularly advantageous for many problems in optical metrology whose physical models are complicated and acquired information is limited. Strong empirical and experimental evidence suggests that using problem-specific deep learning models outperforms conventional knowledge or physical model-based approaches.

“The other side of the coin”

It is not enough for a review to be a summary of historical growth in the literature; it is also expected to provide a discussion about controversial issues in this field. In spite of the promising — in some cases impressive — results that have been documented in the literature, on the other side of the coin, significant challenges do remain in this area. In collaboration with my colleagues, Assoc. Prof. Shijie Feng and Jing Han from Nanjing University of Science and Technology, and exchange student Pengfei Fan from Queen Mary University of London, the critical challenging issues of applying deep learning to optical metrology were gathered and discussed:

• For model training, we need to acquire large amounts of experimental data with labels, which, even if available, is laborious and requires professional experts [Fig. 6].

Fig. 6 | The challenge of deep learning in optical metrology—the high cost of obtaining and labeling training data. Taking fringe projection profilometry as an example, to collect high-quality training data, the traditional multi-frequency temporal method is used, which causes a large number of images to be projected for each set of training data. However, hardware errors, ambient light interference, calibration errors, etc. in actual operation make it difficult to obtain ideal ground truth through traditional algorithms

• So far, there has been no theoretical groundwork that could clearly explain the mechanisms to optimize network structure for a specific task or profoundly comprehend why a particular network structure is effective in a given task or not [Fig. 7].

Fig. 7 | The challenge of deep learning in optical metrology—empiricism in model design and algorithms selection. Taking the phase extraction in fringe projection profilometry as an example, the same task can be implemented by different neural network models with different strategies: the fringe image can be mapped directly to the corresponding phase map by DNN1; we can also output the numerator and denominator of the arctangent function used to calculate the phase information from a fringe image and a uniform by DNN2; with more powerful DNN, we can predict from a fringe image the numerator and denominator

• Generally, deep learning architectures used in optical metrology are highly specialized to a specific domain, and they should be implemented with extreme care and caution when solving issues that do not pertain to the same domain.

• Deep learning approaches have often been regarded as ‘black boxes’, and in optical metrology, accountability is essential and can cause severe consequences.

• Since the information cannot be “born out of nothing”, deep learning cannot always produce a provably correct solution. The success of deep learning depends on the “common” features learned and extracted from the training samples, which may lead to unsatisfactory results when facing “rare samples”.

Listed above are among the most critical issues for optical metrology applications where the accuracy, reliability, repeatability, and traceability of measurement results are primary considerations. After identifying the research gaps, we hope the review paper should leave the reader with explicit opinions on its future trajectory. After another round of brainstorming, we made the following suggestions for potential new research areas to explore in the future.

• Leveraging more emerging technologies of deep learning methods to optical metrology could promote and accelerate the recognition and acceptance of deep learning in more application areas.

• Combining Bayesian statistics with deep neuron networks to obtain quantitative uncertainty estimates allows us to assess when the network yields unreliable predictions (see our recent Optica paper on this point) 7.

• A synergy of the physics-based models that describe the a priori knowledge of the image formation and data-driven models that learn a regularization from the experimental data can bring our domain expertise into deep learning to provide more physically plausible solutions to specific optical metrology problems.

We believe the above-mentioned aspects can provide inspiration for future scopes and continue to attract the interest of deep learning research in the optical metrology community in the years to come. Finally, we would also like to remind readers that the selection between deep learning and traditional algorithms should be considered rationally, given the “no free lunch theorem”. For several problems where traditional methods based on physics models, if implemented properly, can deliver straightforward and more than satisfying solutions, there is no need to use deep learning.

“Revise, revise, revise”

In Jan 2021, we finished the first draft and decided to submit it to Light: Science & Applications, which publishes original articles and reviews of high quality, high interest, and far-reaching impact. My Ph.D. student, Yixuan Li, double-checked the typesetting, grammar, and references according to the journal’s stylistic and formatting guidelines. After that, I consulted Prof. Kemao Qian at Nanyang Technological University (NTU), Singapore, an expert with more than 20 years of experience in optical metrology, to review our draft. Prof. Qian was my (unofficial) Ph.D. advisor when I was a visiting student at NTU from Sep 2012 and Feb 2014. My intuition told me that getting his perspective would be very helpful in enhancing the quality of this review.

After assessing the first draft, Prof. Qian offered three constructive criticisms. (1) The section “brief introduction to deep learning” only introduced the history of deep learning and artificial neural network, and had little relevance to optical metrology. Instead, readers should be interested in learning more about how deep learning can be used in optical metrology from this section. (2) The transition between sections of traditional optical metrology algorithms and their deep learning-enabled counterparts was sudden and there was insufficient evidence as to why deep learning should be used in optical metrology. (3) As a review, its main purpose is to help other researchers enter this field more easily by collecting and summarizing, synthesizing, and analyzing existing research. “Will deep learning be the future of optical metrology?” It is very difficult to draw a conclusion at the current stage, as the place of deep learning in optical metrology is not yet clear. So instead of giving a clear answer to this question, we try out best to paint a full and informative picture for our readers.

“When you draft, you write for yourself. When you revise, you clarify for your readers.” Prof. Qian’s advice epitomized this motto. With these criticisms in hand, the only thing we could do was revise, revise, revise! “We must strive for excellence.” I encouraged Jiaming. It had now become apparent that spending nearly half a year revising is a wise choice, because this made this review not only more in-depth in content but also more logically developed, from beginning to end.

• For the first comment, we added more introduction to the fundamentals of deep learning, including the basic principles of neural networks, network structures, and training algorithms. We focused on the dominant network architecture for image- and vision-related tasks—convolutional neural networks (CNNs), and then discussed in detail the variants of classical CNN architecture—DNNs with a fully convolutional architecture, that shares characteristics with image processing algorithms in optical metrology (transforming the content of arbitrary-sized inputs into pixel-level outputs). By applying different types of training datasets, they can be trained for accomplishing different types of image processing tasks that we encountered in optical metrology. This provides an alternative approach to process images such that the produced results resemble or even outperform conventional image processing operators or their combinations. There are also many other potential desirable factors for such a substitution, e.g., accuracy, speed, generality, and simplicity. All these factors were crucial to enable the fast rise of deep learning in the field of optical metrology.

  
  

  

Fig. 8 | Inverse problems in computer vision. In computer vision, such as image deblurring, the resulting inverse problem is ill-posed since the forward measurement operator mapping from the parameter space to the image space is usually poorly conditioned. The classical approach is to impose certain prior assumptions (smoothing) about the solution that helps in regularizing its retrieval

• For the second comment, we tried to explain the reason for the transition from the perspective of solving inverse problems. In optical metrology, we have to conclude in general from the effect (i.e., the intensity at the pixel) to its cause (i.e., shape, displacement, deformation, or stress of the surface). Such information recovery process is similar to those of computer vision and computational imaging, presenting as an inverse problem that is often ill-posed. Tremendous progress has been achieved in terms of accurate mathematical modeling, regularization techniques, numerical methods, and their efficient implementations [Fig. 8]. For optical metrology, the situation becomes quite different due to the fact that the optical measurements are frequently carried out in a highly controlled environment. Instead of explicitly interpreting optical metrology tasks from the perspective of solving inverse problems, we prefer to reformulate the original ill-posed problem into a well-posed and adequately stable one by actively controlling the image acquisition process [Fig.   9]. However, for many challenging applications, harsh operating conditions may make such active strategies a luxurious or even unreasonable request. Under such conditions, deep learning is particularly advantageous for solving those optical metrology problems because the active strategies are shifted from the actual measurement stage to the preparation (network training) stage, and the “reconstruction algorithm” can be directly learned from the experimental data [Fig.   10]. If the training data is collected under the environment that reproduces the real experimental conditions, and the amount of data is sufficient, the trained model should reflect the reality more precisely and comprehensively, and is expected to produce better reconstruction results than conventional physics-model-based approaches.

Fig. 9 | Inverse problems in optical metrology. Optical metrology uses an “active” approach to transform the ill-posed inverse problem into a well-posed estimation or regression problem: by acquiring additional phase-shifted patterns of different frequencies, absolute phase can be easily determined by multi-frequency phase-shifting and temporal phase unwrapping methods

Fig. 10 | Deep learning-based optical metrology as a constraint optimization problem. a In deep learning-based optical metrology, a set of true object parameters and the corresponding raw measured data are created at the training stage, and their mapping relation (learn a reconstruction algorithm) is established by training a deep neural network with all network parameters   (neural network weights) learned from the dataset. b The principle of obtaining the dataset by real experiments or simulations with the knowledge of the forward model (left) and the obtained dataset (right)

Many great writers have commented on the importance of revision. William Zinsser said, “Revision is the essence of good writing: it’s where the game is won or lost.” Stephen King said, “You need to revise for length. The formula. Second draft = first draft - 10%.” I would have thought revision does not necessarily mean rewriting the whole paper, and a 10% revision was a lot. But after going through countless iterations, more than 1/3 content of the draft had been refreshed. This considerable improvement was mainly attributed to the careful and insightful guidance of Prof. Qian, who was far away in Singapore, devoting significant efforts to revise the manuscript with us round by round, through countless hours of the video conference [Fig. 11].

Fig. 11 | Professor Qian was revising the manuscript with Professor Zuo through the video conference.

“Be the ‘go to’-reference”

In July 2021, we finalized the manuscript, which was carefully read, checked and approved by all co-authors. We submitted it to Light: Science & Applications as planned, along with a cover letter to the editor. The peer-review report came back after two months, and, encouragingly, all three reviewers gave very positive comments on our manuscript. The overall assessments of the three reviewers are provided as follows:

• Reviewer 1: “This is a pretty comprehensive review paper for deep learning in optical metrology. They start from introducing conventional methods in optical metrology and measurement processing, and then go into tutorial of deep learning and how deep learning is applied. I find that this paper is timely and can be considered for publication pending that the following comments are addressed……”

• Reviewer 2: “The review article on Deep Learning in Optical Metrology is an excellent manuscript which is well written……The authors have given a very good introduction to various optical metrological methods such as interferometry, holography, fringe projection, and DIC. The development of these methods through the past, their basics have been well explained. The figures presented have been well organized giving the reader a good comparative picture. ……The manuscript gives a comprehensive review of optical metrology techniques and how deep learning can be tailored. The presentation details are well handled. The manuscript deserves publication as such. ”

• Reviewer 3: “The authors present us with a review paper in the field of deep learning applied to optical metrology. It is a very long manuscript with more than 40 pages of text. However, it is well written and a pleasure to read. As someone who is working in optical metrology for more than 20 years, it was a very good introduction into the basic concepts of deep learning in this field, and I learned a lot……”

The main constructive revision comments came from Reviewer 3, who suggested to give a simple but very detailed example on how to apply deep learning to optical metrology:

Reviewer 3: “The manuscript is clearly a beautiful review paper. BUT, with some small changes it can even become so much more. I would really encourage the authors to give a simple but very detailed example on how to apply deep learning to optical metrology (including the math, the algorithmic implementation, etc.). This could be for example denoising or signal-reconstruction from corrupted data. This could evolve the paper into one of the fundamental “go to”-references if it comes to optical metrology and deep learning, because it is not only a good overview but also a basic tutorial. You could do this at the expense of shortening the first part (see above), because nobody really wants to read 5 to 10 pages of textbook knowledge about phase shifting, phase unwrapping, etc., but everybody wants to LEARN how to apply these new data-driven approaches.”

Finally, the reviewer concluded:

“I believe that this manuscript has big potential to become one of the “go to”-references in deep learning for optical metrology. It is well written, comes at the right time and includes numerous examples. However, I would strongly encourage the authors to consider the above comments and I would like to re-review it.”

I deeply appreciated that the reviewers were here to help our paper succeed and by following their advice, finally we would emerge with a stronger version that will hopefully end up becoming the definitive “go-to” guide on this topic. We were inspired to further include a tutorial of applying deep learning to optical metrology in the Supplementary Information, taking phase demodulation from a single fringe pattern as an example. In addition, we published the source codes and the corresponding datasets for this example. We demonstrate that a well-trained deep neural network can accomplish the phase demodulation task in an accurate and efficient manner, using only a single fringe pattern. Thus, it is capable of combining the single-frame strength of the spatial phase demodulation methods with the high measurement accuracy of the temporal phase demodulation methods. This turned out to be a considerable addition because it made the review more comprehensive and instructive, potentially increasing its readership. It should be noted that it took approximately four months to complete the processes of peer review, revision, and publication. During this period, many new papers and even competing reviews were published. To provide the most up-to-date review, we had to stay abreast of the literature by using Google Scholar, which alerted me daily updates of relevant literature based on keywords.

“Predict Engage the future”

Finally, let us return to the third point raised by Prof. Qian “Is predicting the future futile or necessary?”    Undoubtedly, deep learning is currently prompting increasing interests and leading to a paradigm shift from physics- and knowledge-based modeling to data-driven learning in optical metrology. Strong empirical and experimental evidence suggests that using problem-specific deep learning models outperforms conventional knowledge or physical model-based approaches, especially for many optical metrology tasks whose physical models are complicated and acquired information is limited.

It has to be admitted that deep learning is still at the early stage of development for its applications in optical metrology. Many researchers are still skeptical and maintain a wait-and-see attitude towards its applications involving industrial inspection and medical care, etc. Shall we accept deep learning as the key problem-solving tool? Or should we reject such a black-box solution? These are controversial issues in the optical metrology community today. Looking on the bright side, it has promoted an exciting trend and fostered expectations of the transformative potential it may bring to the optical metrology society. However, we should not overestimate the power of deep learning by considering it as a silver bullet for every challenge encountered in the future development of optical metrology. In practice, we should assess whether the large amount of data and computational resources required to use deep learning for a particular task is worthwhile, especially when other conventional algorithms may yield comparable performance with lower complexity and higher interpretability. “Will deep learning replace the role of traditional technologies within the field of optical metrology for the years to come? “It is clear no one can predict the future, but we can engage it. If you are still an ‘outsider’ or new to this field. I encourage you to try it out! It is easy, and often works!

For more information, please refer to this recent publication: Zuo, C., Qian, J., Feng, S. et al. Deep learning in optical metrology: a review. Light Sci Appl 11, 39 (2022).

DOI: https://doi.org/10.1038/s41377-022-00714-x

Reference

1. Chen, J. X. The evolution of computing: AlphaGo. Comput. Sci. Eng. 18, 4–7 (2016).

2. Zuo, C., Feng, S., Huang, L., Tao, T., Yin, W., & Chen, Q., A.Phase shifting algorithms for fringe projection profilometry: A review. Opt. Lasers Eng. 109, 23–59 (2018).

3. Takeda, M., Ina, H. & Kobayashi, S. Fourier-transform method of fringe-pattern analysis for computer-based topography and interferometry. J Opt Soc Am A 72, 156–160 (1982).

4. Kemao, Q. Windowed Fourier transform for fringe pattern analysis. Appl. Opt. 43, 2695–2702 (2004).

5. Feng, S., Chen, Q., Gu, G., Tao., T., Zhang., L., Hu, Y., Yin, W., Zuo, C., Fringe pattern analysis using deep learning. Adv. Photon. 1, 025001 (2019).

6. Feng, S., Zuo, C., Yin, W., Gu, G. & Chen, Q. Micro deep learning profilometry for high-speed 3D surface imaging. Opt. Lasers Eng. 121, 416–427 (2019).

7. Feng, S., Zuo, C., Hu, Y., Li, Y. & Chen, Q. Deep-learning-based fringe-pattern analysis with uncertainty estimation. Optica 8, 1507–1510 (2021).