Smart Computational Imaging (SCI) Lab

智能计算成像实验室

Real-time 3-D Imaging and System

We are dedicated to innovate high-resolution, high speed , real-time 3-D imaging/sensing methods and systems for numerous applications such as human-to-computer interaction, computer graphics, healthcare, and the manufacturing industry. We have developed a real-time three-dimensional (3D) imaging system based on digital fringe projection, both hardware and software. The system achieves 120 fps real-time data acquisition, processing and visualization. This novel system employs high speed fringe projection [by modified a commercial DLP projector with self-developed Field-Programmable Gate Array (FPGA) circuit], pattern design, direct phase unwrapping (novel 4-pattern strategy to realize phase shifting & temporal phase unwrapping), gamma distortion correction (pre-coding the distorted pattern in the on-chip memory of the FPGA), noise reduction, motion artifacts suppression, filter design and other techniques (implemented in C++ with Visual Studio 2008).  By combining an infrared camera (self-developed 640×512 VOx Microbolometer) with this system, real-time three-dimensional thermal imaging can be realized.

However, traditional fringe projection profilometers are usually designed to measure static object shapes under non-time-critical measurement conditions, their measurement of moving objects is limited. Besides, it is difficult to retrieve the absolute phase for spatially isolated surfaces simultaneously and rapidly since in the phase unwrapping, fringe orders will be ambiguous ( the so-called 2π phase discontinuities problem), making the depth difference between spatially isolated surfaces indiscernible. One typical  phase unwrapping method called mult-frequency temporal phase unwrapping (MFTPU).  By varying fringe frequency on an object several times, the MFTPU algorithm can be used to determine the fringe order number and, based on this, resolve 2π phase discontinuities. However, the MFTPU generally requires quite a lot of fringes (at least 6 for two frequency TPU) that inevitably prolongs the acquisition time.

Another bottleneck for realizing high-speed fringe projection is the fringe projection speed. A typical DLP projector contains two important components: a DMD chip and a color wheel. The DMD chip modulates the projected light after it is reflected off an array of tiny mirrors. Any displayed intensity is made up of pulses of light created by these mirrors switching on and off. The DLP video processor system separates color images into their RGB components for display, employing a rotating color wheel synchronized to RGB image separations displayed by the DMD. The color wheel spins at 120 Hz (in some older types of DLP projector, the color wheel may operate at 60 or 80 Hz) and is divided into multiple sectors: the primary colors: red, green, and blue. The synchronization between the DMD and the color-wheel is controlled by a 120 Hz feedback trigger signal generated by a photodiode located inside the color-wheel carrier. Once the DMD is triggered, it will project the RGB channels of the input color image sequentially by binary temporal pulse-width-modulation.  To triple the speed of the dlp projector, we should first remove the color filter of the projector and then create a sequence of color encoded pattern images, sent them sequentially and repeatedly at 120 Hz (by using proper hardware). When the projector receives this pattern sequence, the gray-scale patterns can be decoded by the DLP video processor system and then projected them in a correct order (360Hz). We are trying to hack the the commercial off-the-shelf DLP projector to realize high-speed 3D imaging system at a very low cost.

3-D measurement using speckle-embedded fringe

This paper presents a novel two-frame fringe projection technique for real-time, accurate, and unambiguous three-dimensional (3-D) measurement. One of the frames is a digital speckle pattern, and the other one is a composite image which is generated by fusing that speckle image with sinusoidal fringes. The contained sinusoidal component is used to obtain a wrapped phase map by Fourier transform profilometry, and the speckle image helps determine the fringe order for phase unwrapping. Compared with traditional methods, the proposed pattern scheme enables measurements of discontinuous surfaces with only two frames, greatly reducing the number of required patterns and thus reducing the sensitivity to movements. This merit makes the method very suitable for inspecting dynamic scenes. Moreover, it shows close performance in measurement accuracy compared with the phase shifting method from our experiments. To process data in real time, a Compute Unified Device Architecture–enabled graphics processing unit is adopted to accelerate some time-consuming computations. With our system, measurements can be performed at 21 frames per second with a resolution of 307,000 points per frame.

References:

• S. Feng, Q. Chen, and C. Zuo, "Graphics processing unit–assisted real-time three-dimensional measurement using speckle-embedded fringe,"Applied optics 54(22), 6865-6873 (2015).

Real-time 3-D shape measurement with multi-view system

In recent years, fringe projection has become an established and essential method for dynamic three-dimensional (3-D) shape measurement in dierent fields such as online inspection and real-time quality control. Numerous high-speed 3-D shape measurement methods have been developed by either employing high speed hardware, minimizing the number of pattern projection, or both. However, dynamic 3-D shape measurement of arbitrarily-shaped objects with full sensor resolution without the necessity of additional pattern projections is still a big challenge. In this work, we introduce a high-speed 3-D shape measurement technique based on composite phase-shifting fringes and a multi-view system. The geometry constraint is adopted to search the corresponding points independently without additional images. Meanwhile, by analysing the 3-D position and the main wrapped phase of the corresponding point, pairs with an incorrect 3-D position or a considerable phase dierence are eectively rejected. All of the qualified corresponding points are then corrected, and the unique one as well as the related period order is selected through the embedded triangular wave. Finally, considering that some points can only be captured by one of the cameras due to the occlusions, these points may have dierent fringe orders in the two views, so a left-right consistency check is employed to eliminate those erroneous period orders in this case. Several experiments on both static and dynamic scenes are performed, verifying that our method can achieve a speed of 120 frames per second (fps) with 25-period fringe patterns for fast, dense, and accurate 3-D measurement.

References:

• T. Tao, Q. Chen, J. Da, S. Feng, Y. Hu, and C. Zuo, "Real-time 3-D shape measurement with composite phase-shifting fringes and multi-view system,"Optics Express 24, 20253-20269 (2016). [code available]

3D infrared imaging using fringe projection

Infrared thermography determines the surface temperature of an object or human body. It is a promising imaging technology for medical and biological observations due to its contactless and completely non-invasive properties. However, traditional two dimensional (2-D) infrared thermography cannot retain the spatial information, and thus provides only qualitative diagnosis information. A novel real-time three dimensional (3-D) infrared imaging system which takes full advantages of high-speed, high-quality, high-sensitivity, and low-cost in 3D thermograph is presented. We demonstrate the real time 3-D thermal imaging at the speed of 24 frames per second (fps), with resolution of 640 × 480 points. Experimental results demonstrate quantitatively measurement of temperature distribution of 3-D surfaces in real-time is realized with this system.

References:

• C. Zuo, Q. Chen, S. Feng, G.Gu, and A. Asundi, "Real-time three-dimensional infrared imaging using fringe projection prof ilometry," Chinese Optics Letters, 1671-7694 (2013).

A comparative review for temporal phase unwrapping

In fringe projection profilometry (FPP), temporal phase unwrapping is an essential procedure to recover an unambiguous absolute phase even in the presence of large discontinuities or spatially isolated sur-faces. So far, the reare typically three groups of temporal phase unwrapping algorithms proposed in the literature: multi-frequency (hierarchical) approach, multi-wavelength (heterodyne) approach, and number theoretical approach. In this paper, the three methods are investigated and compared in detail by analytical, numerical, and experimental means. The basic principles and recent developments of the three kind of algorithms are firstly reviewed. Then, the reliability of different phase unwrapping algorithms is compared based on arigorous stochastic noise model. Furthermore, this noise model is used to predict the optimum fringe period for each unwrapping approach, which is a key factor governing the phase measurement accuracy in FPP. Simulations and experimental results verified the correctness and validity of the proposed noise model as well as the prediction scheme. The results show that the multi-frequency temporal phase unwrapping provides the best unwrapping reliability, while the multi-wavelength approach is the most susceptible to noise-induced unwrapping errors.

References:

• C. Zuo, L. Huang, M. Zhang, and A. Asundi, "Temporal phase unwrapping algorithms for fringeprojection profilometry: A comparative review," Optics and Lasers in Engineering 85, 84-103 (2016).

Calibration considering camera lens distortion

Optical three-dimensional (3-D) profilometry is gaining increasing attention for its simplicity, flexibility, high accuracy,and non-contact nature. Recent advances in imaging sensors and digital projection technology further its progress in high-speed, real-time applications, enabling 3-D shapes reconstruction of moving objects and dynamics cenes. However, the camera lens is never perfect and the lens distortion does influence the accuracy of the measurement result, which is often overlooked in the existing real-time 3-D shape measurement systems.To this end, here we present an ovel high-speed real-time3-D coordinates measuring technique based on fringe projection with the consideration of the camera lens distortion. A pixel mapping relation between a distorted image and a corrected one is pre-determined and stored in computer memory for real-time fringe correction. The out-of-plane height is obtained firstly and the acquisition for the two corresponding in-plane coordinates follows on the basis of the solved height. Besides, a method of look up table (LUT) is introduced as well for fast data processing.

Our experimental results reveal that the measurement error of the in-plane coordinates has been reduced by one order of magnitude and the accuracy of the out-plane coordinate been tripled after the distortions being eliminated. Moreover, owing to the generated LUTs, a 3-D reconstruction speed of 92.34 frames per second can be achieved.

References:

• S. Feng, Q. Chen, C. Zuo, J. Sun, and S. Yu, "High-speed real-time 3-D coordinates measurement based on fringe projection profilometry considering camera lens distortion," Optics Communications 329, 44-56 (2014).

Automatic identification and removal of outliers

Phase-shifting profilometry combining with the two-frequency temporal phase unwrapping is widely used for high-speed, real-time acquisition of three-dimensional (3-D) shapes. However, when the object is not motionless during the acquisition process, some unreliable results may emerge, especially around the contours of the measured object. The main reason for this is that the same point in the projected pattern sequence can map to different points within the camera images resulting from depth changes over time. We present a novel approach for identifying those invalid pixels affected by such an error. By carefully examining the captured fringe pattern, comparing two modulation maps, utilizing the phase relationship between two neighboring pixels, and employing a Gaussian filter to detect the protruding points, the bad measurement pixels can be detected and filtered out effectively. The whole procedure is of low computational complexity because of the introduced lookup table-based fast data processing method.

References:

• S. Feng, Q. Chen, C. Zuo, R. Li, G. Shen, and F. Feng, "Automatic identification and removal of outliers for high speed fringe projection profilometry," Optical Engineering 52(1), 013605-013605 (2013).

  
  

General solution for the measurement of high dynamic range

This paper presents a general solution for realizing high dynamic range three-dimensional (3-D) shape measurement based on fringe projection. Three concrete techniques are involved in the solution for measuring object with large range of reflectivity (LRR) or one with shiny specular surface. For the first technique, the measured surface reflectivities are sub-divided into several groups based on its histogram distribution, then the optimal exposure time for each group can be predicted adaptively so that the bright as well as dark areas on the measured surface are able to be handled without any compromise. Phase-shifted images are then captured at the calculated exposure times and a composite phase-shifted image is generated by extracting the optimally exposed pixels in the raw fringes images.For the second technique, it is proposed by introducing two orthogonal polarizers which are placed separately in front of the camera and projector into the first technique and the third one is developed by combining the second technique with the strategy of properly altering the angle between the transmission axes of the two polarizers.

References:

• S. Feng, Y. Zhang, Q. Chen, C. Zuo, R. Li, and G. Shen, "General solution for high dynamic range three-dimensional shape measurement using the fringe projection technique," Optics and Lasers in Engineering 59, 56-71(2014).

  
  

Dynamic measurement system for shiny surfaces

This paper presents a novel fringe projection technique for fast three-dimensional (3-D) shape measurements of moving highly reflective objects. By combining the standard three-step phase-shifting fringe patterns with a digital speckle image, dynamic 3-D reconstructions of shiny surfaces can be efficiently achieved with only four projected patterns. The phase measurement is performed by three-step phase-shifting algorithm as it uses the theoretical minimum number of fringe patterns for phase-shifting profilometry. To avoid the camera saturation, a dual-camera fringe projection system is built to measure shiny objects from two different directions. The erroneous phase obtained from a saturated pixel is corrected by the phase of its corresponding pixel in the other view which is free from the saturation problem. To achieve high measurement accuracy, the corresponding high light intensity areas in cameras are found by sub-pixel matches of the speckle pattern in either view. Benefited from the trifocal tensor constraint, the corresponding points in the two wrapped phase maps can be directly established, and thus, the difficulties in determining the correct fringe order for the discontinuous or isolated surfaces can be effectively bypassed. Experimental results indicate that the proposed method is able to successfully measure highly reflective surfaces for both stationary and dynamic scenes.

References:

• S. Feng, Q. Chen, C. Zuo, and A. Asundi, "Fast three-dimensional measurements for dynamic scenes with shiny surfaces," Optics communications 382, 18-27(2017).