There are hundreds of welded studs in a car.The posture of a welded stud determines the quality of the body assembly,thus affecting the safety of cars.It is crucial to detect the posture of the welded studs.Considerin...There are hundreds of welded studs in a car.The posture of a welded stud determines the quality of the body assembly,thus affecting the safety of cars.It is crucial to detect the posture of the welded studs.Considering the lack of accurate method in detecting the position of welded studs,this paper aims to detect the weld stud’s pose based on photometric stereo and neural network.Firstly,a machine vision-based stud dataset collection system is built to achieve the stud dataset labelling automatically.Secondly,photometric stereo algorithm is applied to estimate the stud normal map which as input is fed to neural network.Finally,we improve a lightweight YOLOv4 neural network which is applied to achieve the detection of stud position,thus overcoming the shortcomings of traditional testing methods.The research and experimental results show that the stud pose detection system designed achieves rapid detection and high accuracy positioning of the stud.This research provides the foundation combining the photometric stereo and deep learning for object detection in industrial production.展开更多
Colonoscopy screening for the detection and removal of colonic adenomas is central to efforts to reduce the morbidity and mortality of colorectal cancer.However,up to a third of adenomas may be missed at colonoscopy,a...Colonoscopy screening for the detection and removal of colonic adenomas is central to efforts to reduce the morbidity and mortality of colorectal cancer.However,up to a third of adenomas may be missed at colonoscopy,and the majority of post-colonoscopy colorectal cancers are thought to arise from these.Adenomas have three-dimensional surface topographic features that differentiate them from adjacent normal mucosa.However,these topographic features are not enhanced by white light colonoscopy,and the endoscopist must infer these from two-dimensional cues.This may contribute to the number of missed lesions.A variety of optical imaging technologies have been developed commercially to enhance surface topography.However,existing techniques enhance surface topography indirectly,and in two dimensions,and the evidence does not wholly support their use in routine clinical practice.In this narrative review,co-authored by gastroenterologists and engineers,we summarise the evidence for the impact of established optical imaging technologies on adenoma detection rate,and review the development of photometric stereo(PS)for colonoscopy.PS is a machine vision technique able to capture a dense array of surface normals to render three-dimensional reconstructions of surface topography.This imaging technique has several potential clinical applications in colonoscopy,including adenoma detection,polyp classification,and facilitating polypectomy,an inherently three-dimensional task.However,the development of PS for colonoscopy is at an early stage.We consider the progress that has been made with PS to date and identify the obstacles that need to be overcome prior to clinical application.展开更多
Background A photometric stereo method aims to recover the surface normal of a 3D object observed under varying light directions.It is an ill-defined problem because the general reflectance properties of the surface a...Background A photometric stereo method aims to recover the surface normal of a 3D object observed under varying light directions.It is an ill-defined problem because the general reflectance properties of the surface are unknown.Methods This paper reviews existing data-driven methods,with a focus on their technical insights into the photometric stereo problem.We divide these methods into two categories,per-pixel and all-pixel,according to how they process an image.We discuss the differences and relationships between these methods from the perspective of inputs,networks,and data,which are key factors in designing a deep learning approach.Results We demonstrate the performance of the models using a popular benchmark dataset.Conclusions Data-driven photometric stereo methods have shown that they possess a superior performance advantage over traditional methods.However,these methods suffer from various limitations,such as limited generalization capability.Finally,this study suggests directions for future research.展开更多
Non-uniformity of light sources is one of the inevitable error factors causing poor shape recoveryaccuracy of photometric stereo methods under close-range lighting with quasi point lights. Semi-calibrated photometrics...Non-uniformity of light sources is one of the inevitable error factors causing poor shape recoveryaccuracy of photometric stereo methods under close-range lighting with quasi point lights. Semi-calibrated photometricstereo methods are required to avoid repeated, tedious and impractical photometric calibration. In thispaper, two simple, concise but effective mesh-based semi-calibrated photometric stereo methods are proposed.The proposed methods extend the traditional mesh-based photometric stereo methods and further allow joint andaccurate estimation of normals and non-uniform light intensities by alternatively updating normals, depth mapsand intensities. Extensive experiments are conducted to validate the effectiveness and robustness of the proposedalgorithms. Even under extremely severe non-uniform lighting, the proposed methods can still suppress the errorand improve the shape recovery accuracy by up to 65.6% in real-world experiments.展开更多
Photometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination.Traditional methods normally adopt simplified reflectance...Photometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination.Traditional methods normally adopt simplified reflectance models to make the surface orientation computable.However,the real reflectances of surfaces greatly limit applicability of such methods to real-world objects.While deep neural networks have been employed to handle non-Lambertian surfaces,these methods are subject to blurring and errors,especially in high-frequency regions(such as crinkles and edges),caused by spectral bias:neural networks favor low-frequency representations so exhibit a bias towards smooth functions.In this paper,therefore,we propose a self-learning conditional network with multiscale features for photometric stereo,avoiding blurred reconstruction in such regions.Our explorations include:(i)a multi-scale feature fusion architecture,which keeps high-resolution representations and deep feature extraction,simultaneously,and(ii)an improved gradient-motivated conditionally parameterized convolution(GM-CondConv)in our photometric stereo network,with different combinations of convolution kernels for varying surfaces.Extensive experiments on public benchmark datasets show that our calibrated photometric stereo method outperforms the state-of-the-art.展开更多
Fabric pilling evaluation has been considered as an essential element for textile quality inspection. Traditional manual method is still based on human eyes and brain, which is subjective with low efficiency. This pap...Fabric pilling evaluation has been considered as an essential element for textile quality inspection. Traditional manual method is still based on human eyes and brain, which is subjective with low efficiency. This paper proposes an objective evaluation method based on semi-calibrated near-light Photometric Stereo(PS). Fabric images are digitalized by self-developed image acquisition system. The 3D depth information of each point could be obtained by PS algorithm and then mapped to 2D grayscale image. After that, the non-textured image could be filtered by using the Gaussian low-pass filter. The pilling segmentation is conducted by using global iterative threshold segmentation method,and then K-Nearest Neighbor(KNN) is finally selected as a tool for the grade classification of fabric pilling. Our experimental results show that the proposed evaluation system could achieve excellent judging performance for the objective pilling evaluation.展开更多
Under the perspective projection assumption,non-Lambertian photometric stereo is a highly non-linear problem.In this study,we present an optimized framework for reconstructing the surface normal and depth with non-Lam...Under the perspective projection assumption,non-Lambertian photometric stereo is a highly non-linear problem.In this study,we present an optimized framework for reconstructing the surface normal and depth with non-Lambertian reflection models under perspective projection.By decomposing the images into diffuse and specular components,we compute the surface normal and reflectance simultaneously.We also propose a variational formulation that is robust and useful for surface reconstruction.The experiments show that our method accurately reconstructs both the surface shape and reflectance of colorful objects with non-Lambertian surfaces.展开更多
Photometric stereo is a fundamental technique in computer vision known to produce 3D shape with high accuracy. It uses several input images of a static scene taken from one and the same camera position but under varyi...Photometric stereo is a fundamental technique in computer vision known to produce 3D shape with high accuracy. It uses several input images of a static scene taken from one and the same camera position but under varying illumination. The vast majority of studies in this 3D reconstruction method assume orthographic projection for the camera model.In addition, they mainly use the Lambertian reflectance model as the way that light scatters at surfaces.Thus, providing reliable photometric stereo results from real world objects still remains a challenging task. We address 3D reconstruction by use of a more realistic set of assumptions, combining for the first time the complete Blinn–Phong reflectance model and perspective projection. Furthermore, we compare two different methods of incorporating the perspective projection into our model. Experiments are performed on both synthetic and real world images; the latter do not benefit from laboratory conditions. The results show the high potential of our method even for complex real world applications such as medical endoscopy images which may include many specular highlights.展开更多
An effective method for object shape recovery using HDRIs （high dynamic range images） is proposed. The radiance values of each point on the reference sphere and target object are firstly calculated, thus the set of ...An effective method for object shape recovery using HDRIs （high dynamic range images） is proposed. The radiance values of each point on the reference sphere and target object are firstly calculated, thus the set of candidate normals of each target point are found by comparing its radiance to that of each reference sphere point. In single-image shape recovery, a smoothness operation is applied to the target normals to obtain a stable and reasonable result; while in photometric stereo, radiance vectors of reference and target objects formed due to illuminations under different fight source directions are directly compared to get the most suitable target normals. Finally, the height values can be recovered from the resulting normal field. Because diffuse and specular reflection are handled in an unified framework with radiance, our approach eliminates the limitation presented in most recovery strategies, i.e., only Lambertian model can be used. The experiment results from the real and synthesized images show the performance of our approach.展开更多
The integration of surface normals for the purpose of computing the shape of a surface in 3D space is a classic problem in computer vision. However,even nowadays it is still a challenging task to devise a method that ...The integration of surface normals for the purpose of computing the shape of a surface in 3D space is a classic problem in computer vision. However,even nowadays it is still a challenging task to devise a method that is flexible enough to work on non-trivial computational domains with high accuracy, robustness,and computational efficiency. By uniting a classic approach for surface normal integration with modern computational techniques, we construct a solver that fulfils these requirements. Building upon the Poisson integration model, we use an iterative Krylov subspace solver as a core step in tackling the task. While such a method can be very efficient, it may only show its full potential when combined with suitable numerical preconditioning and problem-specific initialisation. We perform a thorough numerical study in order to identify an appropriate preconditioner for this purpose.To provide suitable initialisation, we compute this initial state using a recently developed fast marching integrator. Detailed numerical experiments illustrate the benefits of this novel combination. In addition, we show on real-world photometric stereo datasets that the developed numerical framework is flexible enough to tackle modern computer vision applications.展开更多
基金The work is partly supported by the Natural Science Basic Research Plan in Shaanxi Province of China(No.2016JM6041).
文摘There are hundreds of welded studs in a car.The posture of a welded stud determines the quality of the body assembly,thus affecting the safety of cars.It is crucial to detect the posture of the welded studs.Considering the lack of accurate method in detecting the position of welded studs,this paper aims to detect the weld stud’s pose based on photometric stereo and neural network.Firstly,a machine vision-based stud dataset collection system is built to achieve the stud dataset labelling automatically.Secondly,photometric stereo algorithm is applied to estimate the stud normal map which as input is fed to neural network.Finally,we improve a lightweight YOLOv4 neural network which is applied to achieve the detection of stud position,thus overcoming the shortcomings of traditional testing methods.The research and experimental results show that the stud pose detection system designed achieves rapid detection and high accuracy positioning of the stud.This research provides the foundation combining the photometric stereo and deep learning for object detection in industrial production.
文摘Colonoscopy screening for the detection and removal of colonic adenomas is central to efforts to reduce the morbidity and mortality of colorectal cancer.However,up to a third of adenomas may be missed at colonoscopy,and the majority of post-colonoscopy colorectal cancers are thought to arise from these.Adenomas have three-dimensional surface topographic features that differentiate them from adjacent normal mucosa.However,these topographic features are not enhanced by white light colonoscopy,and the endoscopist must infer these from two-dimensional cues.This may contribute to the number of missed lesions.A variety of optical imaging technologies have been developed commercially to enhance surface topography.However,existing techniques enhance surface topography indirectly,and in two dimensions,and the evidence does not wholly support their use in routine clinical practice.In this narrative review,co-authored by gastroenterologists and engineers,we summarise the evidence for the impact of established optical imaging technologies on adenoma detection rate,and review the development of photometric stereo(PS)for colonoscopy.PS is a machine vision technique able to capture a dense array of surface normals to render three-dimensional reconstructions of surface topography.This imaging technique has several potential clinical applications in colonoscopy,including adenoma detection,polyp classification,and facilitating polypectomy,an inherently three-dimensional task.However,the development of PS for colonoscopy is at an early stage.We consider the progress that has been made with PS to date and identify the obstacles that need to be overcome prior to clinical application.
文摘Background A photometric stereo method aims to recover the surface normal of a 3D object observed under varying light directions.It is an ill-defined problem because the general reflectance properties of the surface are unknown.Methods This paper reviews existing data-driven methods,with a focus on their technical insights into the photometric stereo problem.We divide these methods into two categories,per-pixel and all-pixel,according to how they process an image.We discuss the differences and relationships between these methods from the perspective of inputs,networks,and data,which are key factors in designing a deep learning approach.Results We demonstrate the performance of the models using a popular benchmark dataset.Conclusions Data-driven photometric stereo methods have shown that they possess a superior performance advantage over traditional methods.However,these methods suffer from various limitations,such as limited generalization capability.Finally,this study suggests directions for future research.
基金the National Natural Science Foundation of China(No.61927822)。
文摘Non-uniformity of light sources is one of the inevitable error factors causing poor shape recoveryaccuracy of photometric stereo methods under close-range lighting with quasi point lights. Semi-calibrated photometricstereo methods are required to avoid repeated, tedious and impractical photometric calibration. In thispaper, two simple, concise but effective mesh-based semi-calibrated photometric stereo methods are proposed.The proposed methods extend the traditional mesh-based photometric stereo methods and further allow joint andaccurate estimation of normals and non-uniform light intensities by alternatively updating normals, depth mapsand intensities. Extensive experiments are conducted to validate the effectiveness and robustness of the proposedalgorithms. Even under extremely severe non-uniform lighting, the proposed methods can still suppress the errorand improve the shape recovery accuracy by up to 65.6% in real-world experiments.
基金supported by the National Key Scientific Instrument and Equipment Development Projects of China(41927805)the National Natural Science Foundation of China(61501417,61976123)+1 种基金the Key Development Program for Basic Research of Shandong Province(ZR2020ZD44)the Taishan Young Scholars Program of Shandong Province.
文摘Photometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination.Traditional methods normally adopt simplified reflectance models to make the surface orientation computable.However,the real reflectances of surfaces greatly limit applicability of such methods to real-world objects.While deep neural networks have been employed to handle non-Lambertian surfaces,these methods are subject to blurring and errors,especially in high-frequency regions(such as crinkles and edges),caused by spectral bias:neural networks favor low-frequency representations so exhibit a bias towards smooth functions.In this paper,therefore,we propose a self-learning conditional network with multiscale features for photometric stereo,avoiding blurred reconstruction in such regions.Our explorations include:(i)a multi-scale feature fusion architecture,which keeps high-resolution representations and deep feature extraction,simultaneously,and(ii)an improved gradient-motivated conditionally parameterized convolution(GM-CondConv)in our photometric stereo network,with different combinations of convolution kernels for varying surfaces.Extensive experiments on public benchmark datasets show that our calibrated photometric stereo method outperforms the state-of-the-art.
基金Supported by the National Natural Science Foundation of China(61876106)。
文摘Fabric pilling evaluation has been considered as an essential element for textile quality inspection. Traditional manual method is still based on human eyes and brain, which is subjective with low efficiency. This paper proposes an objective evaluation method based on semi-calibrated near-light Photometric Stereo(PS). Fabric images are digitalized by self-developed image acquisition system. The 3D depth information of each point could be obtained by PS algorithm and then mapped to 2D grayscale image. After that, the non-textured image could be filtered by using the Gaussian low-pass filter. The pilling segmentation is conducted by using global iterative threshold segmentation method,and then K-Nearest Neighbor(KNN) is finally selected as a tool for the grade classification of fabric pilling. Our experimental results show that the proposed evaluation system could achieve excellent judging performance for the objective pilling evaluation.
基金the Technological Program of Cultural Relics Preservation of Zhejiang Province,Chinathe Key Research and Development Program of Zhejiang Province,China(No.2018C03051)the National Standard Development Program of Cultural Relics Protection of China(No.581250-T0170B)。
文摘Under the perspective projection assumption,non-Lambertian photometric stereo is a highly non-linear problem.In this study,we present an optimized framework for reconstructing the surface normal and depth with non-Lambertian reflection models under perspective projection.By decomposing the images into diffuse and specular components,we compute the surface normal and reflectance simultaneously.We also propose a variational formulation that is robust and useful for surface reconstruction.The experiments show that our method accurately reconstructs both the surface shape and reflectance of colorful objects with non-Lambertian surfaces.
基金supported by the Deutsche Forschungsgemeinschaft under grant number BR2245/4–1
文摘Photometric stereo is a fundamental technique in computer vision known to produce 3D shape with high accuracy. It uses several input images of a static scene taken from one and the same camera position but under varying illumination. The vast majority of studies in this 3D reconstruction method assume orthographic projection for the camera model.In addition, they mainly use the Lambertian reflectance model as the way that light scatters at surfaces.Thus, providing reliable photometric stereo results from real world objects still remains a challenging task. We address 3D reconstruction by use of a more realistic set of assumptions, combining for the first time the complete Blinn–Phong reflectance model and perspective projection. Furthermore, we compare two different methods of incorporating the perspective projection into our model. Experiments are performed on both synthetic and real world images; the latter do not benefit from laboratory conditions. The results show the high potential of our method even for complex real world applications such as medical endoscopy images which may include many specular highlights.
基金the National Basic Research Program of China(No.2006CB303105)
文摘An effective method for object shape recovery using HDRIs （high dynamic range images） is proposed. The radiance values of each point on the reference sphere and target object are firstly calculated, thus the set of candidate normals of each target point are found by comparing its radiance to that of each reference sphere point. In single-image shape recovery, a smoothness operation is applied to the target normals to obtain a stable and reasonable result; while in photometric stereo, radiance vectors of reference and target objects formed due to illuminations under different fight source directions are directly compared to get the most suitable target normals. Finally, the height values can be recovered from the resulting normal field. Because diffuse and specular reflection are handled in an unified framework with radiance, our approach eliminates the limitation presented in most recovery strategies, i.e., only Lambertian model can be used. The experiment results from the real and synthesized images show the performance of our approach.
文摘The integration of surface normals for the purpose of computing the shape of a surface in 3D space is a classic problem in computer vision. However,even nowadays it is still a challenging task to devise a method that is flexible enough to work on non-trivial computational domains with high accuracy, robustness,and computational efficiency. By uniting a classic approach for surface normal integration with modern computational techniques, we construct a solver that fulfils these requirements. Building upon the Poisson integration model, we use an iterative Krylov subspace solver as a core step in tackling the task. While such a method can be very efficient, it may only show its full potential when combined with suitable numerical preconditioning and problem-specific initialisation. We perform a thorough numerical study in order to identify an appropriate preconditioner for this purpose.To provide suitable initialisation, we compute this initial state using a recently developed fast marching integrator. Detailed numerical experiments illustrate the benefits of this novel combination. In addition, we show on real-world photometric stereo datasets that the developed numerical framework is flexible enough to tackle modern computer vision applications.
