Multi-modal histological image registration tasks pose significant challenges due to tissue staining operations causing partial loss and folding of tissue.Convolutional neural network(CNN)and generative adversarial ne...Multi-modal histological image registration tasks pose significant challenges due to tissue staining operations causing partial loss and folding of tissue.Convolutional neural network(CNN)and generative adversarial network(GAN)are pivotal inmedical image registration.However,existing methods often struggle with severe interference and deformation,as seen in histological images of conditions like Cushing’s disease.We argue that the failure of current approaches lies in underutilizing the feature extraction capability of the discriminator inGAN.In this study,we propose a novel multi-modal registration approach GAN-DIRNet based on GAN for deformable histological image registration.To begin with,the discriminators of two GANs are embedded as a new dual parallel feature extraction module into the unsupervised registration networks,characterized by implicitly extracting feature descriptors of specific modalities.Additionally,modal feature description layers and registration layers collaborate in unsupervised optimization,facilitating faster convergence and more precise results.Lastly,experiments and evaluations were conducted on the registration of the Mixed National Institute of Standards and Technology database(MNIST),eight publicly available datasets of histological sections and the Clustering-Registration-Classification-Segmentation(CRCS)dataset on the Cushing’s disease.Experimental results demonstrate that our proposed GAN-DIRNet method surpasses existing approaches like DIRNet in terms of both registration accuracy and time efficiency,while also exhibiting robustness across different image types.展开更多
Multi‐modal brain image registration has been widely applied to functional localisation,neurosurgery and computational anatomy.The existing registration methods based on the dense deformation fields involve too many ...Multi‐modal brain image registration has been widely applied to functional localisation,neurosurgery and computational anatomy.The existing registration methods based on the dense deformation fields involve too many parameters,which is not conducive to the exploration of correct spatial correspondence between the float and reference images.Meanwhile,the unidirectional registration may involve the deformation folding,which will result in the change of topology during registration.To address these issues,this work has presented an unsupervised image registration method using the free form deformation(FFD)and the symmetry constraint‐based generative adversarial networks(FSGAN).The FSGAN utilises the principle component analysis network‐based structural representations of the reference and float images as the inputs and uses the generator to learn the FFD model parameters,thereby producing two deformation fields.Meanwhile,the FSGAN uses two discriminators to decide whether the bilateral registration have been realised simultaneously.Besides,the symmetry constraint is utilised to construct the loss function,thereby avoiding the deformation folding.Experiments on BrainWeb,high grade gliomas,IXI and LPBA40 show that compared with state‐of‐the‐art methods,the FSGAN provides superior performance in terms of visual comparisons and such quantitative indexes as dice value,target registration error and computational efficiency.展开更多
In order to improve the registration accuracy of brain magnetic resonance images(MRI),some deep learning registration methods use segmentation images for training model.How-ever,the segmentation values are constant fo...In order to improve the registration accuracy of brain magnetic resonance images(MRI),some deep learning registration methods use segmentation images for training model.How-ever,the segmentation values are constant for each label,which leads to the gradient variation con-centrating on the boundary.Thus,the dense deformation field(DDF)is gathered on the boundary and there even appears folding phenomenon.In order to fully leverage the label information,the morphological opening and closing information maps are introduced to enlarge the non-zero gradi-ent regions and improve the accuracy of DDF estimation.The opening information maps supervise the registration model to focus on smaller,narrow brain regions.The closing information maps supervise the registration model to pay more attention to the complex boundary region.Then,opening and closing morphology networks(OC_Net)are designed to automatically generate open-ing and closing information maps to realize the end-to-end training process.Finally,a new registra-tion architecture,VM_(seg+oc),is proposed by combining OC_Net and VoxelMorph.Experimental results show that the registration accuracy of VM_(seg+oc) is significantly improved on LPBA40 and OASIS1 datasets.Especially,VM_(seg+oc) can well improve registration accuracy in smaller brain regions and narrow regions.展开更多
Mutual information is widely used in medical image registration, because it does not require preprocessing the image. However, the local maximum problem in the registration is insurmountable. We combine mutual informa...Mutual information is widely used in medical image registration, because it does not require preprocessing the image. However, the local maximum problem in the registration is insurmountable. We combine mutual information and gradient information to solve this problem and apply it to the non-rigid deformation image registration. To improve the accuracy, we provide some implemental issues, for example, the Powell searching algorithm, gray interpolation and consideration of outlier points. The experimental results show the accuracy of the method and the feasibility in non-rigid medical image registration.展开更多
Point features, as the basis of lines, surfaces, and bodies, are commonly used in medical image registration. To obtain an elegant spatial transformation of extracted feature points, many point set matching algorithms...Point features, as the basis of lines, surfaces, and bodies, are commonly used in medical image registration. To obtain an elegant spatial transformation of extracted feature points, many point set matching algorithms(PMs) have been developed to match two point sets by optimizing multifarious distance functions. There are ample reviews related to medical image registration and PMs which summarize their basic principles and main algorithms separately. However,to data, detailed summary of PMs used in medical image registration in different clinical environments has not been published. In this paper, we provide a comprehensive review of the existing key techniques of the PMs applied to medical image registration according to the basic principles and clinical applications. As the core technique of the PMs, geometric transformation models are elaborated in this paper, demonstrating the mechanism of point set registration. We also focus on the clinical applications of the PMs and propose a practical classification method according to their applications in different clinical surgeries. The aim of this paper is to provide a summary of pointfeaturebased methods used in medical image registration and to guide doctors or researchers interested in this field to choose appropriate techniques in their research.展开更多
Image fusion aims to integrate complementary information in source images to synthesize a fused image comprehensively characterizing the imaging scene. However, existing image fusion algorithms are only applicable to ...Image fusion aims to integrate complementary information in source images to synthesize a fused image comprehensively characterizing the imaging scene. However, existing image fusion algorithms are only applicable to strictly aligned source images and cause severe artifacts in the fusion results when input images have slight shifts or deformations. In addition,the fusion results typically only have good visual effect, but neglect the semantic requirements of high-level vision tasks.This study incorporates image registration, image fusion, and semantic requirements of high-level vision tasks into a single framework and proposes a novel image registration and fusion method, named Super Fusion. Specifically, we design a registration network to estimate bidirectional deformation fields to rectify geometric distortions of input images under the supervision of both photometric and end-point constraints. The registration and fusion are combined in a symmetric scheme, in which while mutual promotion can be achieved by optimizing the naive fusion loss, it is further enhanced by the mono-modal consistent constraint on symmetric fusion outputs. In addition, the image fusion network is equipped with the global spatial attention mechanism to achieve adaptive feature integration. Moreover, the semantic constraint based on the pre-trained segmentation model and Lovasz-Softmax loss is deployed to guide the fusion network to focus more on the semantic requirements of high-level vision tasks. Extensive experiments on image registration, image fusion,and semantic segmentation tasks demonstrate the superiority of our Super Fusion compared to the state-of-the-art alternatives.The source code and pre-trained model are publicly available at https://github.com/Linfeng-Tang/Super Fusion.展开更多
This paper presents a novel medical image registration algorithm named total variation constrained graphregularization for non-negative matrix factorization(TV-GNMF).The method utilizes non-negative matrix factorizati...This paper presents a novel medical image registration algorithm named total variation constrained graphregularization for non-negative matrix factorization(TV-GNMF).The method utilizes non-negative matrix factorization by total variation constraint and graph regularization.The main contributions of our work are the following.First,total variation is incorporated into NMF to control the diffusion speed.The purpose is to denoise in smooth regions and preserve features or details of the data in edge regions by using a diffusion coefficient based on gradient information.Second,we add graph regularization into NMF to reveal intrinsic geometry and structure information of features to enhance the discrimination power.Third,the multiplicative update rules and proof of convergence of the TV-GNMF algorithm are given.Experiments conducted on datasets show that the proposed TV-GNMF method outperforms other state-of-the-art algorithms.展开更多
The proposed algorithm relies on a group of new formulas for calculating tangent slope so as to address angle feature of edge curves of image. It can utilize tangent angle features to estimate automatically and fully ...The proposed algorithm relies on a group of new formulas for calculating tangent slope so as to address angle feature of edge curves of image. It can utilize tangent angle features to estimate automatically and fully the rotation parameters of geometric transform and enable rough matching of images with huge rotation difference. After angle compensation, it can search for matching point sets by correlation criterion, then calculate parameters of affine transform, enable higher-precision emendation of rotation and transferring. Finally, it fulfills precise matching for images with relax-tense iteration method. Compared with the registration approach based on wavelet direction-angle features, the matching algorithm with tangent feature of image edge is more robust and realizes precise registration of various images. Furthermore, it is also helpful in graphics matching.展开更多
Multi-sensor image registration has been widely used in remote sensing and medical image field, but registration performance is degenerated when heterogeneous images are involved. An image registration method based on...Multi-sensor image registration has been widely used in remote sensing and medical image field, but registration performance is degenerated when heterogeneous images are involved. An image registration method based on multi-resolution shape analysis is proposed in this paper, to deal with the problem that the shape of similar objects is always invariant. The contours of shapes are first detected as visual features using an extended contour search algorithm in order to reduce effects of noise, and the multi-resolution shape descriptor is constructed through Fourier curvature representation of the contour’s chain code. Then a minimum distance function is used to judge the similarity between two contours. To avoid the effect of different resolution and intensity distribution, suitable resolution of each image is selected by maximizing the consistency of its pyramid shapes. Finally, the transformation parameters are estimated based on the matched control-point pairs which are the centers of gravity of the closed contours. Multi-sensor Landsat TM imagery and infrared imagery have been used as experimental data for comparison with the classical contour-based registration. Our results have been shown to be superior to the classical ones.展开更多
Homologous feature point extraction is a key problem in the optical and synthetic aperture radar （SAR） image registration for islands. A new feature point extraction method using a threshold shrink operator and non-...Homologous feature point extraction is a key problem in the optical and synthetic aperture radar （SAR） image registration for islands. A new feature point extraction method using a threshold shrink operator and non-subsampled wavelet transform （TSO-NSWT） for optical and SAR image registration was proposed. Moreover, the matching for this proposed feature was different from the traditional feature matching strategies and was performed using a similarity measure computed from neighborhood circles in low-frequency bands. Then, a number of reliably matched couples with even distributions were obtained, which assured the accuracy of the registration. Application of the proposed algorithm to SPOT-5 （multi-spectral） and YG-1 （SAR） images showed that a large number of accurately matched couples could be identified. Additionally, both of the root mean square error （RMSE） patterns of the registration parameters computed based on the TSO-NSWT algorithm and traditional NSWT algorithm were analyzed and compared, which further demonstrated the effectiveness of the proposed algorithm. The algorithm can supply the crucial step for island image registration and island recognition.展开更多
A new coarse-to-fine strategy was proposed for nonrigid registration of computed tomography(CT) and magnetic resonance(MR) images of a liver.This hierarchical framework consisted of an affine transformation and a B-sp...A new coarse-to-fine strategy was proposed for nonrigid registration of computed tomography(CT) and magnetic resonance(MR) images of a liver.This hierarchical framework consisted of an affine transformation and a B-splines free-form deformation(FFD).The affine transformation performed a rough registration targeting the mismatch between the CT and MR images.The B-splines FFD transformation performed a finer registration by correcting local motion deformation.In the registration algorithm,the normalized mutual information(NMI) was used as similarity measure,and the limited memory Broyden-Fletcher- Goldfarb-Shannon(L-BFGS) optimization method was applied for optimization process.The algorithm was applied to the fully automated registration of liver CT and MR images in three subjects.The results demonstrate that the proposed method not only significantly improves the registration accuracy but also reduces the running time,which is effective and efficient for nonrigid registration.展开更多
Presence of higher breast density(BD)and persistence over time are risk factors for breast cancer.A quantitatively accurate and highly reproducible BD measure that relies on precise and reproducible whole-breast segme...Presence of higher breast density(BD)and persistence over time are risk factors for breast cancer.A quantitatively accurate and highly reproducible BD measure that relies on precise and reproducible whole-breast segmentation is desirable.In this study,we aimed to develop a highly reproducible and accurate whole-breast segmentation algorithm for the generation of reproducible BD measures.Three datasets of volunteers from two clinical trials were included.Breast MR images were acquired on 3T Siemens Biograph mMR,Prisma,and Skyra using 3D Cartesian six-echo GRE sequences with a fat-water separation technique.Two whole-breast segmentation strategies,utiliz-ing image registration and 3D U-Net,were developed.Manual segmentation was performed.A task-based analysis was performed:a previously developed MR-based BD measure,MagDensity,was calculated and assessed using automated and manual segmentation.The mean squared error(MSE)and intraclass correlation coefficient(ICC)between MagDensity were evaluated using the manual segmentation as a reference.The test-retest reproducibility of MagDensity derived from different breast segmentation methods was assessed using the difference between the test and retest measures(Δ_(2-1)),MSE,and ICC.The results showed that MagDensity derived by the registration and deep learning segmentation methods exhibited high concordance with manual segmentation,with ICCs of 0.986(95%CI:0.974-0.993)and 0.983(95%CI:0.961-0.992),respectively.For test-retest analysis,MagDensity derived using the regis-tration algorithm achieved the smallest MSE of 0.370 and highest ICC of 0.993(95%CI:0.982-0.997)when compared to other segmentation methods.In conclusion,the proposed registration and deep learning whole-breast segmentation methods are accurate and reliable for estimating BD.Both methods outperformed a previously developed algorithm and manual segmentation in the test-retest assessment,with the registration exhibiting superior performance for highly reproducible BD measurements.展开更多
Image registration is an important research topic in the field of computer vision,in which the registration and mosaic of side-scan sonar images is the keypoints of underwater navigation.However,the image registration...Image registration is an important research topic in the field of computer vision,in which the registration and mosaic of side-scan sonar images is the keypoints of underwater navigation.However,the image registration method of keypoints is not suitable for sonar images which do not have obvious feature points.Therefore,a method of sonar-image registration and mosaic based on line segment extraction and triangle matching is proposed in this paper.Firstly,in order to extract features from sonar image,the LSD method is introduced to detect line feature from images,and line segments are filtered by the principle of attention;after that,triangles are formed from line segments,an image transformation matrix can be calculated through the heuristic greedy algorithm from these triangles;finally,images are merged based on the transformation information.On the basis of practical tests,it is found that,the feature extraction method used in this paper can better describe the outline of underwater terrain,and there is no obvious stitching gap between the result of sonar images stitched.Experimental results show that the proposed method is effective than the keypoints method of the registration and mosaic of sonar images.展开更多
An improved image registration method is proposed based on mutual infor- mation with hybrid optimizer. Firstly, mutual information measure is combined with morphological gradient information. The essence of the gradie...An improved image registration method is proposed based on mutual infor- mation with hybrid optimizer. Firstly, mutual information measure is combined with morphological gradient information. The essence of the gradient information is that locations a large gradient magnitude should be aligned, but also the orientation of the gradients at those locations should be similar. Secondly, a hybrid optimizer combined PSO with Powell algorithm is proposed to restrain local maxima of mutual information function and improve the registration accuracy to sub-pixel level. Lastly, muhlresolution data structure based on Mallat decomposition can not only improve the behavior of registration function, but also improve the speed of the algorithm. Experimental results demonstrate that the new method can yield good registration result, superior to traditional optimizer with respect to smoothness and attraction basin as well as convergence speed.展开更多
In this paper, a new approach of muhi-modality image registration is represented with not only image intensity, but also features describing image structure. There are two novelties in the proposed method. Firstly, in...In this paper, a new approach of muhi-modality image registration is represented with not only image intensity, but also features describing image structure. There are two novelties in the proposed method. Firstly, instead of standard mutual information （ MI ） based on joint intensity histogram, regional mutual information （ RMI ） is employed, which allows neighborhood information to be taken into account. Secondly, a new feature images obtained by means of phase congruency are invariants to brightness or contrast changes. By incorporating these features and intensity into RMI, we can combine the aspects of both structural and neighborhood information together, which offers a more robust and a high level of registration accuracy.展开更多
In this paper, the authors propose a refined Branch-and-Bound algorithm for affine-transformation based image registration. Given two feature point-sets in two images respectively, the authors first extract a sequence...In this paper, the authors propose a refined Branch-and-Bound algorithm for affine-transformation based image registration. Given two feature point-sets in two images respectively, the authors first extract a sequence of high-probability matched point-pairs by considering well-defined features. Each resultant point-pair can be regarded as a constraint in the search space of Branch-and-Bound algorithm guiding the search process. The authors carry out Branch-and-Bound search with the constraint of a pair-point selected by using Monte Carlo sampling according to the match measures of point-pairs. If such one cannot lead to correct result, additional candidate is chosen to start another search. High-probability matched point-pairs usually results in fewer loops and the search process is accelerated greatly. Experimental results verify the high efficiency and robustness of the author’s approach.展开更多
How to improve the probability of registration and precision of localization is a hard problem, which is desiderated to solve. The two basic approaches (normalized cross-correlation and phase correlation) for image re...How to improve the probability of registration and precision of localization is a hard problem, which is desiderated to solve. The two basic approaches (normalized cross-correlation and phase correlation) for image registration are analysed, two improved approaches based on spatial-temporal relationship are presented. This method adds the correlation matrix according to the displacements in x- cirection and y- directions, and the registration pose is searched in the added matrix. The method overcomes the shortcoming that the probability of registration decreasing with area increasing owing to geometric distortion, improves the probability and the robustness of registration.展开更多
This paper describes a new method for three-dimensional medical image registration. In the interactive image-guided HIFU （ High Intensity Focused Ultrasound） therapy system, a fast and precise localization of the tu...This paper describes a new method for three-dimensional medical image registration. In the interactive image-guided HIFU （ High Intensity Focused Ultrasound） therapy system, a fast and precise localization of the tumor is very important. An automatic system is developed for registering pre-operative MR images with intra-operative ultrasound images based on the vessels visible in both of the modalities. When the MR and the ultrasound images are aligned, the eenterline points of the vessels in the MR image will align with bright intensities in the ultrasound image. The method applies an optimization strategy combining the genetic algorithm with the conjugated gradients algorithm to minimize the objective function. It provides a feasible way of determining the global solution and makes the method robust to local maximum and insensitive to initial position. Two experiments were designed to evaluate the method, and the results show that our method has better registration accuracy and convergence rate than the other two classic algorithms.展开更多
An automatic image registration approach based on wavelet transform is proposed. This proposed method utilizes multiscale wavelet transform to extract feature points. A coarse-to-fine feature matching method is utiliz...An automatic image registration approach based on wavelet transform is proposed. This proposed method utilizes multiscale wavelet transform to extract feature points. A coarse-to-fine feature matching method is utilized in the feature matching phase. A two-way matching method based on cross-correlation to get candidate point pairs and a fine matching based on support strength combine to form the matching algorithm. At last, based on an affine transformation model, the parameters are iteratively refined by using the leastsquares estimation approach. Experimental results have verified that the proposed algorithm can realize automatic registration of various kinds of images rapidly and effectively.展开更多
An intensity-based non-rigid registration algorithm is discussed, which uses Gaussian smoothing to constrain the transformation to be smooth, and thus preserves the topology of images. In view of the insufficiency of ...An intensity-based non-rigid registration algorithm is discussed, which uses Gaussian smoothing to constrain the transformation to be smooth, and thus preserves the topology of images. In view of the insufficiency of the uniform Gaussian filtering of the deformation field, an automatic and accurate non-rigid image registration method based on B-splines approximation is proposed. The regularization strategy is adopted by using multi-level B-splines approximation to regularize the displacement fields in a coarse-to-fine manner. Moreover, it assigns the different weights to the estimated displacements according to their reliabilities. In this way, the level of regularity can be adapted locally. Experiments were performed on both synthetic and real medical images of brain, and the results show that the proposed method improves the registration accuracy and robustness.展开更多
文摘Multi-modal histological image registration tasks pose significant challenges due to tissue staining operations causing partial loss and folding of tissue.Convolutional neural network(CNN)and generative adversarial network(GAN)are pivotal inmedical image registration.However,existing methods often struggle with severe interference and deformation,as seen in histological images of conditions like Cushing’s disease.We argue that the failure of current approaches lies in underutilizing the feature extraction capability of the discriminator inGAN.In this study,we propose a novel multi-modal registration approach GAN-DIRNet based on GAN for deformable histological image registration.To begin with,the discriminators of two GANs are embedded as a new dual parallel feature extraction module into the unsupervised registration networks,characterized by implicitly extracting feature descriptors of specific modalities.Additionally,modal feature description layers and registration layers collaborate in unsupervised optimization,facilitating faster convergence and more precise results.Lastly,experiments and evaluations were conducted on the registration of the Mixed National Institute of Standards and Technology database(MNIST),eight publicly available datasets of histological sections and the Clustering-Registration-Classification-Segmentation(CRCS)dataset on the Cushing’s disease.Experimental results demonstrate that our proposed GAN-DIRNet method surpasses existing approaches like DIRNet in terms of both registration accuracy and time efficiency,while also exhibiting robustness across different image types.
基金supported in part by the National Key Research and Development Program of China under Grant 2018Y FE0206900in part by the National Natural Science Foundation of China under Grant 61871440in part by the CAAIHuawei MindSpore Open Fund.We gratefully acknowledge the support of MindSpore for this research.
文摘Multi‐modal brain image registration has been widely applied to functional localisation,neurosurgery and computational anatomy.The existing registration methods based on the dense deformation fields involve too many parameters,which is not conducive to the exploration of correct spatial correspondence between the float and reference images.Meanwhile,the unidirectional registration may involve the deformation folding,which will result in the change of topology during registration.To address these issues,this work has presented an unsupervised image registration method using the free form deformation(FFD)and the symmetry constraint‐based generative adversarial networks(FSGAN).The FSGAN utilises the principle component analysis network‐based structural representations of the reference and float images as the inputs and uses the generator to learn the FFD model parameters,thereby producing two deformation fields.Meanwhile,the FSGAN uses two discriminators to decide whether the bilateral registration have been realised simultaneously.Besides,the symmetry constraint is utilised to construct the loss function,thereby avoiding the deformation folding.Experiments on BrainWeb,high grade gliomas,IXI and LPBA40 show that compared with state‐of‐the‐art methods,the FSGAN provides superior performance in terms of visual comparisons and such quantitative indexes as dice value,target registration error and computational efficiency.
基金supported by Shandong Provincial Natural Science Foundation(No.ZR2023MF062)the National Natural Science Foundation of China(No.61771230).
文摘In order to improve the registration accuracy of brain magnetic resonance images(MRI),some deep learning registration methods use segmentation images for training model.How-ever,the segmentation values are constant for each label,which leads to the gradient variation con-centrating on the boundary.Thus,the dense deformation field(DDF)is gathered on the boundary and there even appears folding phenomenon.In order to fully leverage the label information,the morphological opening and closing information maps are introduced to enlarge the non-zero gradi-ent regions and improve the accuracy of DDF estimation.The opening information maps supervise the registration model to focus on smaller,narrow brain regions.The closing information maps supervise the registration model to pay more attention to the complex boundary region.Then,opening and closing morphology networks(OC_Net)are designed to automatically generate open-ing and closing information maps to realize the end-to-end training process.Finally,a new registra-tion architecture,VM_(seg+oc),is proposed by combining OC_Net and VoxelMorph.Experimental results show that the registration accuracy of VM_(seg+oc) is significantly improved on LPBA40 and OASIS1 datasets.Especially,VM_(seg+oc) can well improve registration accuracy in smaller brain regions and narrow regions.
文摘Mutual information is widely used in medical image registration, because it does not require preprocessing the image. However, the local maximum problem in the registration is insurmountable. We combine mutual information and gradient information to solve this problem and apply it to the non-rigid deformation image registration. To improve the accuracy, we provide some implemental issues, for example, the Powell searching algorithm, gray interpolation and consideration of outlier points. The experimental results show the accuracy of the method and the feasibility in non-rigid medical image registration.
基金Supported by the National Natural Science Foundation of China(Grant No.61533016)
文摘Point features, as the basis of lines, surfaces, and bodies, are commonly used in medical image registration. To obtain an elegant spatial transformation of extracted feature points, many point set matching algorithms(PMs) have been developed to match two point sets by optimizing multifarious distance functions. There are ample reviews related to medical image registration and PMs which summarize their basic principles and main algorithms separately. However,to data, detailed summary of PMs used in medical image registration in different clinical environments has not been published. In this paper, we provide a comprehensive review of the existing key techniques of the PMs applied to medical image registration according to the basic principles and clinical applications. As the core technique of the PMs, geometric transformation models are elaborated in this paper, demonstrating the mechanism of point set registration. We also focus on the clinical applications of the PMs and propose a practical classification method according to their applications in different clinical surgeries. The aim of this paper is to provide a summary of pointfeaturebased methods used in medical image registration and to guide doctors or researchers interested in this field to choose appropriate techniques in their research.
基金supported by the National Natural Science Foundation of China(62276192,62075169,62061160370)the Key Research and Development Program of Hubei Province(2020BAB113)。
文摘Image fusion aims to integrate complementary information in source images to synthesize a fused image comprehensively characterizing the imaging scene. However, existing image fusion algorithms are only applicable to strictly aligned source images and cause severe artifacts in the fusion results when input images have slight shifts or deformations. In addition,the fusion results typically only have good visual effect, but neglect the semantic requirements of high-level vision tasks.This study incorporates image registration, image fusion, and semantic requirements of high-level vision tasks into a single framework and proposes a novel image registration and fusion method, named Super Fusion. Specifically, we design a registration network to estimate bidirectional deformation fields to rectify geometric distortions of input images under the supervision of both photometric and end-point constraints. The registration and fusion are combined in a symmetric scheme, in which while mutual promotion can be achieved by optimizing the naive fusion loss, it is further enhanced by the mono-modal consistent constraint on symmetric fusion outputs. In addition, the image fusion network is equipped with the global spatial attention mechanism to achieve adaptive feature integration. Moreover, the semantic constraint based on the pre-trained segmentation model and Lovasz-Softmax loss is deployed to guide the fusion network to focus more on the semantic requirements of high-level vision tasks. Extensive experiments on image registration, image fusion,and semantic segmentation tasks demonstrate the superiority of our Super Fusion compared to the state-of-the-art alternatives.The source code and pre-trained model are publicly available at https://github.com/Linfeng-Tang/Super Fusion.
基金supported by the National Natural Science Foundation of China(61702251,41971424,61701191,U1605254)the Natural Science Basic Research Plan in Shaanxi Province of China(2018JM6030)+4 种基金the Key Technical Project of Fujian Province(2017H6015)the Science and Technology Project of Xiamen(3502Z20183032)the Doctor Scientific Research Starting Foundation of Northwest University(338050050)Youth Academic Talent Support Program of Northwest University(360051900151)the Natural Sciences and Engineering Research Council of Canada,Canada。
文摘This paper presents a novel medical image registration algorithm named total variation constrained graphregularization for non-negative matrix factorization(TV-GNMF).The method utilizes non-negative matrix factorization by total variation constraint and graph regularization.The main contributions of our work are the following.First,total variation is incorporated into NMF to control the diffusion speed.The purpose is to denoise in smooth regions and preserve features or details of the data in edge regions by using a diffusion coefficient based on gradient information.Second,we add graph regularization into NMF to reveal intrinsic geometry and structure information of features to enhance the discrimination power.Third,the multiplicative update rules and proof of convergence of the TV-GNMF algorithm are given.Experiments conducted on datasets show that the proposed TV-GNMF method outperforms other state-of-the-art algorithms.
基金Supported by the National Natural Science Foundation of China (No.60141002) the Aviation Basic Science Foundation (02I53073)
文摘The proposed algorithm relies on a group of new formulas for calculating tangent slope so as to address angle feature of edge curves of image. It can utilize tangent angle features to estimate automatically and fully the rotation parameters of geometric transform and enable rough matching of images with huge rotation difference. After angle compensation, it can search for matching point sets by correlation criterion, then calculate parameters of affine transform, enable higher-precision emendation of rotation and transferring. Finally, it fulfills precise matching for images with relax-tense iteration method. Compared with the registration approach based on wavelet direction-angle features, the matching algorithm with tangent feature of image edge is more robust and realizes precise registration of various images. Furthermore, it is also helpful in graphics matching.
基金Project supported by the National Natural Science Foundation of China (No. 60272031), the Hi-Tech Research and Development Program (863) of China (No. 2003AA131032-2), and the Natural Science Foundation of Zhejiang Province (No. M603202), China
文摘Multi-sensor image registration has been widely used in remote sensing and medical image field, but registration performance is degenerated when heterogeneous images are involved. An image registration method based on multi-resolution shape analysis is proposed in this paper, to deal with the problem that the shape of similar objects is always invariant. The contours of shapes are first detected as visual features using an extended contour search algorithm in order to reduce effects of noise, and the multi-resolution shape descriptor is constructed through Fourier curvature representation of the contour’s chain code. Then a minimum distance function is used to judge the similarity between two contours. To avoid the effect of different resolution and intensity distribution, suitable resolution of each image is selected by maximizing the consistency of its pyramid shapes. Finally, the transformation parameters are estimated based on the matched control-point pairs which are the centers of gravity of the closed contours. Multi-sensor Landsat TM imagery and infrared imagery have been used as experimental data for comparison with the classical contour-based registration. Our results have been shown to be superior to the classical ones.
基金The National Natural Science Foundation of China under contract No.41271409the National Key Technology Research and Development Program under contract No.2011BAH23B00the National High Technology Research and Development Program(863 Program)of China under contract No.2012AA12A406
文摘Homologous feature point extraction is a key problem in the optical and synthetic aperture radar （SAR） image registration for islands. A new feature point extraction method using a threshold shrink operator and non-subsampled wavelet transform （TSO-NSWT） for optical and SAR image registration was proposed. Moreover, the matching for this proposed feature was different from the traditional feature matching strategies and was performed using a similarity measure computed from neighborhood circles in low-frequency bands. Then, a number of reliably matched couples with even distributions were obtained, which assured the accuracy of the registration. Application of the proposed algorithm to SPOT-5 （multi-spectral） and YG-1 （SAR） images showed that a large number of accurately matched couples could be identified. Additionally, both of the root mean square error （RMSE） patterns of the registration parameters computed based on the TSO-NSWT algorithm and traditional NSWT algorithm were analyzed and compared, which further demonstrated the effectiveness of the proposed algorithm. The algorithm can supply the crucial step for island image registration and island recognition.
基金Project(61240010)supported by the National Natural Science Foundation of ChinaProject(20070007070)supported by Specialized Research Fund for the Doctoral Program of Higher Education of China
文摘A new coarse-to-fine strategy was proposed for nonrigid registration of computed tomography(CT) and magnetic resonance(MR) images of a liver.This hierarchical framework consisted of an affine transformation and a B-splines free-form deformation(FFD).The affine transformation performed a rough registration targeting the mismatch between the CT and MR images.The B-splines FFD transformation performed a finer registration by correcting local motion deformation.In the registration algorithm,the normalized mutual information(NMI) was used as similarity measure,and the limited memory Broyden-Fletcher- Goldfarb-Shannon(L-BFGS) optimization method was applied for optimization process.The algorithm was applied to the fully automated registration of liver CT and MR images in three subjects.The results demonstrate that the proposed method not only significantly improves the registration accuracy but also reduces the running time,which is effective and efficient for nonrigid registration.
基金This work is partially supported by the National Institute of Health R03CA223052The sulindac trial was supported by R01CA161534The metformin trial was supported by R01CA172444 and P30CA023074。
文摘Presence of higher breast density(BD)and persistence over time are risk factors for breast cancer.A quantitatively accurate and highly reproducible BD measure that relies on precise and reproducible whole-breast segmentation is desirable.In this study,we aimed to develop a highly reproducible and accurate whole-breast segmentation algorithm for the generation of reproducible BD measures.Three datasets of volunteers from two clinical trials were included.Breast MR images were acquired on 3T Siemens Biograph mMR,Prisma,and Skyra using 3D Cartesian six-echo GRE sequences with a fat-water separation technique.Two whole-breast segmentation strategies,utiliz-ing image registration and 3D U-Net,were developed.Manual segmentation was performed.A task-based analysis was performed:a previously developed MR-based BD measure,MagDensity,was calculated and assessed using automated and manual segmentation.The mean squared error(MSE)and intraclass correlation coefficient(ICC)between MagDensity were evaluated using the manual segmentation as a reference.The test-retest reproducibility of MagDensity derived from different breast segmentation methods was assessed using the difference between the test and retest measures(Δ_(2-1)),MSE,and ICC.The results showed that MagDensity derived by the registration and deep learning segmentation methods exhibited high concordance with manual segmentation,with ICCs of 0.986(95%CI:0.974-0.993)and 0.983(95%CI:0.961-0.992),respectively.For test-retest analysis,MagDensity derived using the regis-tration algorithm achieved the smallest MSE of 0.370 and highest ICC of 0.993(95%CI:0.982-0.997)when compared to other segmentation methods.In conclusion,the proposed registration and deep learning whole-breast segmentation methods are accurate and reliable for estimating BD.Both methods outperformed a previously developed algorithm and manual segmentation in the test-retest assessment,with the registration exhibiting superior performance for highly reproducible BD measurements.
文摘Image registration is an important research topic in the field of computer vision,in which the registration and mosaic of side-scan sonar images is the keypoints of underwater navigation.However,the image registration method of keypoints is not suitable for sonar images which do not have obvious feature points.Therefore,a method of sonar-image registration and mosaic based on line segment extraction and triangle matching is proposed in this paper.Firstly,in order to extract features from sonar image,the LSD method is introduced to detect line feature from images,and line segments are filtered by the principle of attention;after that,triangles are formed from line segments,an image transformation matrix can be calculated through the heuristic greedy algorithm from these triangles;finally,images are merged based on the transformation information.On the basis of practical tests,it is found that,the feature extraction method used in this paper can better describe the outline of underwater terrain,and there is no obvious stitching gap between the result of sonar images stitched.Experimental results show that the proposed method is effective than the keypoints method of the registration and mosaic of sonar images.
文摘An improved image registration method is proposed based on mutual infor- mation with hybrid optimizer. Firstly, mutual information measure is combined with morphological gradient information. The essence of the gradient information is that locations a large gradient magnitude should be aligned, but also the orientation of the gradients at those locations should be similar. Secondly, a hybrid optimizer combined PSO with Powell algorithm is proposed to restrain local maxima of mutual information function and improve the registration accuracy to sub-pixel level. Lastly, muhlresolution data structure based on Mallat decomposition can not only improve the behavior of registration function, but also improve the speed of the algorithm. Experimental results demonstrate that the new method can yield good registration result, superior to traditional optimizer with respect to smoothness and attraction basin as well as convergence speed.
文摘In this paper, a new approach of muhi-modality image registration is represented with not only image intensity, but also features describing image structure. There are two novelties in the proposed method. Firstly, instead of standard mutual information （ MI ） based on joint intensity histogram, regional mutual information （ RMI ） is employed, which allows neighborhood information to be taken into account. Secondly, a new feature images obtained by means of phase congruency are invariants to brightness or contrast changes. By incorporating these features and intensity into RMI, we can combine the aspects of both structural and neighborhood information together, which offers a more robust and a high level of registration accuracy.
基金Project supported by the National Basic Research Program (973) of China (No. 2002CB312101), the National Natural Science Founda-tion of China (Nos. 60475013 and 60273053) and Defense Science and Technology Key Lab. Foundation of China (No. 51476070101JW0409)
文摘In this paper, the authors propose a refined Branch-and-Bound algorithm for affine-transformation based image registration. Given two feature point-sets in two images respectively, the authors first extract a sequence of high-probability matched point-pairs by considering well-defined features. Each resultant point-pair can be regarded as a constraint in the search space of Branch-and-Bound algorithm guiding the search process. The authors carry out Branch-and-Bound search with the constraint of a pair-point selected by using Monte Carlo sampling according to the match measures of point-pairs. If such one cannot lead to correct result, additional candidate is chosen to start another search. High-probability matched point-pairs usually results in fewer loops and the search process is accelerated greatly. Experimental results verify the high efficiency and robustness of the author’s approach.
文摘How to improve the probability of registration and precision of localization is a hard problem, which is desiderated to solve. The two basic approaches (normalized cross-correlation and phase correlation) for image registration are analysed, two improved approaches based on spatial-temporal relationship are presented. This method adds the correlation matrix according to the displacements in x- cirection and y- directions, and the registration pose is searched in the added matrix. The method overcomes the shortcoming that the probability of registration decreasing with area increasing owing to geometric distortion, improves the probability and the robustness of registration.
文摘This paper describes a new method for three-dimensional medical image registration. In the interactive image-guided HIFU （ High Intensity Focused Ultrasound） therapy system, a fast and precise localization of the tumor is very important. An automatic system is developed for registering pre-operative MR images with intra-operative ultrasound images based on the vessels visible in both of the modalities. When the MR and the ultrasound images are aligned, the eenterline points of the vessels in the MR image will align with bright intensities in the ultrasound image. The method applies an optimization strategy combining the genetic algorithm with the conjugated gradients algorithm to minimize the objective function. It provides a feasible way of determining the global solution and makes the method robust to local maximum and insensitive to initial position. Two experiments were designed to evaluate the method, and the results show that our method has better registration accuracy and convergence rate than the other two classic algorithms.
文摘An automatic image registration approach based on wavelet transform is proposed. This proposed method utilizes multiscale wavelet transform to extract feature points. A coarse-to-fine feature matching method is utilized in the feature matching phase. A two-way matching method based on cross-correlation to get candidate point pairs and a fine matching based on support strength combine to form the matching algorithm. At last, based on an affine transformation model, the parameters are iteratively refined by using the leastsquares estimation approach. Experimental results have verified that the proposed algorithm can realize automatic registration of various kinds of images rapidly and effectively.
基金Supported by National Natural Science Foundation of China (No60373061)Joint Programof National Natural Science Foundation of ChinaGeneral Administration of Civil Aviation of China (No60672168)
文摘An intensity-based non-rigid registration algorithm is discussed, which uses Gaussian smoothing to constrain the transformation to be smooth, and thus preserves the topology of images. In view of the insufficiency of the uniform Gaussian filtering of the deformation field, an automatic and accurate non-rigid image registration method based on B-splines approximation is proposed. The regularization strategy is adopted by using multi-level B-splines approximation to regularize the displacement fields in a coarse-to-fine manner. Moreover, it assigns the different weights to the estimated displacements according to their reliabilities. In this way, the level of regularity can be adapted locally. Experiments were performed on both synthetic and real medical images of brain, and the results show that the proposed method improves the registration accuracy and robustness.
