Mixed reality(MR)technology is a new digital holographic image technology,which appears in the field of graphics after virtual reality(VR)and augmented reality(AR)technology,a new interdisciplinary frontier.As a new g...Mixed reality(MR)technology is a new digital holographic image technology,which appears in the field of graphics after virtual reality(VR)and augmented reality(AR)technology,a new interdisciplinary frontier.As a new generation of technology,MR has attracted great attention of clinicians in recent years.The emergence of MR will bring about revolutionary changes in medical education training,medical research,medical communication,and clinical treatment.At present,MR technology has become the popular frontline information technology for medical applications.With the popularization of digital technology in the medical field,the development prospects of MR are inestimable.The purpose of this review article is to introduce the application of MR technology in the medical field and prospect its trend in the future.展开更多
Traditional teaching and learning about industrial robots uses abstract instructions,which are difficult for students to understand.Meanwhile,there are safety issues associated with the use of practical training equip...Traditional teaching and learning about industrial robots uses abstract instructions,which are difficult for students to understand.Meanwhile,there are safety issues associated with the use of practical training equipment.To address these problems,this paper developed an instructional system based on mixed-reality(MR)technology for teaching about industrial robots.The Siasun T6A-series robots were taken as a case study,and the Microsoft MR device HoloLens-2 was used as the instructional platform.First,the parameters of the robots were analyzed based on their structural drawings.Then,the robot modules were decomposed,and 1:1 three-dimensional(3D)digital reproductions were created in Maya.Next,a library of digital models of the robot components was established,and a 3D spatial operation interface for the virtual instructional system was created in Unity.Subsequently,a C#code framework was established to satisfy the requirements of interactive functions and data transmission,and the data were saved in JSON format.In this way,a key technique that facilitates the understanding of spatial structures and a variety of human-machine interactions were realized.Finally,an instructional system based on HoloLens-2 was established for understanding the structures and principles of robots.The results showed that the instructional system developed in this study provides realistic 3D visualizations and a natural,efficient approach for human-machine interactions.This system could effectively improve the efficiency of knowledge transfer and the student’s motivation to learn.展开更多
Telemedicine includes remote teleradiology,remote ultrasound diagnostics,telesurgery,telemedicine consultation,and other forms.Telemedicine consultation is the most used form of telemedicine.However,the traditional te...Telemedicine includes remote teleradiology,remote ultrasound diagnostics,telesurgery,telemedicine consultation,and other forms.Telemedicine consultation is the most used form of telemedicine.However,the traditional telemedicine consultation is limited by a lack of communication and presentation methods,and its wide application is greatly limited.Mixed reality technology cuts through the boundaries between virtual reality and actual reality,bringing a new method of remote consultation.展开更多
Recently the Mixed Reality (MR) technology has a rapid development and strong application prospect in various fields. This technology has been successfully applied in clinical work by means of surgical navigation and ...Recently the Mixed Reality (MR) technology has a rapid development and strong application prospect in various fields. This technology has been successfully applied in clinical work by means of surgical navigation and puncture positioning. However, there is few reports about MR technology’s applications in China. Therefore, based on the MR technology, a medical education platform was designed for related professions through the reconstruction of 3D heart model. Because of the various types of congenital heart diseases and the comprehensive medical knowledge system involved, it will start from congenital heart disease and expand to other organs later. The teaching mode was enriched and basic and clinical teaching materials were provided. And students’ interests were motivated and learning efficiency was strengthened. An equal communication mechanism between teachers and students was constructed. Experimental results show that both teachers and students benefit from this proposed platform.展开更多
Augmented-and mixed-reality technologies have pioneered the realization of real-time fusion and interactive projection for laparoscopic surgeries.Indocyanine green fluorescence imaging technology has enabled anatomica...Augmented-and mixed-reality technologies have pioneered the realization of real-time fusion and interactive projection for laparoscopic surgeries.Indocyanine green fluorescence imaging technology has enabled anatomical,functional,and radical hepatectomy through tumor identification and localization of target hepatic segments,driving a transformative shift in themanagement of hepatic surgical diseases,moving away from traditional,empirical diagnostic and treatment approaches toward digital,intelligent ones.The Hepatic Surgery Group of the Surgery Branch of the Chinese Medical Association,Digital Medicine Branch of the Chinese Medical Association,Digital Intelligent Surgery Committee of the Chinese Society of ResearchHospitals,and Liver Cancer Committee of the Chinese Medical Doctor Association organized the relevant experts in China to formulate this consensus.This consensus provides a comprehensive outline of the principles,advantages,processes,and key considerations associated with the application of augmented reality and mixed-reality technology combined with indocyanine green fluorescence imaging technology for hepatic segmental and subsegmental resection.The purpose is to streamline and standardize the application of these technologies.展开更多
The mixed reality conference system proposed in this paper is a robust,real-time video conference application software that makes up for the simple interaction and lack of immersion and realism of traditional video co...The mixed reality conference system proposed in this paper is a robust,real-time video conference application software that makes up for the simple interaction and lack of immersion and realism of traditional video conference,which realizes the entire process of holographic video conference from client to cloud to the client.This paper mainly focuses on designing and implementing a video conference system based on AI segmentation technology and mixed reality.Several mixed reality conference system components are discussed,including data collection,data transmission,processing,and mixed reality presentation.The data layer is mainly used for data collection,integration,and video and audio codecs.The network layer uses Web-RTC to realize peer-to-peer data communication.The data processing layer is the core part of the system,mainly for human video matting and human-computer interaction,which is the key to realizing mixed reality conferences and improving the interactive experience.The presentation layer explicitly includes the login interface of the mixed reality conference system,the presentation of real-time matting of human subjects,and the presentation objects.With the mixed reality conference system,conference participants in different places can see each other in real-time in their mixed reality scene and share presentation content and 3D models based on mixed reality technology to have a more interactive and immersive experience.展开更多
A concurrency control mechanism for collaborative work is akey element in a mixed reality environment. However, conventional lockingmechanisms restrict potential tasks or the support of non-owners, thusincreasing the ...A concurrency control mechanism for collaborative work is akey element in a mixed reality environment. However, conventional lockingmechanisms restrict potential tasks or the support of non-owners, thusincreasing the working time because of waiting to avoid conflicts. Herein, wepropose an adaptive concurrency control approach that can reduce conflictsand work time. We classify shared object manipulation in mixed reality intodetailed goals and tasks. Then, we model the relationships among goal,task, and ownership. As the collaborative work progresses, the proposedsystem adapts the different concurrency control mechanisms of shared objectmanipulation according to the modeling of goal–task–ownership. With theproposed concurrency control scheme, users can hold shared objects andmove and rotate together in a mixed reality environment similar to realindustrial sites. Additionally, this system provides MS Hololens and Myosensors to recognize inputs from a user and provides results in a mixed realityenvironment. The proposed method is applied to install an air conditioneras a case study. Experimental results and user studies show that, comparedwith the conventional approach, the proposed method reduced the number ofconflicts, waiting time, and total working time.展开更多
Background This work aims to provide an overview of the Mixed Reality(MR)technology’s use in maritime industry for training purposes.Current training procedures cover a broad range of procedural operations for Life-S...Background This work aims to provide an overview of the Mixed Reality(MR)technology’s use in maritime industry for training purposes.Current training procedures cover a broad range of procedural operations for Life-Saving Appliances(LSA)lifeboats;however,several gaps and limitations have been identified related to the practical training that can be addressed through the use of MR.Augmented,Virtual and Mixed Reality applications are already used in various fields in maritime industry,but their full potential have not been yet exploited.SafePASS project aims to exploit MR advantages in the maritime training by introducing a relevant application focusing on use and maintenance of LSA lifeboats.Methods An MR Training application is proposed supporting the training of crew members in equipment usage and operation,as well as in maintenance activities and procedures.The application consists of the training tool that trains crew members on handling lifeboats,the training evaluation tool that allows trainers to assess the performance of trainees,and the maintenance tool that supports crew members to perform maintenance activities and procedures on lifeboats.For each tool,an indicative session and scenario workflow are implemented,along with the main supported interactions of the trainee with the equipment.Results The application has been tested and validated both in lab environment and using a real LSA lifeboat,resulting to improved experience for the users that provided feedback and recommendations for further development.The application has also been demonstrated onboard a cruise ship,showcasing the supported functionalities to relevant stakeholders that recognized the added value of the application and suggested potential future exploitation areas.Conclusions The MR Training application has been evaluated as very promising in providing a user-friendly training environment that can support crew members in LSA lifeboat operation and maintenance,while it is still subject to improvement and further expansion.展开更多
Background Mixed reality(MR)video fusion systems merge video imagery with 3D scenes to make the scene more realistic and help users understand the video content and temporal–spatial correlation between them,reducing ...Background Mixed reality(MR)video fusion systems merge video imagery with 3D scenes to make the scene more realistic and help users understand the video content and temporal–spatial correlation between them,reducing the user′s cognitive load.MR video fusion are used in various applications;however,video fusion systems require powerful client machines because video streaming delivery,stitching,and rendering are computationally intensive.Moreover,huge bandwidth usage is another critical factor that affects the scalability of video-fusion systems.Methods Our framework proposes a fusion method for dynamically projecting video images into 3D models as textures.Results Several experiments on different metrics demonstrate the effectiveness of the proposed framework.Conclusions The framework proposed in this study can overcome client limitations by utilizing remote rendering.Furthermore,the framework we built is based on browsers.Therefore,the user can test the MR video fusion system with a laptop or tablet without installing any additional plug-ins or application programs.展开更多
Due to the narrowness of space and the complexity of structure, the assembly of aircraft cabin has become one of the major bottlenecks in the whole manufacturing process. To solve the problem, at the beginning of airc...Due to the narrowness of space and the complexity of structure, the assembly of aircraft cabin has become one of the major bottlenecks in the whole manufacturing process. To solve the problem, at the beginning of aircraft design, the different stages of the lifecycle of aircraft must be thought about, which include the trial manufacture, assembly, maintenance, recycling and destruction of the product. Recently, thanks to the development of the virtual reality and augmented reality, some low-cost and fast solutions are found for the product assembly. This paper presents a mixed reality-based interactive technology for the aircraft cabin assembly, which can enhance the efficiency of the assemblage in a virtual environment in terms of vision, information and operation. In the mixed reality-based assembly environment, the physical scene can be obtained by a camera and then generated by a computer. The virtual parts, the features of visual assembly, the navigation information, the physical parts and the physical assembly environment will be mixed and presented in the same assembly scene. The mixed or the augmented information will provide some assembling information as a detailed assembly instruction in the mixed reality-based assembly environment. Constraint proxy and its match rules help to reconstruct and visualize the restriction relationship among different parts, and to avoid the complex calculation of constraint's match. Finally, a desktop prototype system of virtual assembly has been built to assist the assembly verification and training with the virtual hand.展开更多
Background A task assigned to space exploration satellites involves detecting the physical environment within a certain space.However,space detection data are complex and abstract.These data are not conducive for rese...Background A task assigned to space exploration satellites involves detecting the physical environment within a certain space.However,space detection data are complex and abstract.These data are not conducive for researchers'visual perceptions of the evolution and interaction of events in the space environment.Methods A time-series dynamic data sampling method for large-scale space was proposed for sample detection data in space and time,and the corresponding relationships between data location features and other attribute features were established.A tone-mapping method based on statistical histogram equalization was proposed and applied to the final attribute feature data.The visualization process is optimized for rendering by merging materials,reducing the number of patches,and performing other operations.Results The results of sampling,feature extraction,and uniform visualization of the detection data of complex types,long duration spans,and uneven spatial distributions were obtained.The real-time visualization of large-scale spatial structures using augmented reality devices,particularly low-performance devices,was also investigated.Conclusions The proposed visualization system can reconstruct the three-dimensional structure of a large-scale space,express the structure and changes in the spatial environment using augmented reality,and assist in intuitively discovering spatial environmental events and evolutionary rules.展开更多
This paper explores the transformative impact of virtual worlds, augmented reality (AR), and the metaverse in the healthcare sector. It delves into the ways these technologies are reshaping patient care, medical educa...This paper explores the transformative impact of virtual worlds, augmented reality (AR), and the metaverse in the healthcare sector. It delves into the ways these technologies are reshaping patient care, medical education, and research, while also addressing the challenges and opportunities they present. The paper highlights the potential benefits of these technologies and emphasizes the need for comprehensive regulatory frameworks and ethical guidelines to ensure responsible integration. Finally it outlines their transformative impact and discusses the challenges and opportunities they present for the future of healthcare provision.展开更多
Background: Augmented reality(AR) technology is used to reconstruct three-dimensional(3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the v...Background: Augmented reality(AR) technology is used to reconstruct three-dimensional(3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.Data Sources: The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the Pub Med database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles.Results: In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery,which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology.Conclusions: With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling,and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.展开更多
In the modern era,preoperative planning is substantially facilitated by artificial reality technologies,which permit a better understanding of patient anatomy,thus increasing the safety and accuracy of surgical interv...In the modern era,preoperative planning is substantially facilitated by artificial reality technologies,which permit a better understanding of patient anatomy,thus increasing the safety and accuracy of surgical interventions.In the field of orthopedic surgery,the increase in safety and accuracy improves treatment quality and orthopedic patient outcomes.Artificial reality technologies,which include virtual reality(VR),augmented reality(AR),and mixed reality(MR),use digital images obtained from computed tomography or magnetic resonance imaging.VR replaces the user’s physical environment with one that is computer generated.AR and MR have been defined as technologies that permit the fusing of the physical with the virtual environment,enabling the user to interact with both physical and virtual objects.MR has been defined as a technology that,in contrast to AR,enables users to visualize the depth and perspective of the virtual models.We aimed to shed light on the role that MR can play in the visualization of orthopedic surgical anatomy.The literature suggests that MR could be a valuable tool in orthopedic surgeon’s hands for visualization of the anatomy.However,we remark that confusion exists in the literature concerning the characteristics of MR.Thus,a more clear description of MR is needed in orthopedic research,so that the potential of this technology can be more deeply understood.展开更多
The paper elaborates the features and superiority of applying multimedia technology to aquatic organisms in a Science and Life Technology course. The augmented reality application of multimedia technology in aquatic o...The paper elaborates the features and superiority of applying multimedia technology to aquatic organisms in a Science and Life Technology course. The augmented reality application of multimedia technology in aquatic organisms’ instruction has its own features, teaching information capacity, interactive user interface. The paper concludes the superiority of Unity and Vuforia AR instruction from five perspectives;meanwhile, the paper elaborates its disadvantages. An AR-based instruction attitude questionnaire was conducted at the end of the experiment to obtain learners’ perspectives of the system. Learners in AR approach demonstrate higher motivation and concentrate their attention on the learning tasks. Learners showed more positive attitude in willingness to use the AR-based instruction to improve learning interests. This study provided insights for better understanding the design, theory and practice of e-learning through augmented reality technology.展开更多
BACKGROUND As a new digital holographic imaging technology,mixed reality(MR)technology has unique advantages in determining the liver anatomy and location of tumor lesions.With the popularization of 5 G communication ...BACKGROUND As a new digital holographic imaging technology,mixed reality(MR)technology has unique advantages in determining the liver anatomy and location of tumor lesions.With the popularization of 5 G communication technology,MR shows great potential in preoperative planning and intraoperative navigation,making hepatectomy more accurate and safer.AIM To evaluate the application value of MR technology in hepatectomy for hepatocellular carcinoma(HCC).METHODS The clinical data of 95 patients who underwent open hepatectomy surgery for HCC between June 2018 and October 2020 at our hospital were analyzed retrospectively.We selected 95 patients with HCC according to the inclusion criteria and exclusion criteria.In 38 patients,hepatectomy was assisted by MR(Group A),and an additional 57 patients underwent traditional hepatectomy without MR(Group B).The perioperative outcomes of the two groups were collected and compared to evaluate the application value of MR in hepatectomy for patients with HCC.RESULTS We summarized the technical process of MR-assisted hepatectomy in the treatment of HCC.Compared to traditional hepatectomy in Group B,MR-assisted hepatectomy in Group A yielded a shorter operation time(202.86±46.02 min vs 229.52±57.13 min,P=0.003),less volume of bleeding(329.29±97.31 mL vs 398.23±159.61 mL,P=0.028),and shorter obstructive time of the portal vein(17.71±4.16 min vs 21.58±5.24 min,P=0.019).Group A had lower alanine aminotransferas and higher albumin values on the third day after the operation(119.74±29.08 U/L vs 135.53±36.68 U/L,P=0.029 and 33.60±3.21 g/L vs 31.80±3.51 g/L,P=0.014,respectively).The total postoperative complications and hospitalization days in Group A were significantly less than those in Group B[14(37.84%)vs 35(60.34%),P=0.032 and 12.05±4.04 d vs 13.78±4.13 d,P=0.049,respectively].CONCLUSION MR has some application value in three-dimensional visualization of the liver,surgical planning,and intraoperative navigation during hepatectomy,and it significantly improves the perioperative outcomes of hepatectomy for HCC.展开更多
Pulmonary embolism(PE)is one of the fatal heart attacks,and lower limbs deep vein thrombosis(DVT)is the most common reason for PE.Inferior vena cava filter(IVCF)implantation is a most prevention for PE.But it may carr...Pulmonary embolism(PE)is one of the fatal heart attacks,and lower limbs deep vein thrombosis(DVT)is the most common reason for PE.Inferior vena cava filter(IVCF)implantation is a most prevention for PE.But it may carry a high risk of injury because of the radiation and contrast agent.Patients with nephrotic syndrome(NS)or some other renal diseases may prone to thrombosis due to the excretion of protein C and protein S overmuch.So,it is necessary to develop a new therapy without contrast agent.Mixed-reality(MR)is a new technology as a guidance of inferior vena cava filter implantation exposed under no X-ray and required no contrast agent.展开更多
The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery s...The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery simulation and using the created scenarios in real-time surgery using mixed reality.In this article,we described our experience on developing a dedicated 3 dimensional visualization and reconstruction software for surgeons to be used in advanced liver surgery and living donor liver transplantation.Furthermore,we shared the recent developments in the field by explaining the outreach of the software from virtual reality to augmented reality and mixed reality.展开更多
With the rapid development of holographic imaging technology,virtual reality and augmented reality technology have been used in medicine for decades.However,due to the lack of compatibility between the virtual world a...With the rapid development of holographic imaging technology,virtual reality and augmented reality technology have been used in medicine for decades.However,due to the lack of compatibility between the virtual world and the real world and the lack of human-computer interaction,their widespread use is limited.As a research hotspot in recent years,the mixed reality technology has a huge prospect in the medical field by virtue of its unique advantages combined with the virtual world and the real world,as well as better human-machine realtime interaction and precise matching.This article will focus on the development of this technology in recent years,and introduce the application of mixed reality technology in medical education and clinical application.展开更多
To improve and develop education systems,the communication between instructors and learners in a class during the learning process is of utmost importance.Currently the presentations of 3D models using mixed reality(M...To improve and develop education systems,the communication between instructors and learners in a class during the learning process is of utmost importance.Currently the presentations of 3D models using mixed reality(MR)technology can be used to avoid misinterpretations of oral and 2D model presentations.As an independent concept and MR applications,MR combines the excellent of each virtual reality(VR)and augmented reality(AR).This work aims to present the descriptions of MR systems,which include its devices,applications,and literature reviews and proposes computer vision tracking using the AR Toolkit Tracking Library.The focus of this work will be on creating 3D models and implementing in Unity 3D using the Vuforia SDK platform to develop VR and AR applications for architectural presentations.展开更多
文摘Mixed reality(MR)technology is a new digital holographic image technology,which appears in the field of graphics after virtual reality(VR)and augmented reality(AR)technology,a new interdisciplinary frontier.As a new generation of technology,MR has attracted great attention of clinicians in recent years.The emergence of MR will bring about revolutionary changes in medical education training,medical research,medical communication,and clinical treatment.At present,MR technology has become the popular frontline information technology for medical applications.With the popularization of digital technology in the medical field,the development prospects of MR are inestimable.The purpose of this review article is to introduce the application of MR technology in the medical field and prospect its trend in the future.
文摘Traditional teaching and learning about industrial robots uses abstract instructions,which are difficult for students to understand.Meanwhile,there are safety issues associated with the use of practical training equipment.To address these problems,this paper developed an instructional system based on mixed-reality(MR)technology for teaching about industrial robots.The Siasun T6A-series robots were taken as a case study,and the Microsoft MR device HoloLens-2 was used as the instructional platform.First,the parameters of the robots were analyzed based on their structural drawings.Then,the robot modules were decomposed,and 1:1 three-dimensional(3D)digital reproductions were created in Maya.Next,a library of digital models of the robot components was established,and a 3D spatial operation interface for the virtual instructional system was created in Unity.Subsequently,a C#code framework was established to satisfy the requirements of interactive functions and data transmission,and the data were saved in JSON format.In this way,a key technique that facilitates the understanding of spatial structures and a variety of human-machine interactions were realized.Finally,an instructional system based on HoloLens-2 was established for understanding the structures and principles of robots.The results showed that the instructional system developed in this study provides realistic 3D visualizations and a natural,efficient approach for human-machine interactions.This system could effectively improve the efficiency of knowledge transfer and the student’s motivation to learn.
文摘Telemedicine includes remote teleradiology,remote ultrasound diagnostics,telesurgery,telemedicine consultation,and other forms.Telemedicine consultation is the most used form of telemedicine.However,the traditional telemedicine consultation is limited by a lack of communication and presentation methods,and its wide application is greatly limited.Mixed reality technology cuts through the boundaries between virtual reality and actual reality,bringing a new method of remote consultation.
文摘Recently the Mixed Reality (MR) technology has a rapid development and strong application prospect in various fields. This technology has been successfully applied in clinical work by means of surgical navigation and puncture positioning. However, there is few reports about MR technology’s applications in China. Therefore, based on the MR technology, a medical education platform was designed for related professions through the reconstruction of 3D heart model. Because of the various types of congenital heart diseases and the comprehensive medical knowledge system involved, it will start from congenital heart disease and expand to other organs later. The teaching mode was enriched and basic and clinical teaching materials were provided. And students’ interests were motivated and learning efficiency was strengthened. An equal communication mechanism between teachers and students was constructed. Experimental results show that both teachers and students benefit from this proposed platform.
基金National Key Research and Development Program(2016YFC0106500800)NationalMajor Scientific Instruments and Equipments Development Project of National Natural Science Foundation of China(81627805)+3 种基金National Natural Science Foundation of China-Guangdong Joint Fund Key Program(U1401254)National Natural Science Foundation of China Mathematics Tianyuan Foundation(12026602)Guangdong Provincial Natural Science Foundation Team Project(6200171)Guangdong Provincial Health Appropriate Technology Promotion Project(20230319214525105,20230322152307666).
文摘Augmented-and mixed-reality technologies have pioneered the realization of real-time fusion and interactive projection for laparoscopic surgeries.Indocyanine green fluorescence imaging technology has enabled anatomical,functional,and radical hepatectomy through tumor identification and localization of target hepatic segments,driving a transformative shift in themanagement of hepatic surgical diseases,moving away from traditional,empirical diagnostic and treatment approaches toward digital,intelligent ones.The Hepatic Surgery Group of the Surgery Branch of the Chinese Medical Association,Digital Medicine Branch of the Chinese Medical Association,Digital Intelligent Surgery Committee of the Chinese Society of ResearchHospitals,and Liver Cancer Committee of the Chinese Medical Doctor Association organized the relevant experts in China to formulate this consensus.This consensus provides a comprehensive outline of the principles,advantages,processes,and key considerations associated with the application of augmented reality and mixed-reality technology combined with indocyanine green fluorescence imaging technology for hepatic segmental and subsegmental resection.The purpose is to streamline and standardize the application of these technologies.
基金supported in part by the Major Fundamental Research of Natural Science Foundation of Shandong Province under Grant ZR2019ZD05Joint fund for smart computing of Shandong Natural Science Foundation under Grant ZR2020LZH013+1 种基金Open project of State Key Laboratory of Computer Architecture CARCHA202002Human Video Matting Project of Hisense Co.,Ltd.under Grant QD1170020023.
文摘The mixed reality conference system proposed in this paper is a robust,real-time video conference application software that makes up for the simple interaction and lack of immersion and realism of traditional video conference,which realizes the entire process of holographic video conference from client to cloud to the client.This paper mainly focuses on designing and implementing a video conference system based on AI segmentation technology and mixed reality.Several mixed reality conference system components are discussed,including data collection,data transmission,processing,and mixed reality presentation.The data layer is mainly used for data collection,integration,and video and audio codecs.The network layer uses Web-RTC to realize peer-to-peer data communication.The data processing layer is the core part of the system,mainly for human video matting and human-computer interaction,which is the key to realizing mixed reality conferences and improving the interactive experience.The presentation layer explicitly includes the login interface of the mixed reality conference system,the presentation of real-time matting of human subjects,and the presentation objects.With the mixed reality conference system,conference participants in different places can see each other in real-time in their mixed reality scene and share presentation content and 3D models based on mixed reality technology to have a more interactive and immersive experience.
基金supported by“Regional Innovation Strategy (RIS)”through the National Research Foundation of Korea (NRF)funded by the Ministry of Education (MOE) (2021RIS-004).
文摘A concurrency control mechanism for collaborative work is akey element in a mixed reality environment. However, conventional lockingmechanisms restrict potential tasks or the support of non-owners, thusincreasing the working time because of waiting to avoid conflicts. Herein, wepropose an adaptive concurrency control approach that can reduce conflictsand work time. We classify shared object manipulation in mixed reality intodetailed goals and tasks. Then, we model the relationships among goal,task, and ownership. As the collaborative work progresses, the proposedsystem adapts the different concurrency control mechanisms of shared objectmanipulation according to the modeling of goal–task–ownership. With theproposed concurrency control scheme, users can hold shared objects andmove and rotate together in a mixed reality environment similar to realindustrial sites. Additionally, this system provides MS Hololens and Myosensors to recognize inputs from a user and provides results in a mixed realityenvironment. The proposed method is applied to install an air conditioneras a case study. Experimental results and user studies show that, comparedwith the conventional approach, the proposed method reduced the number ofconflicts, waiting time, and total working time.
基金Supported by the Safe PASS project that has received funding from the European Union’s Horizon 2020 Research and Innovation programme (815146)。
文摘Background This work aims to provide an overview of the Mixed Reality(MR)technology’s use in maritime industry for training purposes.Current training procedures cover a broad range of procedural operations for Life-Saving Appliances(LSA)lifeboats;however,several gaps and limitations have been identified related to the practical training that can be addressed through the use of MR.Augmented,Virtual and Mixed Reality applications are already used in various fields in maritime industry,but their full potential have not been yet exploited.SafePASS project aims to exploit MR advantages in the maritime training by introducing a relevant application focusing on use and maintenance of LSA lifeboats.Methods An MR Training application is proposed supporting the training of crew members in equipment usage and operation,as well as in maintenance activities and procedures.The application consists of the training tool that trains crew members on handling lifeboats,the training evaluation tool that allows trainers to assess the performance of trainees,and the maintenance tool that supports crew members to perform maintenance activities and procedures on lifeboats.For each tool,an indicative session and scenario workflow are implemented,along with the main supported interactions of the trainee with the equipment.Results The application has been tested and validated both in lab environment and using a real LSA lifeboat,resulting to improved experience for the users that provided feedback and recommendations for further development.The application has also been demonstrated onboard a cruise ship,showcasing the supported functionalities to relevant stakeholders that recognized the added value of the application and suggested potential future exploitation areas.Conclusions The MR Training application has been evaluated as very promising in providing a user-friendly training environment that can support crew members in LSA lifeboat operation and maintenance,while it is still subject to improvement and further expansion.
基金Supported by the National Key R&D Program of China(2018YFB2100601)National Natural Science Foundation of China(61872024)。
文摘Background Mixed reality(MR)video fusion systems merge video imagery with 3D scenes to make the scene more realistic and help users understand the video content and temporal–spatial correlation between them,reducing the user′s cognitive load.MR video fusion are used in various applications;however,video fusion systems require powerful client machines because video streaming delivery,stitching,and rendering are computationally intensive.Moreover,huge bandwidth usage is another critical factor that affects the scalability of video-fusion systems.Methods Our framework proposes a fusion method for dynamically projecting video images into 3D models as textures.Results Several experiments on different metrics demonstrate the effectiveness of the proposed framework.Conclusions The framework proposed in this study can overcome client limitations by utilizing remote rendering.Furthermore,the framework we built is based on browsers.Therefore,the user can test the MR video fusion system with a laptop or tablet without installing any additional plug-ins or application programs.
基金supported by National Defence Basic Research Foundation of China (Grant No. B1420060173)National Hi-tech Research and Development Program of China (863 Program, Grant No. 2006AA04Z138)
文摘Due to the narrowness of space and the complexity of structure, the assembly of aircraft cabin has become one of the major bottlenecks in the whole manufacturing process. To solve the problem, at the beginning of aircraft design, the different stages of the lifecycle of aircraft must be thought about, which include the trial manufacture, assembly, maintenance, recycling and destruction of the product. Recently, thanks to the development of the virtual reality and augmented reality, some low-cost and fast solutions are found for the product assembly. This paper presents a mixed reality-based interactive technology for the aircraft cabin assembly, which can enhance the efficiency of the assemblage in a virtual environment in terms of vision, information and operation. In the mixed reality-based assembly environment, the physical scene can be obtained by a camera and then generated by a computer. The virtual parts, the features of visual assembly, the navigation information, the physical parts and the physical assembly environment will be mixed and presented in the same assembly scene. The mixed or the augmented information will provide some assembling information as a detailed assembly instruction in the mixed reality-based assembly environment. Constraint proxy and its match rules help to reconstruct and visualize the restriction relationship among different parts, and to avoid the complex calculation of constraint's match. Finally, a desktop prototype system of virtual assembly has been built to assist the assembly verification and training with the virtual hand.
文摘Background A task assigned to space exploration satellites involves detecting the physical environment within a certain space.However,space detection data are complex and abstract.These data are not conducive for researchers'visual perceptions of the evolution and interaction of events in the space environment.Methods A time-series dynamic data sampling method for large-scale space was proposed for sample detection data in space and time,and the corresponding relationships between data location features and other attribute features were established.A tone-mapping method based on statistical histogram equalization was proposed and applied to the final attribute feature data.The visualization process is optimized for rendering by merging materials,reducing the number of patches,and performing other operations.Results The results of sampling,feature extraction,and uniform visualization of the detection data of complex types,long duration spans,and uneven spatial distributions were obtained.The real-time visualization of large-scale spatial structures using augmented reality devices,particularly low-performance devices,was also investigated.Conclusions The proposed visualization system can reconstruct the three-dimensional structure of a large-scale space,express the structure and changes in the spatial environment using augmented reality,and assist in intuitively discovering spatial environmental events and evolutionary rules.
文摘This paper explores the transformative impact of virtual worlds, augmented reality (AR), and the metaverse in the healthcare sector. It delves into the ways these technologies are reshaping patient care, medical education, and research, while also addressing the challenges and opportunities they present. The paper highlights the potential benefits of these technologies and emphasizes the need for comprehensive regulatory frameworks and ethical guidelines to ensure responsible integration. Finally it outlines their transformative impact and discusses the challenges and opportunities they present for the future of healthcare provision.
基金supported by grants from the Mission Plan Program of Beijing Municipal Administration of Hospitals(SML20152201)Beijing Municipal Administration of Hospitals Clinical Medicine Development of Special Funding(ZYLX201712)+1 种基金the National Natural Science Foundation of China(81427803)Beijing Tsinghua Changgung Hospital Fund(12015C1039)
文摘Background: Augmented reality(AR) technology is used to reconstruct three-dimensional(3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.Data Sources: The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the Pub Med database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles.Results: In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery,which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology.Conclusions: With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling,and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.
文摘In the modern era,preoperative planning is substantially facilitated by artificial reality technologies,which permit a better understanding of patient anatomy,thus increasing the safety and accuracy of surgical interventions.In the field of orthopedic surgery,the increase in safety and accuracy improves treatment quality and orthopedic patient outcomes.Artificial reality technologies,which include virtual reality(VR),augmented reality(AR),and mixed reality(MR),use digital images obtained from computed tomography or magnetic resonance imaging.VR replaces the user’s physical environment with one that is computer generated.AR and MR have been defined as technologies that permit the fusing of the physical with the virtual environment,enabling the user to interact with both physical and virtual objects.MR has been defined as a technology that,in contrast to AR,enables users to visualize the depth and perspective of the virtual models.We aimed to shed light on the role that MR can play in the visualization of orthopedic surgical anatomy.The literature suggests that MR could be a valuable tool in orthopedic surgeon’s hands for visualization of the anatomy.However,we remark that confusion exists in the literature concerning the characteristics of MR.Thus,a more clear description of MR is needed in orthopedic research,so that the potential of this technology can be more deeply understood.
文摘The paper elaborates the features and superiority of applying multimedia technology to aquatic organisms in a Science and Life Technology course. The augmented reality application of multimedia technology in aquatic organisms’ instruction has its own features, teaching information capacity, interactive user interface. The paper concludes the superiority of Unity and Vuforia AR instruction from five perspectives;meanwhile, the paper elaborates its disadvantages. An AR-based instruction attitude questionnaire was conducted at the end of the experiment to obtain learners’ perspectives of the system. Learners in AR approach demonstrate higher motivation and concentrate their attention on the learning tasks. Learners showed more positive attitude in willingness to use the AR-based instruction to improve learning interests. This study provided insights for better understanding the design, theory and practice of e-learning through augmented reality technology.
文摘BACKGROUND As a new digital holographic imaging technology,mixed reality(MR)technology has unique advantages in determining the liver anatomy and location of tumor lesions.With the popularization of 5 G communication technology,MR shows great potential in preoperative planning and intraoperative navigation,making hepatectomy more accurate and safer.AIM To evaluate the application value of MR technology in hepatectomy for hepatocellular carcinoma(HCC).METHODS The clinical data of 95 patients who underwent open hepatectomy surgery for HCC between June 2018 and October 2020 at our hospital were analyzed retrospectively.We selected 95 patients with HCC according to the inclusion criteria and exclusion criteria.In 38 patients,hepatectomy was assisted by MR(Group A),and an additional 57 patients underwent traditional hepatectomy without MR(Group B).The perioperative outcomes of the two groups were collected and compared to evaluate the application value of MR in hepatectomy for patients with HCC.RESULTS We summarized the technical process of MR-assisted hepatectomy in the treatment of HCC.Compared to traditional hepatectomy in Group B,MR-assisted hepatectomy in Group A yielded a shorter operation time(202.86±46.02 min vs 229.52±57.13 min,P=0.003),less volume of bleeding(329.29±97.31 mL vs 398.23±159.61 mL,P=0.028),and shorter obstructive time of the portal vein(17.71±4.16 min vs 21.58±5.24 min,P=0.019).Group A had lower alanine aminotransferas and higher albumin values on the third day after the operation(119.74±29.08 U/L vs 135.53±36.68 U/L,P=0.029 and 33.60±3.21 g/L vs 31.80±3.51 g/L,P=0.014,respectively).The total postoperative complications and hospitalization days in Group A were significantly less than those in Group B[14(37.84%)vs 35(60.34%),P=0.032 and 12.05±4.04 d vs 13.78±4.13 d,P=0.049,respectively].CONCLUSION MR has some application value in three-dimensional visualization of the liver,surgical planning,and intraoperative navigation during hepatectomy,and it significantly improves the perioperative outcomes of hepatectomy for HCC.
基金supported by Capital Clinical Application Research Project(No.Z181100001718042)
文摘Pulmonary embolism(PE)is one of the fatal heart attacks,and lower limbs deep vein thrombosis(DVT)is the most common reason for PE.Inferior vena cava filter(IVCF)implantation is a most prevention for PE.But it may carry a high risk of injury because of the radiation and contrast agent.Patients with nephrotic syndrome(NS)or some other renal diseases may prone to thrombosis due to the excretion of protein C and protein S overmuch.So,it is necessary to develop a new therapy without contrast agent.Mixed-reality(MR)is a new technology as a guidance of inferior vena cava filter implantation exposed under no X-ray and required no contrast agent.
文摘The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery simulation and using the created scenarios in real-time surgery using mixed reality.In this article,we described our experience on developing a dedicated 3 dimensional visualization and reconstruction software for surgeons to be used in advanced liver surgery and living donor liver transplantation.Furthermore,we shared the recent developments in the field by explaining the outreach of the software from virtual reality to augmented reality and mixed reality.
文摘With the rapid development of holographic imaging technology,virtual reality and augmented reality technology have been used in medicine for decades.However,due to the lack of compatibility between the virtual world and the real world and the lack of human-computer interaction,their widespread use is limited.As a research hotspot in recent years,the mixed reality technology has a huge prospect in the medical field by virtue of its unique advantages combined with the virtual world and the real world,as well as better human-machine realtime interaction and precise matching.This article will focus on the development of this technology in recent years,and introduce the application of mixed reality technology in medical education and clinical application.
文摘To improve and develop education systems,the communication between instructors and learners in a class during the learning process is of utmost importance.Currently the presentations of 3D models using mixed reality(MR)technology can be used to avoid misinterpretations of oral and 2D model presentations.As an independent concept and MR applications,MR combines the excellent of each virtual reality(VR)and augmented reality(AR).This work aims to present the descriptions of MR systems,which include its devices,applications,and literature reviews and proposes computer vision tracking using the AR Toolkit Tracking Library.The focus of this work will be on creating 3D models and implementing in Unity 3D using the Vuforia SDK platform to develop VR and AR applications for architectural presentations.
