摘要
Brain encoding and decoding via functional magnetic resonance imaging(fMRI)are two important aspects of visual perception neuroscience.Although previous researchers have made significant advances in brain encoding and decoding models,existing methods still require improvement using advanced machine learning techniques.For example,traditional methods usually build the encoding and decoding models separately,and are prone to overfitting on a small dataset.In fact,effectively unifying the encoding and decoding procedures may allow for more accurate predictions.In this paper,we first review the existing encoding and decoding methods and discuss the potential advantages of a“bidirectional”modeling strategy.Next,we show that there are correspondences between deep neural networks and human visual streams in terms of the architecture and computational rules.Furthermore,deep generative models(e.g.,variational autoencoders(VAEs)and generative adversarial networks(GANs))have produced promising results in studies on brain encoding and decoding.Finally,we propose that the dual learning method,which was originally designed for machine translation tasks,could help to improve the performance of encoding and decoding models by leveraging large-scale unpaired data.
Brain encoding and decoding via functional magnetic resonance imaging(fMRI) are two important aspects of visual perception neuroscience.Although previous researchers have made significant advances in brain encoding and decoding models,existing methods still require improvement using advanced machine learning techniques.For example,traditional methods usually build the encoding and decoding models separately,and are prone to overfitting on a small dataset.In fact,effectively unifying the encoding and decoding procedures may allow for more accurate predictions.In this paper,we first review the existing encoding and decoding methods and discuss the potential advantages of a "bidirectional" modeling strategy.Next,we show that there are correspondences between deep neural networks and human visual streams in terms of the architecture and computational rules.Furthermore,deep generative models(e.g.,variational autoencoders(VAEs) and generative adversarial networks(GANs)) have produced promising results in studies on brain encoding and decoding.Finally,we propose that the dual learning method,which was originally designed for machine translation tasks,could help to improve the performance of encoding and decoding models by leveraging large-scale unpaired data.
基金
This work was supported by the National Key Research and Development Program of China(2018YFC2001302)
National Natural Science Foundation of China(91520202)
Chinese Academy of Sciences Scientific Equipment Development Project(YJKYYQ20170050)
Beijing Municipal Science and Technology Commission(Z181100008918010)
Youth Innovation Promotion Association of Chinese Academy of Sciences,and Strategic Priority Research Program of Chinese Academy of Sciences(XDB32040200).
