Conversational large language models(LLMs)such as ChatGPT and GPT-4 have recently exhibited remarkable capabilities across various domains,capturing widespread attention from the public.To facilitate this line of rese...Conversational large language models(LLMs)such as ChatGPT and GPT-4 have recently exhibited remarkable capabilities across various domains,capturing widespread attention from the public.To facilitate this line of research,in this paper,we report the development of MOSS,an open-sourced conversational LLM that contains 16 B parameters and can perform a variety of instructions in multi-turn interactions with humans.The base model of MOSS is pre-trained on large-scale unlabeled English,Chinese,and code data.To optimize the model for dialogue,we generate 1.1 M synthetic conversations based on user prompts collected through our earlier versions of the model API.We then perform preference-aware training on preference data annotated from AI feedback.Evaluation results on real-world use cases and academic benchmarks demonstrate the effectiveness of the proposed approaches.In addition,we present an effective practice to augment MOSS with several external tools.Through the development of MOSS,we have established a complete technical roadmap for large language models from pre-training,supervised fine-tuning to alignment,verifying the feasibility of chatGPT under resource-limited conditions and providing a reference for both the academic and industrial communities.Model weights and code are publicly available at https://github.com/OpenMOSS/MOSS.展开更多
A diffeomorphism f:M→M is pointwise partially hyperbolic on an open invariant subset N⊂M if there is an invariant decomposition TNM=Eu⊕Ec⊕Es such that Dxf is strictly expanding on Eu x and contracting on Esx at eac...A diffeomorphism f:M→M is pointwise partially hyperbolic on an open invariant subset N⊂M if there is an invariant decomposition TNM=Eu⊕Ec⊕Es such that Dxf is strictly expanding on Eu x and contracting on Esx at each x∈N.We show that under certain conditions f has unstable and stable manifolds,and admits a nite or an in nite u-Gibbs measure μ.If f is pointwise hyperbolic on N,then μ is a Sinai-Ruelle-Bowen(SRB)measure or an in nite SRB measure.As applications,we show that some almost Anosov di eomorphisms and gentle perturbations of Katok's map have the properties.展开更多
基金supported by the National Natural Science Foundation of China(No.62022027).
文摘Conversational large language models(LLMs)such as ChatGPT and GPT-4 have recently exhibited remarkable capabilities across various domains,capturing widespread attention from the public.To facilitate this line of research,in this paper,we report the development of MOSS,an open-sourced conversational LLM that contains 16 B parameters and can perform a variety of instructions in multi-turn interactions with humans.The base model of MOSS is pre-trained on large-scale unlabeled English,Chinese,and code data.To optimize the model for dialogue,we generate 1.1 M synthetic conversations based on user prompts collected through our earlier versions of the model API.We then perform preference-aware training on preference data annotated from AI feedback.Evaluation results on real-world use cases and academic benchmarks demonstrate the effectiveness of the proposed approaches.In addition,we present an effective practice to augment MOSS with several external tools.Through the development of MOSS,we have established a complete technical roadmap for large language models from pre-training,supervised fine-tuning to alignment,verifying the feasibility of chatGPT under resource-limited conditions and providing a reference for both the academic and industrial communities.Model weights and code are publicly available at https://github.com/OpenMOSS/MOSS.
基金The third author was supported by National Natural Science Foundation of China(Grant Nos.11871120 and 11671093).
文摘A diffeomorphism f:M→M is pointwise partially hyperbolic on an open invariant subset N⊂M if there is an invariant decomposition TNM=Eu⊕Ec⊕Es such that Dxf is strictly expanding on Eu x and contracting on Esx at each x∈N.We show that under certain conditions f has unstable and stable manifolds,and admits a nite or an in nite u-Gibbs measure μ.If f is pointwise hyperbolic on N,then μ is a Sinai-Ruelle-Bowen(SRB)measure or an in nite SRB measure.As applications,we show that some almost Anosov di eomorphisms and gentle perturbations of Katok's map have the properties.
