An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jianxin Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, Zhihao Fan (2024)
The Qwen2 Technical Report presents the Qwen2 series, a range of advanced language models that include dense and Mixture-of-Experts models, ranging from 0.5 to 72 billion parameters. This series significantly enhances functionalities in language understanding, generation, and multilingual communication compared to its predecessor Qwen1.5 and other proprietary models. The flagship model, Qwen2-72B, demonstrates impressive performance across various benchmarks, including achieving scores of 84.2 on MMLU and 89.5 on GSM8K. Additionally, an instruction-tuned variant, Qwen2-72B-Instruct, showcases competitive results on MT-Bench and Arena-Hard. The release aims to promote accessibility via platforms like Hugging Face and GitHub, facilitating community innovation. Key innovations in the model include advancements in the architecture based on Transformer principles and improved pre-training methodologies involving extensive datasets of over 7 trillion tokens, inclusive of multilingual and code-related content. The paper stresses ongoing efforts to mitigate safety and contamination concerns while aiming for responsible AI development.
This paper employs the following methods:
The following datasets were used in this research:
The authors identified the following limitations: