上海交大团队研发大模型红队评测框架，能用于提醒大模型的滥用风险

上海交大团队研发大模型红队评测框架，能用于提醒大模型的滥用风险

随着大型语言模型生成能力的增强，它们被滥用的风险和潜在危害也越来越大，例如隐私泄漏、输出有害或偏见性内容等。为了限制模型的有害行为，许多安全对齐技术被陆续提出。然而，这些技术主要基于自然语言数据进行训练，对于输入和输出均为自然语言的情况表现良好。但是，这些安全行为可能无法泛化到新颖的使用场景，比如无法泛化到输入和输出均为非自然语言的情况。

在近期一项研究中，一支国内团队探索了大型语言模型在远离安全训练数据的情况下，其安全对齐面临的泛化挑战。研究中，他们选择代码作为研究对象。因为在当前主流的大型语言模型中，代码被广泛用作训练数据，并且代码与自然语言存在着显著差异。

研究期间，他们提出了一款自动化代码红队评测框架——CodeAttack，该框架能将“文本生成任务”建模为“代码生成任务”，借此评估模型在生成代码时的安全行为。

具体来说，CodeAttack 包括三个模块：

• 输入编码：将自然语言形式的输入，编码为数据结构，比如堆栈或队列，借此生成语义上等价、但是数据分布上存在显著差异的输入。
• 任务理解：即设计一个 decode 函数，使大型语言模型能从第一步得到的结构化输入中提取目标任务。
• 输出规范：引导大型语言模型把对于任务的回答填入数据结构中，以此作为代码程序的输出返回。

（来源：arXiv）

目前，他们正在尝试通过设计安全提示词来激发大模型的安全行为。此外，课题组也正在尝试解释 CodeAttack 成功的原因，以启发更多人开展安全对齐的研究。

参考资料

1. Daniil A. Boiko, Robert MacKnight, and Gabe Gomes. 2023. Emergent autonomous scientific research capabilities of large language models. arXiv, abs/2304.05332.
2. Zheng Qinkai, Xia Xiao, Zou Xu, Dong Yuxiao, Wang Shan, Xue Yufei, Shen Lei, Wang Zihan, Wang Andi, Li Yang, Su Teng, Yang Zhilin, and Tang Jie. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 5673–5684, New York, NY, USA. Association for Computing Machinery.
3. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Ex- tracting training data from large language models. In 30th USENIX Security Symposium（USENIX Security 21）, pages 2633–2650. USENIX Association.
4. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adver- sarial attacks on aligned language models. arXiv, abs/2307.15043.
5. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and et al. 2022. Training language models to fol- low instructions with human feedback. arXiv, abs/2203.02155.
6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training a help- ful and harmless assistant with reinforcement learn- ing from human feedback. arXiv, abs/2204.05862.
7. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, and Anna Goldie et al. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv.
8. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, and et al. 2022. Red teaming language models to re- duce harms: Methods, scaling behaviors, and lessons learned. arXiv,abs/2209.07858.
9. OpenAI. 2024. GPT-4 technical report. https://arXiv.org/abs/2303.08774
10. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. In The Twelfth International Conference on Learning Representations.
11. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? In Neural Information Processing Systems.
12. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv, abs/2307.09288.
13. Anthropic. 2023. Model card and evaluations for claude models. https://www-files.anthropic. com/production/images/Model-Card-Claude-2. pdf.
14. OpenAI. 2023. https://openai.com/chatgpt.