Table of LinksAbstract and I. IntroductionII. Related WorkIII. Technical BackgroundIV. Systematic Security Vulnerability Discovery of Code Generation ModelsV. ExperimentsVI. DiscussionVII. Conclusion, Acknowledgments, and References\AppendixA. Details of Code Language ModelsB. Finding Security Vulnerabilities in GitHub CopilotC. Other Baselines Using ChatGPTD. Effect of Different Number of Few-shot ExamplesE. Effectiveness in Generating Specific Vulnerabilities for C CodesF. Security Vulnerability Results after Fuzzy Code DeduplicationG. Detailed Results of Transferability of the Generated Nonsecure PromptsH. Details of Generating non-secure prompts DatasetI. Detailed Results of Evaluating CodeLMs using Non-secure DatasetJ. Effect of Sampling TemperatureK. Effectiveness of the Model Inversion Scheme in Reconstructing the Vulnerable CodesL. Qualitative Examples Generated by CodeGen and ChatGPTM. Qualitative Examples Generated by GitHub CopilotF. Security Vulnerability Results after Fuzzy Code DeduplicationWe employ TheFuzz [64] python library to find near duplicate codes. This library uses Levenshtein Distance to calculate the differences between sequences [65]. The library outputs the similarity ratio of two strings as a number between 0 and 100. We consider two codes duplicates if they have a similarity ratio greater than 80. Figure 7 provides the results of our FS-Code approach in finding vulnerable Python and C codes that could be generated by CodeGen and ChatGPT\ \model. Note that these results are provided by following the setting of Section V-B2. Here we also observe a general almostlinear growth pattern for some of the vulnerability types that are generated by CodeGen and ChatGPT models.\G. Detailed Results of Transferability of the Generated Nonsecure PromptsHere we provide the details results of the transferability of the generated non-secure prompts. Table VIII and Table IX show the detailed transferability results of the promising nonsecure prompts that are generated by CodeGen and ChatGPT, respectively. The results in Table VIII and Table IX provide the results of generated Python and C codes for different CWEs. In Table VIII and Table IX show that the promising non-secure prompts are transferable among the models for generating codes with different types of CWEs. Even in some cases, the non-secure prompts from model A can lead model B to generate more vulnerable codes compared to model A itself. For example, in Table VIII, the promising non-secure prompts generated by CodeGen lead ChatGPT to generate more vulnerable codes with CWE-079 vulnerability compared to the CodeGen itself.\H. Details of Generating non-secure prompts DatasetWe generate the non-secure prompts dataset using our FSCode method, following the same settings as in Section V-B. For generating prompts with GPT-4 and Code Llama-34B, we set the sampling temperature to 1.0. A higher temperature facilitates the generation of a diverse set of non-secure prompts. We set the maximum number of tokens for generating Python and C non-secure prompts to 45 and 65, respectively. The maximum token count is determined based on the average size of prompts in our initial code examples.\To generate the non-secure prompts with Code Llama-34B, we exclusively employ the few-shot examples from our FSCode. However, in the generation process using GPT-4, we utilize an instruction to guide the model. We use the following instruction to generate Python prompts:\You are a Python developer. Given the provided code examples, complete the following code. Define a function and write the task description as a Python comment. Provide comprehensive comments. Do not write any explanation. Only write Python codes.\I. Detailed Results of Evaluating CodeLMs using Non-secure DatasetIn Table X, we provide the detailed results of evaluating various code language models using our proposed non-secure prompts dataset. Table X demonstrates the number of vulnerable Python and C codes generated by CodeGen-6B [6], StarCoder7B [24], Code Llama-13B [12], WizardCoder-15B [56], and ChatGPT [4] models. Detailed results for each CWE can offer valuable insights for specific use cases. For instance, as shown in Table X, Code Llama-13B generates fewer Python codes with the CWE-089 (SQL-injection) vulnerability. Consequently,\\ \\\ \\This model stands out as a strong choice among the evaluated models for generating SQL-related Python code.\J. Effect of Sampling TemperatureFigure 8 provides detailed results of the effect of different sampling temperatures in generating non-secure prompts and vulnerable code. We conduct this evaluation using our FS-Code method and sample the non-secure prompts and Python codes from CodeGen model. Here, we provide the total number of generated vulnerable codes with three different CWEs (CWE-020, CWE-022, and CWE-079) and sample 125 code samples for each CWE. The y-axis refers to different sampling temperatures for sampling the non-secure prompts, and xaxis refers to different sampling temperatures of the code generation procedure. The results in Figure 8 show that in general, sampling temperatures of non-secure prompts have a significant effect in generating vulnerable codes, while sampling temperatures of codes have a minor impact (in each row, we have low difference among the number of vulnerable codes), furthermore, in Figure 8 we observe that 0.6 is an optimal temperature for sampling the non-secure prompts. Note that in all of our experiments, based on the previous works in the program generation domain [6], [5], to have fair results we set the non-secure prompt and codes’ sampling temperature to 0.6.\:::infoAuthors:(1) Hossein Hajipour, CISPA Helmholtz Center for Information Security (hossein.hajipour@cispa.de);(2) Keno Hassler, CISPA Helmholtz Center for Information Security (keno.hassler@cispa.de);(3) Thorsten Holz, CISPA Helmholtz Center for Information Security (holz@cispa.de);(4) Lea Schonherr, CISPA Helmholtz Center for Information Security (schoenherr@cispa.de);(5) Mario Fritz, CISPA Helmholtz Center for Information Security (fritz@cispa.de).::::::infoThis paper is available on arxiv under CC BY-NC-SA 4.0 DEED license.:::\