Exploring User Needs and Satisfaction with GitHub Copilot

Wait 5 sec.

Table of LinksAbstract and 1. Introduction2. Methodology and 2.1. Research Questions 2.2. Data Collection 2.3. Data Labelling 2.4. Data Extraction 2.5. Data Analysis 3. Results and Interpretation and 3.1. Type of Problems (RQ1) 3.2. Type of Causes (RQ2) 3.3. Type of Solutions (RQ3) 4. Implications 4.1. Implications for the Copilot Users 4.2. Implications for the Copilot Team 4.3. Implications for Researchers 5. Threats to Validity 6. Related Work 6.1. Evaluating the Quality of Code Generated by Copilot 6.2. Copilot’s Impact on Practical Development and 6.3. Conclusive Summary 7. Conclusions, Data availability, Acknowledgments, CRediT authorship contribution statement and References4.3. Implications for ResearchersThe use of Copilot may alter the coding process and increase the time cost of verifying code suggestions, making code explanation highly important. In our research, INCOMPREHENSIBLE SUGGESTION ranks as the fourth most common Suggestion Content Issue, and the code explanation feature is also frequently requested. Some users mentioned that the code suggestions generated by Copilot are excessively long, resulting in reduced readability. This indicates that when Copilot provides relatively complex suggestions, or when users lack coding experience in a particular domain, understanding the code logic and verifying its correctness can be time-consuming. In that case, users may refrain from adopting code suggestions of Copilot as it makes them feel a loss of control over the coding task, according to the research of Vaithilingam et al. (2022). The study by Wang et al. (2023b) also shows that using AI-generated code can lead to substantial review pressure. Therefore, we believe that AI coding tools like Copilot will change the allocation of time spent on various tasks in software development. As expected by some users, the code explanation feature would be helpful. GitHub Copilot Chat now is generally available for organizations and individuals (Zhao, 2023), which is powered by GPT-4. The chat feature enables Copilot to interact more flexibly with users, providing the potential for implementing functions such as code explanation, code refactoring, and vulnerability detection. For future research, we plan to explore how the chat feature can influence the programming process and enhance efficiency of coding tasks.\To accurately gauge user satisfaction with Copilot, it is essential to consider programming tasks across various application domains, as well as the purposes for which users employ Copilot. We identified relatively few Suggestion Content Issues, and one potential reason is that users are less inclined to report specific code-related issues to public Q&A platforms, indicating a certain degree of tolerance for AI-generated erroneous suggestions (Weisz et al., 2021). Therefore, our study results may not fully reflect user satisfaction with the Copilot generated code content. Users often propose new Feature Requests based on their individual needs and coding habits, coupled with their experience of using Copilot. Some of these requests are closely associated with the domains of the applications that Copilot users develop, such as front-end web development and game development. Consequently, user satisfaction levels with Copilot may vary across different application domains. Thus, assessing the capabilities of Copilot through the lens of single programming tasks, such as addressing algorithmic problems or writing SQL statements, may not represent the actual user satisfaction in other application domains. Furthermore, our previous research found that users have diverse objectives when using Copilot (Zhang et al., 2023), directly influencing their expectations. For instance, users looking to quickly grasp new technologies with the assistance of Copilot are more concerned with whether the code suggestions provide helpful programming guidance for subsequent steps; conversely, those who wish to use Copilot for repetitive coding tasks, such as CRUD operations on a database, are more concerned with whether Copilot can generate correct code to reduce the significant effort they spend on fixing similar issues in code suggestions. Therefore, the intended utilization of Copilot also stands as a critical factor influencing the satisfaction of users with Copilot. A focused study on user satisfaction with Copilot based on these two factors will provide an insightful assessment.\:::infoAuthors:(1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China (xiyuzhou@whu.edu.cn);(2) Peng Liang (Corresponding Author), School of Computer Science, Wuhan University, Wuhan, China (liangp@whu.edu.cn);(3) Beiqi Zhang, School of Computer Science, Wuhan University, Wuhan, China (zhangbeiqi@whu.edu.cn);(4) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China (zengyangli@ccnu.edu.cn);(5) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany (ahmad.aakash@gmail.com);(6) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia (mojtaba.shahin@rmit.edu.au);(7) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland (muhammad.m.waseem@jyu.fi).::::::infoThis paper is available on arxiv under CC BY 4.0 DEED license.:::\