GitHub’s AI coding assistant, Copilot, has been around for two years, which prompted the creator to conduct an extensive study on the topic of code quality. This shows that code created with the help of AI passes more unit tests and has fewer errors. Reviewers have also rated it more readable and reliable.
Advertisement
GitHub has 202 Python developers Must have at least five years of professional experience when given the task of writing API endpoints for web servers, especially rating systems for restaurants. Of those, 104 were allowed to use GitHub Copilot, 98 were not allowed to use the AI assistant. Testers ran all instances through ten unit tests to check correct functionality, and the result was clearly in favor of AI assistance: 60.8 percent of all programs passed all ten tests with CoPilot, AI-assisted. Only 39.2 percent without.
Programs written with the help of AI have a much higher chance of passing all ten unit tests.
(Image: GitHub)
25 selected developers whose code passed all ten tests were asked to conduct a blind, anonymous review of the programs, with each program being checked a total of ten times. In this stage, errors no longer mean functional defects, but rather qualitative defects in terms of maintainability or readability: inconsistent naming, ambiguous identifiers, lines that are too long, loops that are too nested, missing comments, repeated expressions ( DO NOT REPEAT) YOURSELF, DRY) or partition unclean functions.
AI does well here too, but not as clearly. On average, reviewers found 4.63 errors in programs with AI assistance and 5.35 errors in programs without AI assistance. The number of lines of code per error code was also larger: 18.2 lines versus 16 lines without the help of AI. This contradicts earlier studies that specifically suspected bloated code and violations of the DRY principle. Other recent studies support the idea that AI improves quality.
AI-assisted programs also perform better when it comes to the number of incorrect lines of code.
(Image: GitHub)
Additionally, reviewers should give soft comments about how readable, reliable, maintainable, and concise the code appears to be. Here the advantage for the co-pilot is between 3 and 5 percent, although the study does not fully explain how the value was arrived at.
The study also looked at subjects’ commitments, which were more frequent and shorter with the help of AI.
In conclusion, the study suggests: “Our hypothesis is that because developers need less time to make code functional, it allows them to focus more on refining quality.”
(Who)
