AI Tools Made Open Source Software Developers 19% Slower, Study Finds
Even as excitement over artificial intelligence (AI) continues to build in software development, a new study published today finds that AI coding assistants sometimes do more harm than good – especially if you’re an experienced developer.
AI-Based Coding Tools & Participants
The study was conducted by researchers at University of California, Davis and other academic collaborators, who investigated the influence of AI-assisted programming tools on open-source contributors. In a surprise twist, the study found that developers using tools like GitHub Copilot were actually 19 percent slower than their non-assisted peers.
A Closer Look at the Study
The researchers performed the study with more than 100 experienced open-source contributors, and their results have been peer-reviewed in one of the top computer science journals. Subjects were randomly assigned to receive or not receive assistance from an AI coding assistant.
The test tasks involved real-world programming work, such as:
- Bug fixing
- Feature implementation
These tasks were designed to replicate what contributors encounter daily in open-source projects.
The problems were selected from actual reported issues on popular GitHub repositories, reflecting real-world complexity. The study recorded:
- Time spent
- Code quality
- Developer satisfaction
This comprehensive approach provided an in-depth view of performance.
Why the Slowdown?
The chief cause of the decline in productivity, the study concluded, was the extra work needed to inspect and adjust AI-generated code. While AI tools may offer boilerplate advice or quick implementations, developers frequently spent additional time:
- Debugging
- Tweaking for accuracy
- Rewriting suboptimal code
“AI codegen tools are not mature enough to be absolutely trusted right now,” said study lead researcher Dr. Megha Srivastava, an associate professor of computer science.
“You still have to have really talented developers thinking really hard about code quality and correctness.”
Mismatch Between Suggestion and Intention
A repeated issue observed during the study was that AI-generated code, although syntactically correct, often missed the developer’s intent at a semantic level.
For example, participants working on an API improvement task with AI assistance received code suggestions that used a different approach than intended. This led to:
- Rewriting sections of code
- Being blocked from further contributions
This semantic mismatch—when the tool misunderstands context or the user’s goal—became a significant barrier to productivity.
Developers stated that interpreting the AI’s recommendations and deciding on their relevance took more time than simply constructing the solution by hand.
Overconfidence and Cognitive Load
The study also uncovered a psychological factor contributing to the decline in performance. Developers sometimes felt obligated to accommodate AI recommendations, even when unsure of their correctness. This led to:
- Increased cognitive load, as developers switched between trusting the AI and themselves
- A false sense of security in the AI’s confident suggestions
- Bugs that took longer to trace than if the code had been written manually
“Developers using AI tools dedicated significantly more time in reviewing and testing the produced code than when they based the code written on their experience and knowledge,” according to the report.
Not All AI Tools Are Equal
While the study focused on general-purpose AI coding assistants like GitHub Copilot, performance varied with other tools:
- Domain-specific tools showed slightly better results when applied to domain-specific repositories
- Developers familiar with the AI assistant’s quirks performed about as well as the control group
- Developers new to the tool showed reduced productivity
This supports the idea that tool familiarity plays a role in its effectiveness.
Future of AI in Development – Perceptions and Opportunities
This research should serve as a note of caution for companies and open-source projects adopting AI tools as productivity boosters.
While AI shows strong potential for tasks such as:
- Autocompletion
- Code formatting
- Auto-generating documentation
…it may not yet be suitable for handling complex development tasks independently.
“AI remains a co-pilot, not the driver,” said co-author Dr. Ananya Kothari.
“We’re not to the point where these tools will replace skilled developers, particularly in more nuanced or critical situations.”
Nonetheless, researchers are optimistic. They see the study not as a criticism but as a call to understand limitations and strive for improvement.
With:
- Advances in large language models
- More context-aware suggestions
- Richer training data
…the gap between human intent and AI output is expected to narrow.
The Industry Response
After publication, the study sparked widespread discussion across:
- Tech forums
- Developer networks
- Software engineering conferences
Some developers felt vindicated, sharing stories of AI-generated code introducing bugs or flawed logic. Others noted that AI assistants still benefit beginners and junior developers.
GitHub, the company behind Copilot, acknowledged the findings:
“We understand AI tools continue to evolve, and how developers interact with them will, too,” the company said in a statement.
“We are committed to working closely with the developer community to further refine the accuracy, relevance, and usability of Copilot.”
The Bottom Line
AI has already disrupted the software development process, but the dream of dramatic productivity boosts—particularly for experienced open-source developers—remains unrealized.
The 19% productivity drop highlighted in this study underscores the need for:
- Thoughtful AI integration
- Proper training for AI users
- Ongoing improvement of AI coding tools
Ultimately, the interaction between human developers and AI tools is a collaborative journey—one that calls for balance, adaptability, and mutual understanding.
AI can be a powerful sidekick, but for now, it still requires a skilled human hand at the wheel.
