Organization Experience

Publication

An Evaluation of the Effectiveness of OpenAI's ChatGPT for Automated Python Program Bug Fixing using QuixBugs | IEEE · Sep 16, 2023

Presented at iSemantic Conference 2023 held by UDINUS.
In recent years, the use of Artificial Intelligence (AI) has become increasingly common in various fields, including in software development. One such field is where AI can automatically detect and fix bugs in code. GPT-3.5 is a state-of-the-art language model developed by OpenAI that has been trained on a massive amount of text data to generate natural language responses to a wide range of prompts. One of the main challenges in software development is bug fixing, which can be a time- consuming and complicated process. QuixBugs is a framework for evaluating automatic program repair techniques, which can be used to test the effectiveness of GPT-3.5 and similar bug-fixing tools. This paper evaluates the effectiveness of GPT-3.5 in automatically fixing bugs in Python code using QuixBugs. Through testing with 40 different Python bugs, We discovered that GPT-3.5 was able to accurately fix 30 out of 40 bugs cases from QuixBugs benchmark. Compared with other tools like standard program repair and Codex, ChatGPT outperformed them significantly. These findings highlight the potential of ChatGPT as a powerful tool for enhancing code quality and reducing the burden of manual bug fixing.