AI-Powered, But Power-Hungry? Energy Efficiency of LLM-Generated Code

Large language models (LLMs) are used in software development to assist in various tasks, e.g., code generation and code completion, but empirical evaluations of the quality of the results produced by these models focus on correctness and ignore other relevant aspects, such as their performance and energy efficiency. Studying the performance of LLM-produced programs is essential to understand how well LLMs can support the construction of performance- and energy-critical software, such as operating systems, servers, and mobile applications. This paper presents the first study analyzing the energy efficiency and performance of LLM-generated code for three programming languages Python, Java, and C++, on two platforms, a Mac and a PC, leveraging three frontier LLMs, Github Copilot, GPT-4o, and the recently-released OpenAI o1-mini, and targeting "hard"programming problems from LeetCode. Our results show that the models are much more successful in generating Python and Java than C++ code. Also, LLM-generated code sometimes surpasses an efficient human-written solution, although that is language-dependent and the language with the best results, Python, is the one where application performance and energy consumption tend to matter the least in practice. Furthermore, the performance of generated code is highly correlated across the two platforms, hinting at potential for results to be portable across platforms.