Past as a Guide
Leveraging Retrospective Learning for Python Code Completion
Seungyoun Shin 1,2 Seunggyu Chang2 Sungjoon Choi2
1Korea University 2Naver Cloud
Instruction Workshop @ NeurIPS 2023
Paper | Code
Abstract
This work presents Past as a Guide (PaG), a simple approach for Large Language Models (LLMs) to improve the coding capabilities by integrating the past history with interactive and iterative code refinements. To be specific, inspired by human cognitive processes, the proposed method enables LLMs to utilize previous programming and debugging experiences to enhance the Python code completion tasks. The framework facilitates LLMs to iteratively refine the Python code based on previous execution and debugging results and optimize learning and reasoning capabilities. The proposed methodology achieved a 92% pass@1 on HumanEval, demonstrating the potential to advance the field by leveraging retrospection from past experiences and interactive and iterative refinement processes without external correctness indicators. |
Method Overview
![]() |
First, an initial question is presented. Then, a similar Prior Experience is retrieved in response to this question. Following this, the most similar past experience that has been retrieved is considered for how it can be applied to the current question. This is the Retrospection step. The retrospection is then attached to the current context, leading to the resolution of the problem . If Python Code is involved, it is executed to and it's execution result is appended to the context. Finally, this achieved result is added back into the Memory Back. |
Experiments
Code Completion Experiments
Task: This experiment is based on the HumanEval task, aimed at completing Python code skeletons. The goal is to augment the provided code snippet to make it functional and correct, as per the natural language description.
Results: The effectiveness of our methods was compared against baseline models. Notably, the method GPT-4 + † (max 12 tries) + PaG achieved a state-of-the-art pass rate of 92.07%, outperforming other approaches.
Method | % Pass@1 |
---|---|
GPT-4 | 67.00 |
GPT-4 + † (max 6 tries) | 84.15 |
GPT-4 + † (max 12 tries) | 90.85 |
Reflexion | 91.00 |
GPT-4 + † (max 12 tries) + PaG | 92.07 |
The approach of combining retrospective guidance with interactive and iterative code refinements proved effective, even without external correctness feedback.
Citation
|