Past as a Guide
Leveraging Retrospective Learning for Python Code Completion

Seungyoun Shin 1,2    Seunggyu Chang2    Sungjoon Choi2   

1Korea University    2Naver Cloud   

Instruction Workshop @ NeurIPS 2023

Paper | Code

Abstract

This work presents Past as a Guide (PaG), a simple approach for Large Language Models (LLMs) to improve the coding capabilities by integrating the past history with interactive and iterative code refinements. To be specific, inspired by human cognitive processes, the proposed method enables LLMs to utilize previous programming and debugging experiences to enhance the Python code completion tasks. The framework facilitates LLMs to iteratively refine the Python code based on previous execution and debugging results and optimize learning and reasoning capabilities. The proposed methodology achieved a 92% pass@1 on HumanEval, demonstrating the potential to advance the field by leveraging retrospection from past experiences and interactive and iterative refinement processes without external correctness indicators.

Method Overview

First, an initial question is presented. Then, a similar Prior Experience is retrieved in response to this question. Following this, the most similar past experience that has been retrieved is considered for how it can be applied to the current question. This is the Retrospection step. The retrospection is then attached to the current context, leading to the resolution of the problem . If Python Code is involved, it is executed to and it's execution result is appended to the context. Finally, this achieved result is added back into the Memory Back.

Experiments

Code Completion Experiments

Task: This experiment is based on the HumanEval task, aimed at completing Python code skeletons. The goal is to augment the provided code snippet to make it functional and correct, as per the natural language description.

Results: The effectiveness of our methods was compared against baseline models. Notably, the method GPT-4 + † (max 12 tries) + PaG achieved a state-of-the-art pass rate of 92.07%, outperforming other approaches.

Method % Pass@1
GPT-4 67.00
GPT-4 + † (max 6 tries) 84.15
GPT-4 + † (max 12 tries) 90.85
Reflexion 91.00
GPT-4 + † (max 12 tries) + PaG 92.07

The approach of combining retrospective guidance with interactive and iterative code refinements proved effective, even without external correctness feedback.

Citation


      @misc{shin2023pag,
        title={Past as a Guide : Leveraging Retrospective Learning for Python Code Completion},
        author={Shin, Seungyoun and Chang, Seunggyu and Choi, Sungjoon},
        eprint={2311.07635},
        archivePrefix={arXiv},
        primaryClass={cs.AI}
        year={2023}
      }