Publications and Preprints
Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codes
Preprint, 2024
paper
This paper emphasizes the importance of code diversity, beyond functional correctness, in evaluating code language models (LMs). We propose a framework that includes novel metrics for inter-code similarity and introduce a pairwise similarity measure that correlates well with human judgment. We show that while current LMs generate functionally correct code, they often lack diversity, which is essential for robust and varied code solutions.
Preprint, 2024
paper
This paper emphasizes the importance of code diversity, beyond functional correctness, in evaluating code language models (LMs). We propose a framework that includes novel metrics for inter-code similarity and introduce a pairwise similarity measure that correlates well with human judgment. We show that while current LMs generate functionally correct code, they often lack diversity, which is essential for robust and varied code solutions.
Eliciting Instruction-tuned Code Language Models' Capabilities to Utilize Auxiliary Function for Code Generation
EMNLP, 2024
paper
The paper explores how instruction-tuned models can effectively utilize auxiliary functions in code generation. By designing prompts that encourage models to use auxiliary functions, the authors show improved performance compared to models without this functionality. Their method surpasses even powerful proprietary models like GPT-4, highlighting the potential of open-source models when auxiliary function usage is integrated
EMNLP, 2024
paper
The paper explores how instruction-tuned models can effectively utilize auxiliary functions in code generation. By designing prompts that encourage models to use auxiliary functions, the authors show improved performance compared to models without this functionality. Their method surpasses even powerful proprietary models like GPT-4, highlighting the potential of open-source models when auxiliary function usage is integrated