AlchemistCoder-L-7B: Enhanced Code Generation via Hindsight Tuning

AlchemistCoder-L-7B is a 7 billion parameter Code Large Language Model (LLM) developed by InternLM, designed to significantly improve code generation and generalization capabilities. Unlike many Code LLMs fine-tuned on single-source data, AlchemistCoder leverages a novel approach called hindsight tuning on multi-source data.

Key Capabilities & Innovations

• AlchemistPrompts: Introduces data-specific prompts with hindsight relabeling to harmonize diverse data sources and mitigate instruction/response misalignment, addressing inherent conflicts within multi-source code corpora.
• Code Comprehension Tasks: Incorporates data construction processes, such as instruction evolution, data filtering, and code review, directly into the fine-tuning data to enhance code understanding.
• Harmonized Multi-source Data: Instruction-tuned on 200 million tokens, comprising 6 types of high-quality data, ensuring comprehensive and diverse training.
• Superior Performance: Demonstrates a clear lead among open-source models of the same size (6.7B/7B) and rivals or surpasses larger models (15B/33B/70B/ChatGPT) across 6 code benchmarks.
• Advanced Generic Capabilities: Shows significant improvements on general benchmarks like MMLU, BBH, and GSM8K, indicating strong overall reasoning abilities beyond just coding.

Ideal Use Cases

• Code Generation: For developers requiring high-quality and accurate code generation across various programming tasks.
• Code Comprehension: When tasks involve understanding and processing complex code structures, instruction evolution, or code review.
• Instruction Following: For applications demanding precise adherence to coding instructions and multi-turn interactions.
• Benchmarking: As a strong baseline or competitive model for evaluating code intelligence tasks against other leading LLMs.