codefuse-ai/CodeFuse-CodeLlama-34B
TEXT GENERATIONConcurrency Cost:2Model Size:34BQuant:FP8Ctx Length:32kPublished:Sep 7, 2023License:otherArchitecture:Transformer0.1K Cold
CodeFuse-CodeLlama-34B is a 34 billion parameter Code-LLM developed by codefuse-ai, fine-tuned using QLoRA on 600k instruction/answer pairs. Based on CodeLlama-34b-Python, this model specializes in code generation and achieves a 74.4% pass@1 on HumanEval, outperforming GPT-4 and other open-source models. It is optimized for code-related tasks, offering strong performance in programming contexts.
Loading preview...
CodeFuse-CodeLlama-34B Overview
CodeFuse-CodeLlama-34B is a 34 billion parameter Code-LLM developed by codefuse-ai, specifically fine-tuned for various code tasks. It leverages QLoRA on the CodeLlama-34b-Python base model, utilizing 600,000 instruction/answer pairs for training. While fine-tuned with a 4K context length, it supports extension to 16K if needed.
Key Capabilities & Performance
- Exceptional Code Generation: Achieves a notable 74.4% pass@1 on the HumanEval benchmark (greedy decoding), positioning it as a leading open-source model for code generation. This performance surpasses models like GPT-4 (67.0%) and GPT-3.5 (48.1%) on this specific metric.
- Quantized Version Available: A 4-bit quantized version maintains strong performance, achieving 73.8% pass@1 on HumanEval, making it suitable for environments with memory constraints.
- Specialized Fine-tuning: Optimized through multi-task fine-tuning (MFTCoder) for enhanced code understanding and generation.
Good For
- Code Development: Ideal for developers requiring high-accuracy code generation and completion.
- Research and Experimentation: Provides a strong base for further fine-tuning or deployment in code-centric AI applications.
- Resource-Efficient Deployment: The availability of a 4-bit quantized version allows for deployment in environments with limited GPU memory.