Elfsong/Qwen2.5-Coder-3B-Instruct-Venus-Cold-Start
Elfsong/Qwen2.5-Coder-3B-Instruct-Venus-Cold-Start is a 3.1 billion parameter instruction-tuned language model, fine-tuned from Qwen/Qwen2.5-Coder-3B-Instruct. This model is specifically adapted for code-related tasks, leveraging its base architecture for programming contexts. It features a 32768-token context length, making it suitable for handling extensive codebases and complex coding instructions. The fine-tuning on the venus_cold_start dataset aims to enhance its performance in specific coding scenarios.
Loading preview...
Model Overview
Elfsong/Qwen2.5-Coder-3B-Instruct-Venus-Cold-Start is a 3.1 billion parameter language model, derived from the Qwen/Qwen2.5-Coder-3B-Instruct base model. It has been further fine-tuned on the venus_cold_start dataset, indicating a specialization for particular coding or development-related tasks. The model maintains a substantial context length of 32768 tokens, which is beneficial for processing and generating longer code sequences or understanding complex programming contexts.
Key Characteristics
- Base Model: Fine-tuned from Qwen/Qwen2.5-Coder-3B-Instruct, suggesting a foundation optimized for code generation and understanding.
- Parameter Count: 3.1 billion parameters, offering a balance between performance and computational efficiency.
- Context Length: Supports a 32768-token context window, enabling the model to handle extensive code snippets and detailed instructions.
- Specialized Fine-tuning: Adapted using the
venus_cold_startdataset, implying a focus on specific coding challenges or environments.
Training Details
The model was trained with a learning rate of 5e-05, using an AdamW optimizer with 8-bit quantization. Training involved 2 epochs with a total batch size of 16 (achieved with gradient accumulation) across 4 GPUs. The training utilized Transformers 4.51.3 and PyTorch 2.6.0+cu124.