hemlang/Hemlock-Codex-7B
Hemlock-Codex-7B is a 7.6 billion parameter language model developed by hemlang, fine-tuned from the Hemlock2-Coder-7B base model. This instruction-tuned model is optimized for code generation and understanding tasks, leveraging a 32,768 token context length. It is specifically designed for applications requiring robust performance in programming-related contexts.
Loading preview...
Hemlock-Codex-7B Overview
Hemlock-Codex-7B is a 7.6 billion parameter instruction-tuned language model developed by hemlang, built upon the hemlang/Hemlock2-Coder-7B base model. It was trained using a Supervised Fine-Tuning (SFT) approach over 3 epochs, with a maximum sequence length of 8192 tokens during training, and supports an inference context length of 32,768 tokens.
Key Training Configuration Details
- Base Model:
hemlang/Hemlock2-Coder-7B - Training Mode: Supervised Fine-Tuning (SFT)
- Max Sequence Length (Training): 8192
- Quantization: 4-bit (NF4)
- LoRA Configuration: Rank (r) 128, Alpha 128, Dropout 0.05, targeting key attention and feed-forward projection modules.
This model is particularly suited for tasks requiring strong code generation and comprehension capabilities, benefiting from its specialized fine-tuning process.