rombodawg/Llama-3-8B-Instruct-Coder
rombodawg/Llama-3-8B-Instruct-Coder is an 8 billion parameter instruction-tuned causal language model based on Meta's Llama-3-8B-Instruct architecture. This model is specifically fine-tuned on a combined 215,000-entry Codefeedback dataset, making it highly optimized for code generation and understanding tasks. It leverages the Qalore training method, which combines QLoRA and GaLore techniques, enabling efficient training on consumer-grade GPUs.
Loading preview...
Overview
rombodawg/Llama-3-8B-Instruct-Coder is an 8 billion parameter instruction-tuned model built upon Meta's Llama-3-8B-Instruct. Its primary distinction lies in its specialized training, having been fine-tuned on a comprehensive dataset of 215,000 code-related entries, including the full 65k Codefeedback dataset and an additional 150k Code Feedback Filtered Instruction dataset. This extensive code-centric training aims to enhance its performance in programming and code generation tasks.
Key Capabilities
- Code Generation and Understanding: Optimized for tasks requiring an understanding of code and generating programming solutions due to its specialized dataset.
- Efficient Training: Utilizes the novel Qalore method, which integrates QLoRA and GaLore techniques. This method significantly reduces VRAM requirements, allowing for the training of models like Llama-3-8B on hardware with as little as 14.5 GB of VRAM, such as an RTX A4000 16GB.
Training Details
The model was trained using the Qalore method, developed by "walmartbag" from Replete-AI. This approach enabled the training of the Llama-3-8B model on an RTX A4000 16GB GPU over 130 hours, demonstrating a cost-effective and accessible training methodology. A Colab notebook is available for those interested in the Qalore training process.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.