Overview

rombodawg/Llama-3-8B-Instruct-Coder is an 8 billion parameter instruction-tuned model built upon Meta's Llama-3-8B-Instruct. Its primary distinction lies in its specialized training, having been fine-tuned on a comprehensive dataset of 215,000 code-related entries, including the full 65k Codefeedback dataset and an additional 150k Code Feedback Filtered Instruction dataset. This extensive code-centric training aims to enhance its performance in programming and code generation tasks.

Key Capabilities

• Code Generation and Understanding: Optimized for tasks requiring an understanding of code and generating programming solutions due to its specialized dataset.
• Efficient Training: Utilizes the novel Qalore method, which integrates QLoRA and GaLore techniques. This method significantly reduces VRAM requirements, allowing for the training of models like Llama-3-8B on hardware with as little as 14.5 GB of VRAM, such as an RTX A4000 16GB.

Training Details

The model was trained using the Qalore method, developed by "walmartbag" from Replete-AI. This approach enabled the training of the Llama-3-8B model on an RTX A4000 16GB GPU over 130 hours, demonstrating a cost-effective and accessible training methodology. A Colab notebook is available for those interested in the Qalore training process.