elinas/Llama-3-15B-Instruct-ft-v2
elinas/Llama-3-15B-Instruct-ft-v2 is a 15 billion parameter instruction-tuned language model, a QLoRA finetune based on a passthrough merge of Llama-3-15B-Instruct-zeroed. It was finetuned with an 8192 token context length, targeting all LoRA modules for enhanced training. This model is primarily an experimental finetune aimed at stabilizing performance after a complex merge, with future versions focusing on writing, logic, and coherency.
Loading preview...
Model Overview
elinas/Llama-3-15B-Instruct-ft-v2 is a 15 billion parameter instruction-tuned model, developed by elinas. It represents a QLoRA finetune of a passthrough merge, specifically building upon the Llama-3-15B-Instruct-zeroed base. This version was an experimental finetune to assess the response of a passthrough merge to further training across all LoRA modules.
Key Characteristics
- Architecture: QLoRA finetune of a merged Llama-3-15B-Instruct variant.
- Parameter Count: 15 billion parameters.
- Context Length: Finetuned on an 8192 token context length, with potential for extension up to 32k using RoPE.
- Finetuning Approach: All LoRA modules (
gate_proj,down_proj,up_proj,q_proj,v_proj,k_proj,o_proj) were targeted, along withembed_tokensandlm_head. - Dataset: Utilized a small, high-quality, curated dataset,
Chat-Error/Pure-dove-sharegpt, for validation and stabilization. - Training Details: Trained for 1 epoch using
paged_adamw_8bitoptimizer and Deepspeed ZeRO 3, with a learning rate of 1e-5, leveraging Unsloth for efficiency.
Future Development
Version 3 of this model is planned to incorporate significantly more human-focused data, with the goal of excelling in writing, maintaining logic, coherency, and continuity.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.