elinas/Llama-3-15B-Instruct-ft-v2

Model Overview

elinas/Llama-3-15B-Instruct-ft-v2 is a 15 billion parameter instruction-tuned model, developed by elinas. It represents a QLoRA finetune of a passthrough merge, specifically building upon the Llama-3-15B-Instruct-zeroed base. This version was an experimental finetune to assess the response of a passthrough merge to further training across all LoRA modules.

Key Characteristics

• Architecture: QLoRA finetune of a merged Llama-3-15B-Instruct variant.
• Parameter Count: 15 billion parameters.
• Context Length: Finetuned on an 8192 token context length, with potential for extension up to 32k using RoPE.
• Finetuning Approach: All LoRA modules (gate_proj, down_proj, up_proj, q_proj, v_proj, k_proj, o_proj) were targeted, along with embed_tokens and lm_head.
• Dataset: Utilized a small, high-quality, curated dataset, Chat-Error/Pure-dove-sharegpt, for validation and stabilization.
• Training Details: Trained for 1 epoch using paged_adamw_8bit optimizer and Deepspeed ZeRO 3, with a learning rate of 1e-5, leveraging Unsloth for efficiency.

Future Development

Version 3 of this model is planned to incorporate significantly more human-focused data, with the goal of excelling in writing, maintaining logic, coherency, and continuity.