ermiaazarkhalili/LFM2.5-1.2B-SFT-Fable5-Glint

Model Overview

ermiaazarkhalili/LFM2.5-1.2B-SFT-Fable5-Glint is a 1.2 billion parameter language model, developed by ermiaazarkhalili. It is a LoRA (Low-Rank Adaptation) fine-tune of the LiquidAI/LFM2.5-1.2B-Instruct base model, utilizing the Lfm2ForCausalLM architecture.

Key Characteristics & Training

• Base Model: LiquidAI/LFM2.5-1.2B-Instruct
• Parameters: 1.2 billion
• Fine-tuning Data: Supervised fine-tuning was performed on the private ermiaazarkhalili/Fable-5-Glint-Clean dataset.
• Methodology: LoRA fine-tuning was executed using Unsloth and TRL, with a LoRA rank of 16 and alpha of 16.
• Training Configuration: Trained for 3 epochs with a learning rate of 0.0002, an effective batch size of 8, and a maximum sequence length of 4096, using 4-bit QLoRA precision.

Performance & Evaluation

• Next-Token Accuracy: On a held-out split of the Fable-5-Glint-Clean dataset, this model achieved a Top-1 accuracy of 0.6268 and a Top-5 accuracy of 0.8534. This represents a significant improvement of +0.1025 (Top-1) and +0.0970 (Top-5) over its base model.
• Limitations: No external benchmark evaluations have been conducted; reported numbers are based on training loss observations and held-out accuracy on the specific fine-tuning dataset. The model inherits biases and limitations from its base model and is primarily tested for instruction-following within its training distribution.

Usage

This model is suitable for tasks requiring instruction following, leveraging its fine-tuning on a dedicated instruction dataset. The LoRA adapters are merged into the base weights, creating a unified model.