SPAISS6F1/qwen-1b-pruned-th
SPAISS6F1/qwen-1b-pruned-th is a 1.70 billion parameter Thai language model derived from Qwen/Qwen2.5-3B through depth pruning and subsequent healing SFT. This model retains 18 of the original 36 layers, focusing on input and output processing layers, and was fine-tuned on SEA-PILE v2 Thai data. It is optimized for fluent Thai text generation and serves as a strong base for further instruction tuning.
Loading preview...
Model Overview
SPAISS6F1/qwen-1b-pruned-th is a compact 1.70 billion parameter Thai language model, created by applying Depth Pruning (Layer Dropping) to the larger Qwen/Qwen2.5-3B base model. This process involved reducing the original 36 layers to 18, specifically retaining the initial and final layers while discarding intermediate ones. Following pruning, the model underwent Healing SFT (Supervised Fine-Tuning) using the SEA-PILE v2 Thai dataset (approximately 8,000 documents) to restore and enhance its capabilities.
Key Characteristics
- Pruned Architecture: Reduced from 3.09B parameters (36 layers) to 1.70B parameters (18 layers) by selectively keeping layers [0-8, 27-35].
- Thai Language Focus: Specifically healed and optimized for generating fluent Thai text.
- Training Data: Fine-tuned on a substantial Thai corpus to ensure grammatical correctness and natural language flow.
- Context Length: Inherits a context length of 32768 tokens from its base model.
Use Cases and Limitations
- Ideal as a Base Model: This model is primarily intended as a base for further fine-tuning with instruction datasets, rather than for direct question-answering or complex reasoning tasks.
- Fluent Thai Generation: Excels at producing grammatically sound and natural-sounding Thai text.
- Limitations: Due to its pruned nature and base-model healing, it may exhibit weaknesses in factual accuracy and mathematical reasoning. It is recommended to use a
repetition_penalty >= 1.2during inference to prevent repetitive outputs. - Pruning Intensity: The 50% layer reduction is significant; for higher quality, longer healing or less aggressive pruning might be considered.