safal312/Iterative-Qwen-7layers-SGR-Dolci
safal312/Iterative-Qwen-7layers-SGR-Dolci is a 7.6 billion parameter, depth-pruned version of Qwen2.5-7B-Instruct, developed by Safal Shrestha and team. It was created using iterative layer pruning and post-trained with QLoRA, incorporating Self-Generated Responses (SGR) on the Dolci dataset. This model is specifically intended for research and analysis into the limits of layer pruning and recovery for generative reasoning in large language models.
Loading preview...
Model Overview
safal312/Iterative-Qwen-7layers-SGR-Dolci is a 7.6 billion parameter language model derived from Qwen2.5-7B-Instruct. It represents a significant exploration into the efficiency and performance of depth-pruned models, specifically focusing on generative reasoning capabilities.
Key Characteristics
- Base Model: Qwen2.5-7B-Instruct, a robust foundation for language tasks.
- Pruning Method: Utilizes iterative layer pruning to reduce its depth to 7 layers, making it a more compact architecture.
- Post-training: Enhanced through QLoRA (Quantized Low-Rank Adaptation) for efficient fine-tuning.
- Training Data & Method: Incorporates Self-Generated Responses (SGR) on the Dolci dataset, a novel approach to supervision.
Intended Use
This model is primarily designed for research and analysis related to the effects and recovery mechanisms of layer pruning in large language models. It is not recommended for production environments but serves as a valuable tool for understanding the trade-offs between model size and generative reasoning performance, as detailed in the associated paper: On the Limits of Layer Pruning for Generative Reasoning in LLMs.