foss22/pruned26L-RuadaptQwen3-4B-10L
The foss22/pruned26L-RuadaptQwen3-4B-10L is a 4 billion parameter language model, derived from RefalMachine/RuadaptQwen3-4B-Instruct through significant depth pruning. This model has 26 layers removed, resulting in a 65.48% parameter reduction and 72.22% layer reduction, reducing its original 36 layers to 10. While demonstrating substantial compression, its current generation capabilities are severely degraded, making it primarily a case study in aggressive model pruning rather than a functional LLM for general use.
Loading preview...
Overview
The foss22/pruned26L-RuadaptQwen3-4B-10L is an experimental language model created by foss22 through aggressive depth pruning of the RefalMachine/RuadaptQwen3-4B-Instruct model. The original 4 billion parameter, 36-layer model was reduced to a 10-layer architecture, resulting in a 65.48% parameter reduction (from 4,007,937,536 to 1,383,736,320 parameters) and a 72.22% layer reduction (26 layers removed).
Key Characteristics
- Pruned Architecture: A significantly compressed version of
RuadaptQwen3-4B-Instruct. - Parameter Reduction: Achieves a substantial 65.48% reduction in parameters.
- Layer Reduction: Reduced from 36 layers to just 10 layers.
- Degraded Performance: As demonstrated by the provided generation examples, the pruning process has severely impacted the model's ability to produce coherent and meaningful text, indicating a loss of core language understanding.
Intended Use
This model is primarily a demonstration of the effects of aggressive depth pruning on a large language model. It serves as a valuable artifact for researchers and developers interested in:
- Studying model compression techniques.
- Understanding the impact of layer removal on model performance.
- Exploring the trade-offs between model size and capability.
Limitations
Due to the extreme pruning, this model is not suitable for direct use in applications requiring coherent text generation or language understanding. Its output is largely nonsensical, as shown in the examples provided in the original model card. Users should be aware of these significant limitations.