oopere/gemma-3-270m-14L-distilled
oopere/gemma-3-270m-14L-distilled is a 0.3 billion parameter model, a surgically optimized and distilled version of Google's Gemma-3-270m. Developed as part of the "Rearchitecting LLMs" book, it demonstrates depth pruning and knowledge distillation techniques. This model achieves 94.2% overall retention of the teacher's capabilities, making it highly parameter-efficient for educational purposes and exploring LLM optimization.
Loading preview...
Model Overview
This model, oopere/gemma-3-270m-14L-distilled, is a 0.3 billion parameter language model derived from google/gemma-3-270m. It was created as the final challenge for Chapter 6 of the book "Rearchitecting LLMs" to showcase advanced LLM optimization techniques.
Key Capabilities & Techniques
- Surgical Optimization: Utilizes depth pruning and knowledge distillation (labels-only with Skew KL Divergence) to reduce model size while retaining performance.
- Parameter Efficiency: Achieves an 8.32% parameter reduction by removing 4 transformer layers, resulting in a student model with 14 blocks.
- High Retention: Demonstrates strong capability retention, with 89.95% of reasoning power and 119.00% of linguistic quality (PPL) preserved compared to the teacher model, leading to an overall retention of 94.20%.
- Educational Focus: Serves as a practical example for understanding the complete optimization pipeline: Prune → Distill → Evaluate.
Intended Use & Limitations
This model is primarily an educational artifact for learning about LLM rearchitecting and is not intended for production use. It is best suited for simple completion tasks and demonstrating optimization principles. Its knowledge coverage is reduced compared to full-scale models, and complex reasoning may be degraded. The model is English-only and trained on the general-purpose Cosmopedia corpus.