Mihaiii/Llama-3-pruned-45B-Drobeta-Turnu-Severin
Mihaiii/Llama-3-pruned-45B-Drobeta-Turnu-Severin is a pruned version of Meta-Llama-3-70B-Instruct, reducing its parameter count from 70.6 billion to 44.9 billion (approximately 63% of the original size). This model is specifically designed for fine-tuning rather than direct inference, offering a more compact base for further specialization. It was created by Mihaiii using a layer elimination method based on signal-to-noise ratio (SNR) analysis of self_attn.v_proj layers. Its primary utility lies in providing a smaller, more efficient foundation for developers to build upon with custom fine-tuning tasks.
Loading preview...
Model Overview
Mihaiii/Llama-3-pruned-45B-Drobeta-Turnu-Severin is a compact variant of the powerful Meta-Llama-3-70B-Instruct model. This version has been significantly reduced in size, from 70.6 billion parameters to 44.9 billion, representing approximately 63% of the original model's parameter count.
Key Characteristics
- Pruned Architecture: Derived from Meta-Llama-3-70B-Instruct through a systematic layer elimination process.
- Parameter Reduction: Achieves a substantial reduction in size, making it potentially more efficient for certain applications.
- Fine-tuning Focus: Explicitly intended for use as a base model for fine-tuning, rather than for direct inference.
Pruning Methodology
The pruning process involved identifying and eliminating specific layers from the original Meta-Llama-3-70B-Instruct model. This was achieved by using the laserQlora.ipynb script from cognitivecomputations/laserRMT to analyze the signal-to-noise ratio (SNR) of self_attn.v_proj layers. Layers with an SNR of Infinity were targeted for removal. The specific layers eliminated were: 11, 17, 37, 40, 41, 42, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69. The mergekit tool was then used to reconstruct the model with the remaining layers.
Intended Use
This model is designed to serve as an optimized starting point for developers who need a smaller, more manageable Llama-3-based model for specific fine-tuning tasks. It is not recommended for direct inference without further fine-tuning.