kuotient/Llama-3-6B-Instruct-pruned
kuotient/Llama-3-6B-Instruct-pruned is an experimental 8 billion parameter instruction-tuned language model based on Meta-Llama-3-8B-Instruct. Developed by kuotient, this model has been pruned using the PruneMe toolkit to remove specific layers (22-28) from the original Llama 3 architecture. It aims to explore the effects of structural pruning on large language models, potentially offering a more compact model while retaining core instruction-following capabilities.
Loading preview...
Overview
kuotient/Llama-3-6B-Instruct-pruned is an experimental language model derived from the meta-llama/Meta-Llama-3-8B-Instruct base model. This project utilizes the PruneMe toolkit to perform structural pruning, specifically removing layers 22 through 28 from the original 32-layer Llama 3 architecture. The goal is to investigate the impact of pruning on model efficiency and performance.
Pruning Methodology
The model was created using a 'passthrough' merge method via mergekit. The pruning strategy involved retaining layers 0-21 and layers 29-32 from the original Meta-Llama-3-8B-Instruct model. The developer notes that while the pruning analysis suggested layers 22-30 as the optimal range for removal, a more conservative approach was taken due to concerns about drastic changes.
Current Status and Future Work
This is an experimental release, and the developer explicitly states that no post-training or 'healing' process (as suggested by relevant research papers like 2403.17887) has been applied yet. This indicates that the model's performance might be further optimized with subsequent fine-tuning steps. The project serves as a demonstration of applying pruning techniques to large, instruction-tuned models.