pszemraj/stablelm-4e1t-2b-v0.1
pszemraj/stablelm-4e1t-2b-v0.1 is a 2.2 billion parameter causal language model, derived from stablelm-3b-4e1t through a layer pruning experiment. It features a 32768 token context length and was subsequently fine-tuned with continued pretraining on FineWeb data. This model is optimized for efficiency, demonstrating a reduced parameter count while maintaining performance for general text generation tasks.
Loading preview...
Model Overview
pszemraj/stablelm-4e1t-2b-v0.1 is a 2.2 billion parameter causal language model developed by pszemraj. This model is an experimental variant of stablelm-3b-4e1t, created by pruning 10 layers using the PruneMe tool and a dataset based on FineWeb. Following pruning, the model underwent brief continued pretraining at a context length of 4096 tokens on additional FineWeb and curated data.
Key Characteristics
- Parameter Count: Reduced to 2.2 billion parameters from its 3B base model, focusing on efficiency.
- Context Length: Supports a substantial context window of 32768 tokens.
- Pruning Methodology: Utilizes a layer pruning technique to optimize model size and potentially inference speed.
- Continued Pretraining: Enhanced with additional pretraining on diverse data after the pruning process.
Training Details
The model was trained using axolotl version 0.4.0, with a sequence length of 4096 and a learning rate of 5e-5. The training involved 2 epochs with a cosine learning rate scheduler and adamw_torch_fused optimizer. Training loss progressively decreased, indicating effective learning during the continued pretraining phase.
Intended Use Cases
This model is suitable for applications requiring a compact yet capable language model, particularly where resource efficiency is a priority. Its design makes it a good candidate for general text generation, summarization, and other NLP tasks where a smaller footprint is advantageous.