li-muyang/zephyr-7b-sft-full
TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kTool Calling:SupportedPublished:Jan 4, 2025License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold
The li-muyang/zephyr-7b-sft-full model is a 7 billion parameter language model, fine-tuned from mistralai/Mistral-7B-v0.1. It was trained with a learning rate of 2e-05 and a batch size of 128 over 1 epoch, achieving a final validation loss of 0.9418. This model is intended for general text generation tasks, leveraging the Mistral architecture for efficient processing within a 4096 token context window.
Loading preview...
Overview
This model, zephyr-7b-sft-full, is a fine-tuned version of the mistralai/Mistral-7B-v0.1 base model. It features 7 billion parameters and was trained for 1 epoch, resulting in a validation loss of 0.9418. The training utilized a learning rate of 2e-05 and a total batch size of 128 across 8 GPUs.
Key Capabilities
- General Text Generation: As a fine-tuned language model, it is capable of generating human-like text based on given prompts.
- Mistral Architecture: Benefits from the efficient and performant architecture of the Mistral-7B-v0.1 base model.
Training Details
- Base Model:
mistralai/Mistral-7B-v0.1 - Learning Rate: 2e-05
- Optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
- Epochs: 1.0
- Distributed Training: Multi-GPU setup with 8 devices and a gradient accumulation of 2 steps.
Good for
- Exploratory Text Generation: Suitable for experimenting with text generation tasks based on a fine-tuned Mistral model.
- Further Fine-tuning: Can serve as a base for additional domain-specific fine-tuning due to its established training process.