AdvertLlama-7b by Around6827 is a 7 billion parameter language model fine-tuned from NousResearch/Llama-2-7b-hf. This model was trained using 8-bit quantization and a cosine learning rate schedule over 3 epochs. While specific use cases and training data are not detailed, its Llama-2 base suggests general language understanding and generation capabilities.
Loading preview...
AdvertLlama-7b Overview
AdvertLlama-7b is a 7 billion parameter language model developed by Around6827, fine-tuned from the NousResearch/Llama-2-7b-hf base model. This model was trained using the Axolotl framework, indicating a focus on efficient fine-tuning processes.
Training Details
The model underwent a fine-tuning procedure utilizing bitsandbytes 8-bit quantization (load_in_8bit: True), which helps reduce memory footprint during training. Key training hyperparameters included a learning rate of 0.0002, a batch size of 2 (with 4 gradient accumulation steps for an effective batch size of 32), and a cosine learning rate scheduler with 10 warmup steps. The training was conducted over 3 epochs on a multi-GPU setup.
Key Characteristics
- Base Model: NousResearch/Llama-2-7b-hf
- Parameter Count: 7 billion
- Quantization: Trained with 8-bit quantization for efficiency.
- Training Framework: Built with Axolotl, a popular tool for LLM fine-tuning.
Intended Use Cases
Given its Llama-2 base and general fine-tuning approach, AdvertLlama-7b is likely suitable for a range of natural language processing tasks. However, without specific details on the fine-tuning dataset or intended applications, its specialized strengths remain undefined. Developers should evaluate its performance for their particular needs, especially for tasks requiring general text generation, summarization, or question answering.