Tensoic/Llama-2-7B-alpaca-2k-test-merged
Tensoic/Llama-2-7B-alpaca-2k-test-merged is a 7 billion parameter Llama-2-based causal language model fine-tuned by Tensoic. It was fine-tuned using peft-LORA on the alpaca_2k_test dataset, making it suitable for instruction-following tasks. The model leverages an extended context length of 4096 tokens and was trained with 8-bit quantization for efficiency.
Loading preview...
Model Overview
Tensoic/Llama-2-7B-alpaca-2k-test-merged is a 7 billion parameter language model derived from Meta's Llama-2-7b-hf base model. It has been fine-tuned by Tensoic using the PEFT (Parameter-Efficient Fine-Tuning) LoRA method on the henrichsen/alpaca_2k_test dataset. This fine-tuning process aims to enhance the model's ability to follow instructions and generate responses aligned with the Alpaca instruction format.
Key Characteristics
- Base Model: Llama-2-7b-hf
- Parameter Count: 7 billion
- Fine-tuning Method: LoRA (Low-Rank Adaptation) with
lora_r: 32andlora_alpha: 16 - Dataset:
henrichsen/alpaca_2k_test - Context Length: Configured for a sequence length of 4096 tokens
- Quantization: Trained with
load_in_8bit: trueusingbitsandbytesfor memory efficiency.
Training Details
The model was trained for 3 epochs on 8x NVIDIA V100 GPUs (32GB each) with a micro batch size of 2 and a learning rate of 0.0002. Gradient accumulation steps were set to 4, and gradient_checkpointing was enabled. The training utilized fp16 precision and xformers_attention for optimized performance.