atsumoto/phi-1_5-alpaca-cleaned
atsumoto/phi-1_5-alpaca-cleaned is a 1.4 billion parameter instruction-tuned causal language model based on Microsoft's phi-1_5 architecture. It was fine-tuned using full parameter training on the yahma/alpaca-cleaned dataset. This model is optimized for following instructions and generating responses in an Alpaca-like format, making it suitable for general-purpose conversational AI tasks.
Loading preview...
Overview
This model, atsumoto/phi-1_5-alpaca-cleaned, is an instruction-tuned variant of Microsoft's phi-1_5 model, featuring 1.4 billion parameters and a 2048 token context length. It was developed by atsumoto through full parameter fine-tuning, rather than LoRA, on the yahma/alpaca-cleaned dataset.
Key Capabilities
- Instruction Following: Designed to accurately follow instructions and generate appropriate responses, as demonstrated by its training on the Alpaca-cleaned dataset.
- Alpaca-style Interaction: Produces output formatted similarly to the Alpaca instruction-following models, making it suitable for tasks requiring clear instruction-response pairs.
- Compact Size: At 1.4 billion parameters, it offers a balance between performance and computational efficiency, making it accessible for various deployment scenarios.
Training Details
The model underwent full parameter fine-tuning over 3 epochs using 8 A6000 48GB GPUs. Key training parameters included a per_device_train_batch_size of 8, gradient_accumulation_steps of 8, and a learning_rate of 2e-5. The training utilized pytorch 2.1.2, transformers 4.38.0.dev0, accelerate 0.26.1, and deepspeed 0.13.1.