bhenrym14/platypus-yi-34b
The bhenrym14/platypus-yi-34b is a 34 billion parameter instruction-tuned causal language model based on the Yi-34B architecture, specifically chargoddard/Yi-34B-Llama, which uses Llama2 model definitions and tokenizer. Fine-tuned with the Open-Platypus dataset, this model is optimized for instruction-following tasks. It leverages a QLoRA fine-tuning approach and is designed for general-purpose language generation and understanding, particularly in scenarios requiring instruction adherence.
Loading preview...
Overview
The bhenrym14/platypus-yi-34b is a 34 billion parameter instruction-tuned language model. It is built upon the chargoddard/Yi-34B-Llama base, which itself is a version of the 01-ai/Yi-34B model adapted to use Llama2 model definitions and tokenizer, thereby removing remote code requirements. The instruction tuning was performed using the garage-bAInd/Open-Platypus dataset.
Key Characteristics
- Base Model:
01-ai/Yi-34Bviachargoddard/Yi-34B-Llama. - Parameter Count: 34 billion parameters.
- Fine-tuning Method: QLoRA (rank 64) fine-tune, with the current checkpoint at 1 epoch.
- Training Data: Instruction-tuned with the
Open-Platypusdataset. - Tokenizer: Utilizes Llama2-compatible tokenizer and model definitions.
Usage and Prompting
This model can be used like any other Llama-2 model. It was trained with a legacy Airoboros <2.0 system prompt. For detailed prompting instructions, users can refer to the model card for bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-fp16.
Development Notes
The fine-tuning process was conducted on a single RTX 6000 Ada GPU, taking approximately 18 hours to reach the current checkpoint. The developer notes that the model might be undertrained at 1 epoch and suggests that further hyperparameter tuning could yield better performance.