kyujinpy/PlatYi-34B-Llama-Q-v3
PlatYi-34B-Llama-Q-v3 by Kyujin Han (kyujinpy) is a 34 billion parameter auto-regressive language model built on the Yi-34B transformer architecture, fine-tuned using Q-LoRA on the Open-Platypus dataset. This model features a 32768 token context length and is designed for general text generation tasks. It aims to improve upon previous PlatYi versions by addressing identified bugs in templates and warmup steps, achieving an average score of 61.15 on the Open LLM Leaderboard.
Loading preview...
PlatYi-34B-Llama-Q-v3 Overview
PlatYi-34B-Llama-Q-v3 is a 34 billion parameter auto-regressive language model developed by Kyujin Han (kyujinpy). It is based on the Yi-34B transformer architecture and was fine-tuned using Q-LoRA with a lora_r value of 64. The model leverages the chargoddard/Yi-34B-Llama as its base and was trained on the garage-bAInd/Open-Platypus dataset.
Key Characteristics & Improvements
- Architecture: Built upon the robust Yi-34B transformer, known for its strong performance in large language models.
- Fine-tuning: Utilizes Q-LoRA for efficient fine-tuning, allowing for significant performance gains with reduced computational overhead.
- Bug Fixes: This v3 iteration specifically addresses and corrects identified mistakes in previous model templates and warmup steps, aiming for improved stability and performance.
- Context Length: Supports a substantial context window of 32768 tokens, enabling processing of longer inputs and generating more coherent, extended outputs.
Performance Benchmarks
On the Hugging Face Open LLM Leaderboard, PlatYi-34B-Llama-Q-v3 achieved an average score of 61.15. Specific benchmark results include:
- ARC: 64.33
- HellaSwag: 84.88
- MMLU: 74.98
- TruthfulQA: 51.80
- Winogrande: 82.79
- GSM8K: 6.67
While showing strong performance in areas like MMLU and HellaSwag, its GSM8K score indicates potential limitations in complex mathematical reasoning compared to some other models in its class. This model represents an iterative improvement over its predecessors, focusing on refining the training process and model stability.