Hugging Face Overview
Overview
Deathsquad10/TinyLlama-Remix is a 1.1 billion parameter language model built upon the TinyLlama project, which aims to pretrain a 1.1B Llama model on 3 trillion tokens. This specific model is a chat-finetuned variant, leveraging the same architecture and tokenizer as Llama 2, ensuring compatibility with many open-source projects.
Key Capabilities & Training
- Compact Size: With only 1.1 billion parameters, it is designed for applications requiring a restricted computation and memory footprint.
- Llama 2 Compatibility: Adopts the exact architecture and tokenizer of Llama 2, allowing for seamless integration into existing Llama-based workflows.
- Instruction-Tuned: Fine-tuned following the Zephyr training recipe, initially on a variant of the UltraChat dataset for diverse synthetic dialogues.
- DPO Alignment: Further aligned using 🤗 TRL's
DPOTraineron the openbmb/UltraFeedback dataset, which includes 64k prompts and GPT-4 ranked model completions.
Performance Benchmarks
The model's performance on various tasks is provided, including:
- ARC Challenge: 26.19% acc, 28.92% acc_norm
- ARC Easy: 47.77% acc, 44.61% acc_norm
- BoolQ: 62.97% acc
- HellaSwag: 39.34% acc, 49.30% acc_norm
- OpenBookQA: 21.20% acc, 32.60% acc_norm
- PIQA: 69.15% acc, 68.77% acc_norm
- WinoGrande: 57.14% acc
Additionally, MMLU and CMMLU evaluations show average scores around 26.29% and 24.98% respectively, across various categories like STEM, Social Sciences, Humanities, and Other.
Good For
- Chatbot Applications: Specifically fine-tuned for conversational tasks.
- Resource-Constrained Environments: Its compact size makes it suitable for deployment where computational and memory resources are limited.
- Llama Ecosystem Integration: Benefits from its Llama 2-compatible architecture for easy adoption.