fblgit/juanako-7b-UNA
The fblgit/juanako-7b-UNA model is a 7 billion parameter language model developed by Xavier M. at fblgit, fine-tuned from fblgit/juanako-7b-UNA-v2-phase-1 on the HuggingFaceH4/ultrafeedback_binarized dataset. It utilizes Uniform Neural Alignment (UNA), a novel training technique, to enhance alignment between transformer layers. This model demonstrates strong performance across various benchmarks, achieving an average score of 59.91 on the HuggingFace LLM Leaderboard, outperforming many Mistral-based models in its class.
Loading preview...
Model Overview
fblgit/juanako-7b-UNA is a 7 billion parameter language model developed by Xavier M. at fblgit. It is a fine-tuned version of fblgit/juanako-7b-UNA-v2-phase-1, trained on the HuggingFaceH4/ultrafeedback_binarized dataset. A key differentiator of this model is its use of Uniform Neural Alignment (UNA), a proprietary training technique designed to improve alignment between transformer layers.
Performance Highlights
This model demonstrates competitive performance, achieving an average score of 59.91 on the HuggingFace LLM Leaderboard, surpassing many current Mistral-based models. Notable scores include:
- ARC (25-s): 68.17
- HellaSwag (10-s): 85.34
- TruthfulQA (MC) (0-s): 65.13
- Winogrande (5-s): 78.85
Training Details
The model was trained with a learning rate of 0.0001, using an Adam optimizer and a linear learning rate scheduler. It underwent 1 epoch of training with a total batch size of 224 across 14 GPUs.
Recommended Use Cases
Given its strong benchmark results, particularly in reasoning and truthfulness, juanako-7b-UNA is suitable for applications requiring robust general language understanding and generation. Its performance on tasks like ARC and TruthfulQA suggests its utility in question-answering and factual recall scenarios.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.