Model Overview

fblgit/juanako-7b-UNA is a 7 billion parameter language model developed by Xavier M. at fblgit. It is a fine-tuned version of fblgit/juanako-7b-UNA-v2-phase-1, trained on the HuggingFaceH4/ultrafeedback_binarized dataset. A key differentiator of this model is its use of Uniform Neural Alignment (UNA), a proprietary training technique designed to improve alignment between transformer layers.

Performance Highlights

This model demonstrates competitive performance, achieving an average score of 59.91 on the HuggingFace LLM Leaderboard, surpassing many current Mistral-based models. Notable scores include:

• ARC (25-s): 68.17
• HellaSwag (10-s): 85.34
• TruthfulQA (MC) (0-s): 65.13
• Winogrande (5-s): 78.85

Training Details

The model was trained with a learning rate of 0.0001, using an Adam optimizer and a linear learning rate scheduler. It underwent 1 epoch of training with a total batch size of 224 across 14 GPUs.

Recommended Use Cases

Given its strong benchmark results, particularly in reasoning and truthfulness, juanako-7b-UNA is suitable for applications requiring robust general language understanding and generation. Its performance on tasks like ARC and TruthfulQA suggests its utility in question-answering and factual recall scenarios.