vmajor/Orca2-13B-selfmerge-26B
vmajor/Orca2-13B-selfmerge-26B is a 13 billion parameter language model derived from merging the Microsoft Orca-2-13B model with itself using 'mergekit-legacy'. This self-merged model demonstrates marginal perplexity improvement and notably more than doubles the original model's performance on the GSM8K benchmark, indicating enhanced mathematical reasoning. It is designed for tasks requiring improved logical and arithmetic capabilities over its base model, maintaining a 4096-token context length.
Loading preview...
Model Overview
vmajor/Orca2-13B-selfmerge-26B is a 13 billion parameter language model created by applying a self-merge operation to the microsoft/Orca-2-13b model using mergekit-legacy with specific parameters (--weight 0.5 --density 0.5). This merging technique aims to enhance model performance through architectural recombination.
Key Performance Improvements
This self-merged model shows a slight improvement in perplexity, moving from 7.595 to 7.550. More significantly, benchmark results indicate a substantial gain in mathematical reasoning:
- GSM8K: Performance more than doubled, increasing from 17.97 to 39.2.
- Overall Average: The model achieved an average score of 62.24 across various benchmarks, compared to the base model's 58.64.
Other benchmarks like ARC, HellaSwag, MMLU, TruthfulQA, and Winogrande show minor changes, with some slight improvements and some negligible dips. The most notable differentiator is the significant boost in GSM8K, suggesting enhanced arithmetic and logical problem-solving capabilities.
Use Cases
This model is particularly well-suited for applications where improved mathematical reasoning and problem-solving are critical, building upon the strong foundation of the Orca-2-13B architecture.