vmajor/Orca2-13B-selfmerge-39B
vmajor/Orca2-13B-selfmerge-39B is a 13 billion parameter language model derived from merging three Orca2-13B models. This self-merged model demonstrates improved perplexity scores and significantly enhanced performance on the GSM8K benchmark compared to the base Orca-2-13b. It is optimized for tasks requiring stronger mathematical reasoning, as indicated by its benchmark results.
Loading preview...
Model Overview
vmajor/Orca2-13B-selfmerge-39B is a 13 billion parameter language model created by self-merging three instances of the Orca2-13B model using 'mergekit-legacy'. This merging process involved specific weight and density parameters for the second merge, aiming to enhance model performance.
Key Capabilities & Performance
This self-merged model shows a marginal improvement in perplexity scores over the base Orca-2-13b. More notably, it achieves a significant boost in mathematical reasoning, with its GSM8K benchmark score more than doubling from 17.97 to 39.2 after the first self-merge. While the second self-merge (resulting in the 39B model) did not yield further gains, the overall benchmark average improved from 58.64 to 62.24. Performance across other benchmarks like ARC, HellaSwag, MMLU, TruthfulQA, and Winogrande remained largely consistent with the base model, with minor fluctuations.
When to Use This Model
This model is particularly well-suited for applications requiring enhanced mathematical problem-solving and reasoning capabilities, as evidenced by its strong GSM8K performance. Developers looking for a 13B parameter model with improved numerical understanding compared to the original Orca-2-13b may find this self-merged version beneficial.