pszemraj/Mistral-v0.3-6B

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kTool Calling:SupportedPublished:May 25, 2024License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Cold

pszemraj/Mistral-v0.3-6B is a 7 billion parameter Mistral-based causal language model, developed by pszemraj, that has undergone continued pretraining at a 4096 token context length. This model was specifically trained to 'heal' the effects of layer-pruning from its base model, pszemraj/Mistral-7B-v0.3-prune6. It demonstrates competitive performance on various benchmarks, including ARC-Easy and BoolQ, making it suitable for general language understanding and reasoning tasks.

Loading preview...

Model Overview

pszemraj/Mistral-v0.3-6B is a 7 billion parameter language model built upon the Mistral architecture. It is a fine-tuned iteration of pszemraj/Mistral-7B-v0.3-prune6, specifically undergoing continued pretraining to mitigate the impact of prior layer-pruning. The training was conducted with a sequence length of 4096 tokens, utilizing the BEE-spoke-data/knowledge-inoc-concat-v1 dataset.

Key Capabilities & Performance

This model exhibits solid performance across several zero-shot evaluation benchmarks:

  • ARC-Easy: Achieves an accuracy of 0.7109.
  • BoolQ: Demonstrates an accuracy of 0.7930.
  • Lambada OpenAI: Records a perplexity of 4.9892 and an accuracy of 0.6746.
  • PIQA: Shows an accuracy of 0.7350.

On the Open LLM Leaderboard, the model achieves an average score of 49.23, with notable results in HellaSwag (71.65) and Winogrande (72.77). The training process involved 2000 steps with a learning rate of 2e-05 and a total batch size of 16, using a paged_adamw_32bit optimizer.

Use Cases

This model is well-suited for applications requiring:

  • General language understanding: Its performance on benchmarks like BoolQ and ARC-Easy suggests proficiency in comprehending and responding to various queries.
  • Reasoning tasks: The model's design and evaluation indicate capabilities in logical inference and problem-solving within its context window.
  • Integration into resource-constrained environments: As a pruned and then 'healed' model, it may offer a balance of performance and efficiency for certain deployments.

Finetunes
3 models