soyrsoyr/erebus-v2-1.5b-instruct
soyrsoyr/erebus-v2-1.5b-instruct is a 1.5 billion parameter instruction-tuned chat model developed by soyrsoyr, fine-tuned from erebus-v2-1.5b-base on the SmolTalk dataset. This model is designed for instruction following, offering a compact solution for conversational AI tasks. It provides a balance between model size and interactive capabilities, suitable for applications where smaller models are preferred.
Loading preview...
Model Overview
soyrsoyr/erebus-v2-1.5b-instruct is a 1.5 billion parameter instruction-following chat model, fine-tuned by soyrsoyr from its base model, erebus-v2-1.5b-base. The instruction tuning was performed using the HuggingFaceTB/smoltalk dataset, comprising approximately 1 million examples, over a single epoch.
Training Details
- Base Model:
erebus-v2-1.5b-base, pretrained on 5.5 billion tokens. - SFT Dataset: HuggingFaceTB/smoltalk, used for instruction fine-tuning.
- Training Duration: 34.5 hours on 4x A100-SXM4-80GB GPUs.
Key Capabilities & Limitations
- Instruction Following: Designed to respond to user instructions effectively.
- Code Generation: Capable of producing simple correct functions, though explanations may suffer from repetition.
- Repetition: The model exhibits a tendency to repeat phrases, especially in longer outputs. Using
repetition_penalty=1.2is recommended to mitigate this. - Reasoning: Due to a smaller pretraining budget, its multi-step mathematical reasoning is weak, with a GSM8K score of approximately 1.4% (flexible) / 0.08% (strict).
- Output Control: May not always know when to stop generating, leading to verbose or looping responses.
Use Cases
This model is suitable for applications requiring a compact, instruction-following chatbot where computational resources are a consideration. It can handle basic conversational tasks and simple code generation, provided its limitations regarding repetition and complex reasoning are managed.