vorenthiclabs/Vorenthos-r1
Vorenthos-r1 is a 1.5 billion parameter causal language model developed by vorenthiclabs, fine-tuned from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B. It utilizes QLoRA for efficient fine-tuning and is designed to follow a specific prompt format including a chain-of-thought reasoning section. This model is suitable for tasks requiring structured responses and explicit reasoning steps within its 32768 token context length.
Loading preview...
Vorenthos-r1: A Fine-Tuned DeepSeek-R1-Distill-Qwen-1.5B Model
Vorenthos-r1 is a 1.5 billion parameter language model developed by vorenthiclabs, built upon the deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B architecture. It was fine-tuned using the efficient QLoRA method (r=16, alpha=16) and saved in float16 precision after merging. The training involved 1 epoch with a learning rate of 0.0002, utilizing a maximum sequence length of 2048 and an effective batch size of 8.
Key Characteristics
- Base Model: Fine-tuned from
deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B. - Efficient Fine-tuning: Leverages Unsloth and QLoRA for optimized training.
- Prompt Format: Designed to work with a specific prompt structure that includes a
<think>block for chain-of-thought reasoning, enabling more structured and explicit response generation. - Context Length: Supports a substantial context length of 32768 tokens, allowing for processing longer inputs and generating more comprehensive outputs.
Ideal Use Cases
- Structured Reasoning Tasks: Excellent for applications where explicit step-by-step reasoning or internal thought processes are beneficial.
- Instruction Following: Optimized for tasks requiring precise adherence to instructions and output formats due to its fine-tuning and prompt structure.
- Resource-Efficient Deployment: As a 1.5B parameter model, it offers a balance of capability and efficiency, suitable for environments with limited computational resources.