Ollam311/Qwen2.5-1.5B

Qwen2.5-1.5B Overview

Ollam311/Qwen2.5-1.5B is a 1.54 billion parameter base causal language model from the Qwen2.5 series, developed by Qwen. This model builds upon the Qwen2 architecture, incorporating improvements across several key areas. It is designed for pretraining and serves as a foundation for further fine-tuning, such as SFT or RLHF, rather than direct conversational applications.

Key Capabilities & Improvements

• Enhanced Knowledge & Reasoning: Significantly improved capabilities in coding and mathematics, leveraging specialized expert models.
• Instruction Following: Offers substantial improvements in adhering to instructions and generating structured outputs, including JSON.
• Long-Context & Generation: Supports a context length of up to 32,768 tokens and can generate texts up to 8,000 tokens.
• Multilingual Support: Provides robust support for over 29 languages, including major global languages like Chinese, English, French, Spanish, German, and Japanese.
• Structured Data Understanding: Better at understanding and processing structured data, such as tables.
• Robustness: More resilient to diverse system prompts, which benefits role-play and chatbot condition-setting.

Model Architecture & Specifications

This base model utilizes a transformer architecture with RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings. It comprises 28 layers and 12 attention heads (GQA) for Q, with 2 for KV. The model has 1.54 billion parameters, with 1.31 billion non-embedding parameters.

Usage Recommendation

As a base language model, Qwen2.5-1.5B is intended for continued pretraining, supervised fine-tuning (SFT), or reinforcement learning from human feedback (RLHF). It is not recommended for direct use in conversational agents without further post-training. For detailed evaluation results and benchmarks, users are encouraged to refer to the official Qwen2.5 blog and GitHub repository.