Gensyn/Qwen2.5-0.5B-Instruct is an unmodified 0.49 billion parameter instruction-tuned causal language model from the Qwen2.5 family, featuring a 32,768 token context length. Developed by Qwen, this model utilizes a transformer architecture with RoPE, SwiGLU, and RMSNorm. It is specifically intended for local fine-tuning via peer-to-peer reinforcement learning within the Gensyn RL Swarm system. Its primary use case is as a base model for distributed RL training, after which it can be deployed in general workflows.
Loading preview...
Gensyn/Qwen2.5-0.5B-Instruct Overview
This model is an unmodified instruction-tuned version of the Qwen2.5-0.5B model, developed by Qwen. It is primarily designed for integration into the Gensyn RL Swarm for local, peer-to-peer reinforcement learning fine-tuning. After this specialized training, the model can be utilized in standard language model workflows.
Key Technical Specifications
- Architecture: Causal Language Model based on transformers, incorporating RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings.
- Parameters: 0.49 billion total parameters (0.36 billion non-embedding parameters).
- Context Length: Supports a full context length of 32,768 tokens, with a generation length of 8,192 tokens.
- Layers: Comprises 24 layers.
- Attention Heads: Features 14 attention heads for Q and 2 for KV (GQA).
Intended Use Case
- Gensyn RL Swarm Integration: The model's core purpose is to serve as a base for distributed fine-tuning within the Gensyn RL Swarm system. Developers can deploy it into a swarm and participate in the Gensyn Testnet for post-training reinforcement learning.
- General Workflows (Post-Finetuning): Once fine-tuned within the Gensyn ecosystem, the model can be used for various natural language processing tasks, similar to other instruction-tuned language models. For details on using the original model, refer to the original Qwen documentation.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.