gold24k/v10
gold24k/v10 is a 35.1 billion parameter BF16 checkpoint derived from tojointhecommunity/affine-5efg6cm3yl-king. It integrates a scaled selective-fallback LoRA, trained on preserved positive turns and sanitized negative turns, without requiring a runtime router or PEFT adapter. This model is designed for specific task-oriented applications where high-confidence responses are critical. It offers a standalone solution for integrating fine-tuned capabilities directly.
Loading preview...
Model Overview
gold24k/v10 is a 35.1 billion parameter BF16 checkpoint, representing a merged version of tojointhecommunity/affine-5efg6cm3yl-king. This model incorporates a scaled selective-fallback LoRA adapter, which was trained to enhance performance on specific tasks by preserving positive turns and utilizing sanitized, task-specific alternatives for high-confidence negative turns.
Key Characteristics
- Architecture: Merged BF16 checkpoint from
tojointhecommunity/affine-5efg6cm3yl-king. - LoRA Integration: Applies a scaled selective-fallback LoRA (r16, alpha32, dropout0.0) across multiple projection layers (
q_proj,k_proj,v_proj,o_proj,in_proj_qkv,in_proj_z,in_proj_a,in_proj_b,out_proj). - Deployment Simplicity: Designed to operate without the need for a runtime router, custom Python code, or a PEFT adapter, simplifying integration.
- Training Objective: Optimized using DPO (Direct Preference Optimization) with a beta of 0.2 and a learning rate of 3.0e-08 over 1.0 epoch, utilizing a context of 8192 tokens during training.
Intended Use Cases
This model is an experimental candidate suitable for applications requiring a fine-tuned language model with enhanced control over response quality, particularly in scenarios where high-confidence positive and carefully managed negative turns are desired. It is ideal for developers looking for a standalone, pre-merged solution for specific conversational or task-oriented AI systems.