HerrHruby/MR_midtrain_9B_v4_condgen
MR_midtrain_9B_v4_condgen is a 9 billion parameter model from HerrHruby, based on the MR_midtrain_9B_v4 architecture, with a 32768 token context length. This version specifically re-keys the vision tower to resolve compatibility issues with Megatron training frameworks. It is designed for use in RL/Megatron training environments, while the original layout remains suitable for inference with standard tools like transformers and vLLM.
Loading preview...
Overview
HerrHruby/MR_midtrain_9B_v4_condgen is a specialized version of the MR_midtrain_9B_v4 model, featuring 9 billion parameters and a 32768 token context length. Its primary distinction lies in the re-keying of its vision tower from model.language_model.visual.* to model.visual.*. This modification ensures compatibility with verl's Megatron bridge (mbridge), which previously encountered KeyError issues with the original key layout during training.
Key Capabilities
- Megatron Training Compatibility: Specifically engineered to enable seamless integration and training within Megatron frameworks, resolving key resolution errors.
- Byte-Identical Weights: Despite the key name changes, the model's weights are byte-identical to the original
MR_midtrain_9B_v4, ensuring no performance degradation or alteration in model behavior. - Inference Flexibility: While optimized for Megatron training, this model, like its predecessor, remains fully compatible for inference with standard libraries such as Hugging Face Transformers (v5.5.4) and vLLM.
Good For
- RL/Megatron Training: This model is explicitly recommended for reinforcement learning (RL) and Megatron-based training workflows where the original key layout caused issues.
- Developers using Megatron Bridge: Ideal for developers utilizing verl's Megatron bridge who require a compatible model for their training setups.
This version reconciles against the working v3 export with zero unmatched keys, providing a stable base for advanced training paradigms.