formalmathatepfl/qwen3-8b-post-trained-cpt
The formalmathatepfl/qwen3-8b-post-trained-cpt is an 8 billion parameter language model fine-tuned from Qwen/Qwen3-8B. It was specifically trained on the lean_docs dataset, suggesting an optimization for tasks related to formal mathematics or document processing. This model is designed for specialized applications requiring understanding and generation within its fine-tuned domain, leveraging its 32768 token context length for handling extensive inputs.
Loading preview...
Model Overview
The formalmathatepfl/qwen3-8b-post-trained-cpt is an 8 billion parameter language model, derived from the base Qwen/Qwen3-8B architecture. This model has undergone a specific fine-tuning process using the lean_docs dataset, indicating a specialization in areas related to formal mathematics, theorem proving, or structured document analysis.
Key Capabilities
- Specialized Domain Understanding: Fine-tuned on
lean_docs, suggesting enhanced performance for tasks within formal mathematics or technical documentation. - Large Context Window: Inherits a 32768 token context length, enabling the processing of extensive documents and complex logical structures.
- Qwen3-8B Foundation: Benefits from the robust capabilities of the Qwen3-8B base model, providing a strong general language understanding before specialization.
Training Details
The model was trained with a learning rate of 5e-06, using a cosine learning rate scheduler with a 0.03 warmup ratio. Training involved 1 epoch across 8 GPUs, with a total batch size of 8. The optimizer used was AdamW_Torch. This configuration aims to adapt the base model effectively to the target domain.
Good For
- Applications requiring deep understanding and generation within formal mathematical contexts.
- Processing and analyzing structured technical documents, potentially related to theorem provers or formal verification.
- Research and development in AI for mathematics and logic.