XuanYuan3-70B: A Financial Domain LLM
Duxiaoman-DI's XuanYuan3-70B is the third generation of their large language models, built upon the Llama3-70B architecture. This 70 billion parameter model is specifically engineered to address challenges in financial applications, leveraging extensive incremental pre-training on both Chinese and English corpora, followed by high-quality instruction tuning and reinforcement learning alignment.
Key Capabilities
- Financial Event Interpretation: Provides in-depth analysis of financial events using professional terminology, aligning with expert human logic.
- Financial Business Analysis: Offers robust business analysis capabilities, summarizing and extracting information with financial expert-level precision.
- Investment Research: Generates insightful research reports, moving beyond simple data presentation to provide deep analysis and multi-dimensional expansion.
- Compliance & Risk Management: Adheres to financial compliance requirements, accurately identifying and analyzing risks to offer legally sound advice.
- Extended Context: Supports a 16k context length, suitable for long-form financial report analysis and financial Agent development.
Technical Innovations
Compared to its predecessor, XuanYuan3-70B incorporates several innovations:
- Refined Data Organization: Employs sophisticated data organization and dynamic control strategies during incremental pre-training and SFT, enhancing Chinese processing and financial understanding while maintaining strong English performance.
- Omni-directional Financial Reward Model (UFRM): Developed a UFRM pre-trained for general preference alignment and fine-tuned with high-quality financial data, utilizing contrastive learning and inverse reinforcement learning to boost financial preference learning.
- Iterative Reinforcement Training (PEI-RLHF): Implements an iterative "pre-train-evaluate-improve" RLHF method to optimize model alignment with human expectations and further enhance financial performance, reducing alignment tax.
Performance
In financial scenario evaluations, XuanYuan3-70B-Chat demonstrates overall performance comparable to GPT-4o, outperforming recent Chinese open-source models. It specifically surpasses closed-source models in dimensions like financial compliance, risk management, investment research, and event interpretation. The model's financial business capabilities are particularly strong, outperforming other 72B open-source models across eight key financial task categories.