Duxiaoman-DI/Llama3-XuanYuan3-70B
The XuanYuan3-70B is a 70 billion parameter Llama3-based large language model developed by Duxiaoman-DI, specifically designed for financial applications. It features a 16k context length and excels in financial event interpretation, business analysis, investment research, and compliance/risk management. The model demonstrates performance comparable to GPT-4o in financial benchmarks, surpassing other open-source and some closed-source models in specialized financial tasks.
Loading preview...
XuanYuan3-70B: A Financial Domain LLM
Duxiaoman-DI's XuanYuan3-70B is the third generation of their large language models, built upon the Llama3-70B architecture. This 70 billion parameter model is specifically engineered to address challenges in financial applications, leveraging extensive incremental pre-training on both Chinese and English corpora, followed by high-quality instruction tuning and reinforcement learning alignment.
Key Capabilities
- Financial Event Interpretation: Provides in-depth analysis of financial events using professional terminology, aligning with expert human logic.
- Financial Business Analysis: Offers robust business analysis capabilities, summarizing and extracting information with financial expert-level precision.
- Investment Research: Generates insightful research reports, moving beyond simple data presentation to provide deep analysis and multi-dimensional expansion.
- Compliance & Risk Management: Adheres to financial compliance requirements, accurately identifying and analyzing risks to offer legally sound advice.
- Extended Context: Supports a 16k context length, suitable for long-form financial report analysis and financial Agent development.
Technical Innovations
Compared to its predecessor, XuanYuan3-70B incorporates several innovations:
- Refined Data Organization: Employs sophisticated data organization and dynamic control strategies during incremental pre-training and SFT, enhancing Chinese processing and financial understanding while maintaining strong English performance.
- Omni-directional Financial Reward Model (UFRM): Developed a UFRM pre-trained for general preference alignment and fine-tuned with high-quality financial data, utilizing contrastive learning and inverse reinforcement learning to boost financial preference learning.
- Iterative Reinforcement Training (PEI-RLHF): Implements an iterative "pre-train-evaluate-improve" RLHF method to optimize model alignment with human expectations and further enhance financial performance, reducing alignment tax.
Performance
In financial scenario evaluations, XuanYuan3-70B-Chat demonstrates overall performance comparable to GPT-4o, outperforming recent Chinese open-source models. It specifically surpasses closed-source models in dimensions like financial compliance, risk management, investment research, and event interpretation. The model's financial business capabilities are particularly strong, outperforming other 72B open-source models across eight key financial task categories.