Llama-3-SEC: Domain-Specific Financial Analysis Model
Llama-3-SEC, developed by arcee-ai, is a 70 billion parameter large language model built on Meta-Llama-3-70B-Instruct, specifically designed for analyzing SEC (Securities and Exchange Commission) data. This release is an intermediate checkpoint, having undergone continual pre-training on 20 billion tokens of SEC filings data, alongside 1 billion tokens from Together AI's RedPajama dataset to balance domain expertise with general language understanding. The full model is still in training and will eventually be trained on 72 billion tokens of SEC data.
Key Capabilities
- Domain Specialization: Deep understanding of SEC filings and related financial information.
- Advanced Training: Utilizes Continual Pre-Training (CPT) with Megatron-Core and TIES merging technique from Arcee Mergekit, followed by supervised fine-tuning.
- Robust Evaluation: Demonstrated improvements in domain-specific perplexity and extractive numerical reasoning tasks (TAT-QA, ConvFinQA), while maintaining strong performance on general benchmarks like BIG-bench, AGIEval, GPT4all, and TruthfulQA.
- Chat Template: Trained using the chatml template for strong conversational abilities.
Good For
- In-depth Investment Analysis: Supporting decision-making with detailed financial insights.
- Risk Management: Assessing and identifying potential financial risks.
- Regulatory Compliance: Ensuring adherence to regulations and detecting violations.
- Corporate Governance: Studying practices and promoting transparency.
- Market Research: Tracking industry trends and conducting sector-specific analysis.