gradientai/v-alpha-tross
gradientai/v-alpha-tross is a 69 billion parameter language model developed by Gradient, based on Llama-2-70B, specifically pre-trained and fine-tuned for finance applications. It excels in mathematical reasoning, tabular data understanding, and open-book retrieval and summarization within the financial domain. This model demonstrates performance comparable to Mixtral-8x7B-Instruct-v0.1 on H6 Average and GSM8K benchmarks, and matches GPT-3.5-turbo for extracting information from financial tabular data.
Loading preview...
Overview
gradientai/v-alpha-tross is an early, limited-capability version of Gradient's Albatross framework, a collection of domain-specific language models tailored for finance applications. This 69 billion parameter model is built upon meta-llama/Llama-2-70b-hf, undergoing additional finance-specific pre-training, fine-tuning, and instruction tuning. It is designed to showcase performance in key financial tasks.
Key Capabilities & Performance
- Mathematical Reasoning: Substantially outperforms Llama2-70B models on GSM8K, achieving scores similar to
mistralai/Mixtral-8x7B-Instruct-v0.1. - Tabular Understanding: Reaches
gpt-3.5-turboperformance in extracting information from tabular data, such as SEC filings. - Open-Book Retrieval & Summarization: Optimized for RAG and summarization tasks within the financial domain.
- Benchmarking: Achieves a H6 Average score of 73.28, surpassing Llama-2-70B variants and comparable to Mixtral-8x7B-Instruct-v0.1.
Training Strategy
The model's training involved further pre-training a Llama2-70B base on finance-specific data, curated using a novel data gathering approach that filters public repositories and involves human review. Supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) were applied, focusing on financial anchoring, mathematical reasoning, tabular understanding, conversational communication, and summarization. The DPO training utilized a learning rate of 5e-07 and a total batch size of 120 over 1 epoch.
Intended Use
v-alpha-tross serves as a demonstration of Gradient's Albatross framework. Users should follow Llama-2 chat formatting requirements, including INST, <<SYS>> tags, and <s> tokens, to achieve expected performance. It is important to note that this model has not been specifically aligned for safety and may produce problematic outputs.