sudipto-ducs/InLegalLLaMA
InLegalLLaMA is a 7 billion parameter causal language model developed by sudipto-ducs, fine-tuned from Meta's Llama-2-7b-hf architecture. This model specializes in legal domain tasks, having been trained on specific legal datasets including inlegalllama-laws and inlegalllama-sci. It is designed for applications requiring legal text understanding and generation, offering a 4096-token context length.
Loading preview...
InLegalLLaMA Overview
InLegalLLaMA is a specialized 7 billion parameter language model, fine-tuned by sudipto-ducs from the meta-llama/Llama-2-7b-hf base model. Its primary distinction lies in its domain-specific training, which focused on legal datasets, specifically inlegalllama-laws and inlegalllama-sci. This targeted fine-tuning aims to enhance its performance and relevance for legal applications.
Key Capabilities
- Legal Domain Specialization: Optimized for tasks within the legal field due to its training on specific legal datasets.
- Llama-2 Architecture: Benefits from the robust and widely recognized Llama-2 base architecture.
- 7 Billion Parameters: Offers a balance of performance and computational efficiency for legal text processing.
Training Details
The model was trained using the following hyperparameters:
- Learning Rate: 0.0003
- Batch Size: 2 (train), 8 (eval) with 8 gradient accumulation steps, totaling 16.
- Optimizer: Adam with standard betas and epsilon.
- Scheduler: Cosine learning rate scheduler with 2000 warmup steps.
- Epochs: 3.0
- Mixed Precision: Utilized Native AMP for efficient training.
Intended Use Cases
While specific intended uses and limitations require further definition, its legal domain training suggests applicability in areas such as legal document analysis, legal research assistance, and generating legally-relevant text.