louisbrulenaudet/Romulus-cpt-Llama-3.1-8B-v0.1
Romulus-cpt-Llama-3.1-8B-v0.1 by louisbrulenaudet is a continually pre-trained Llama-3.1-8B model specifically enriched with French legal data. This 8 billion parameter model, with a 4096 token context length, is designed to serve as a foundational base for fine-tuning on labeled French legal tasks. It is optimized for specialized applications in French law, requiring further alignment for direct text generation.
Loading preview...
Romulus: Continually Pre-trained for French Law
Romulus is a series of models developed by louisbrulenaudet, based on the meta-llama/Meta-Llama-3.1-8B architecture. This specific version, Romulus-cpt-Llama-3.1-8B-v0.1, has undergone continual pre-training with a corpus of approximately 34.8 million tokens of French legal data. The model is intended to provide a strong foundation for downstream fine-tuning tasks within the domain of French law.
Key Characteristics
- Base Model: Meta-Llama-3.1-8B, an 8 billion parameter model.
- Specialization: Continually pre-trained on French legal texts.
- Context Length: Utilizes a maximum sequence length of 4096 tokens.
- Training Method: Trained using Unsloth with LoRA adapters (r=128, lora_alpha=32) on a Nvidia H100 Azure instance.
- Purpose: Designed as a base model for further fine-tuning on specific French legal tasks, rather than for direct text generation.
When to Use This Model
- Fine-tuning for French Legal Applications: Ideal for developers looking to build specialized AI applications for French law, such as legal document analysis, contract review, or legal research.
- Research in Legal NLP: Useful for researchers exploring the impact of continual pre-training on domain-specific language models, particularly in the legal sector.
- Developing Custom Legal LLMs: Serves as an excellent starting point for creating highly accurate and context-aware language models tailored to the nuances of French legal language.