PleIAs/Cassandre-RAG
PleIAs/Cassandre-RAG is an 8 billion parameter Llama-3.1-based model fine-tuned for Retrieval Augmented Generation (RAG) on French administrative documents, specifically focusing on school administration sources. It is designed to efficiently perform RAG tasks by expecting a predefined input structure and clearly citing specific excerpts and source documents in its generated answers. The model was trained using synthetic queries, BM25-retrieved documents, and generated responses to enhance its performance in this specialized domain. Its 32768 token context length supports processing substantial administrative texts.
Loading preview...
Cassandre-RAG: Specialized RAG for French Administrative Documents
Cassandre-RAG is an 8 billion parameter model built upon the Llama-3.1 architecture, specifically fine-tuned for Retrieval Augmented Generation (RAG) tasks. Its primary focus is on processing and generating responses based on French administrative documents, with a particular emphasis on school administration sources.
Key Capabilities
- Efficient RAG: Designed to handle RAG tasks very efficiently by adhering to a predefined input structure.
- Source Citation: Generates answers that clearly cite specific excerpts and the source documents used, ensuring traceability and accuracy.
- Specialized Training: Fine-tuned on a unique corpus comprising synthetic queries, documents retrieved via BM25, and generated answers, all derived from French administrative texts.
- Structured Input/Output: Expects a specific prompt format including "Query," "Source," and "Answer" sections, and outputs answers with embedded references using a
<ref text="[Quoted text from source]">[Source ID]</ref>format.
Training Details
The model underwent 3000 training steps with a learning rate of 3e-4 and a maximum sequence length of 8192. It utilized LoRA configuration with an alpha of 16, dropout of 0.1, and R of 64, targeting key attention and feed-forward modules. Quantization was applied using 4-bit nf4 type with float16 compute dtype.
Good For
- Applications requiring precise information retrieval and generation from French administrative documents.
- Systems needing verifiable answers with clear source attribution.
- Developers building RAG solutions for educational or public sector administrative contexts in France.