OpenLLM-France/luciole-ablation-1B-en0.99-fr0.01
OpenLLM-France/luciole-ablation-1B-en0.99-fr0.01 is a 1 billion parameter decoder-only language model developed by LINAGORA as part of the OpenLLM France project. This model is one of a collection of Luciole ablation models specifically designed to research the impact of varying English and French language proportions on multilingual performance. It is trained on 100 billion tokens with a Llama 3.2 1B architecture and a 2048 token sequence length, primarily for studying language proportion effects rather than direct downstream application.
Loading preview...
Model Overview
OpenLLM-France/luciole-ablation-1B-en0.99-fr0.01 is a 1 billion parameter decoder-only language model from the Luciole family, developed by LINAGORA under the OpenLLM France project. It is part of a series of ablation models specifically created to investigate how different proportions of English (99%) and French (1%) in the training data affect multilingual performance, as detailed in the paper "EIFFEL: a novel benchmark to measure bias of English heavy training on French idiomatic expressions".
Key Characteristics
- Architecture: Llama 3.2 1B architecture with 16 layers, 32 attention heads, and a hidden size of 2048.
- Training Data: Trained on 100 billion tokens, using random samples from FineWeb (English) and FineWeb-2 (French) datasets. The data was preprocessed with retrospective robots.txt rules.
- Tokenization: Utilizes the Luciole tokenizer with a 128,000 vocabulary size, trained on a multilingual corpus including French, English, Arabic, and programming languages.
- Context Length: Features a sequence length of 2048 tokens during training.
- Research Focus: Primarily intended for research into language proportion impact on multilingual models, not for general-purpose fine-tuning or direct application in standard LLM pipelines.
Intended Use
This model is strictly for research purposes to study the effects of language proportions on benchmark performance. It is not optimized for downstream use cases and should not be fine-tuned or deployed in production environments. The developers also provide intermediate checkpoints to facilitate interpretability studies. Users should be aware that the training data was not extensively cleaned and may contain harmful or biased content.