OpenLLM-France/luciole-ablation-1B-fr1.0

Luciole French-English Ablation Models: French-Only Variant

OpenLLM-France/luciole-ablation-1B-fr1.0 is a 1 billion parameter decoder-only language model developed by LINAGORA for the OpenLLM France project. This specific model is trained exclusively on 100 billion French tokens from the FineWeb-2 dataset, using a Luciole tokenizer with a 128,000 vocabulary size. It adopts a Llama 3.2 1B architecture with a sequence length of 2048 tokens.

Key Characteristics

• Research Focus: Primarily intended for research into the impact of language proportions on multilingual performance, as detailed in the paper "EIFFEL: a novel benchmark to measure bias of English heavy training on French idiomatic expressions."
• French-Only Training: Unlike other models in the Luciole ablation series, this variant is trained solely on French data, providing a baseline for language-specific studies.
• Architecture: Utilizes a Llama 3.2 1B architecture with 16 layers, 32 attention heads, and a hidden size of 2048, trained on 64 H100 GPUs.
• Intermediate Checkpoints: Publicly available intermediate checkpoints facilitate interpretability studies.

Intended Use

• Research Purposes: Ideal for academic research on language model behavior, bias, and the effects of training data composition.
• Linguistic Studies: Useful for analyzing model performance and characteristics when exposed only to French data.

Limitations

• Research-Oriented: Not optimized for general downstream use cases or fine-tuning in standard LLM pipelines.
• Data Quality: Trained on web data without extensive cleaning, making it susceptible to harmful and biased content.