Occiglot-7B-DE-EN-Instruct Overview
Occiglot-7B-DE-EN-Instruct is a 7 billion parameter instruction-tuned language model from the Occiglot Research Collective, designed for robust performance in both German and English, alongside code generation. It is an instruct version of the occiglot-7b-eu5 model, further fine-tuned on 180 million tokens of multilingual and code instructions.
Key Capabilities
- Bilingual Proficiency: Strong performance in German and English, making it suitable for applications requiring fluency in both languages.
- Instruction Following: Enhanced ability to follow instructions due to dedicated instruction tuning.
- Code Support: Includes support for code-related tasks, indicating its versatility beyond natural language.
- Research-Oriented: Part of an ongoing open research project, with an invitation for collaborations on multilingual language models and evaluations.
Training and Data
The model was instruction-tuned from occiglot-7b-de-en using an 8xH100 setup, employing axolotl framework with bf16 precision. The training data was evenly split between German and English, incorporating datasets like Open-Hermes-2.5 (English and Code), DiscoLM German Dataset, OASST-2 (German subset), and Aya-Dataset (German subset).
Important Considerations
- Safety Alignment: The model has not been safety-aligned and may produce problematic outputs.
- Evaluation Nuances: Preliminary evaluation results, especially for non-English languages, are based on partially machine-translated datasets and English prompts, requiring cautious interpretation.