occiglot/occiglot-7b-de-en-instruct

TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Mar 5, 2024License:apache-2.0Architecture:Transformer0.0K Open Weights Cold

Occiglot-7B-DE-EN-Instruct is a 7 billion parameter instruction-tuned causal decoder-only transformer language model developed by the Occiglot Research Collective. It supports German, English, and code, having been trained on 180M tokens of additional multilingual and code instructions. This model is optimized for instruction-following in both German and English contexts, building upon the occiglot-7b-eu5 base model.

Loading preview...

Occiglot-7B-DE-EN-Instruct Overview

Occiglot-7B-DE-EN-Instruct is a 7 billion parameter instruction-tuned language model from the Occiglot Research Collective, designed for robust performance in both German and English, alongside code generation. It is an instruct version of the occiglot-7b-eu5 model, further fine-tuned on 180 million tokens of multilingual and code instructions.

Key Capabilities

  • Bilingual Proficiency: Strong performance in German and English, making it suitable for applications requiring fluency in both languages.
  • Instruction Following: Enhanced ability to follow instructions due to dedicated instruction tuning.
  • Code Support: Includes support for code-related tasks, indicating its versatility beyond natural language.
  • Research-Oriented: Part of an ongoing open research project, with an invitation for collaborations on multilingual language models and evaluations.

Training and Data

The model was instruction-tuned from occiglot-7b-de-en using an 8xH100 setup, employing axolotl framework with bf16 precision. The training data was evenly split between German and English, incorporating datasets like Open-Hermes-2.5 (English and Code), DiscoLM German Dataset, OASST-2 (German subset), and Aya-Dataset (German subset).

Important Considerations

  • Safety Alignment: The model has not been safety-aligned and may produce problematic outputs.
  • Evaluation Nuances: Preliminary evaluation results, especially for non-English languages, are based on partially machine-translated datasets and English prompts, requiring cautious interpretation.

Finetunes
5 models