arcee-ai/MyAlee-Qwen-Instruct-v2-16k-v1

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7.6BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Jun 11, 2024License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

MyAlee-Qwen-Instruct-v2-16k-v1 is a 7.6 billion parameter instruction-tuned language model developed by arcee-ai, based on the Qwen2-7B architecture. This model is fine-tuned with a substantial context length of 32768 tokens, making it suitable for processing longer inputs and complex conversational tasks. It leverages a chatml template and was trained using specific educational and conversational datasets, indicating its potential for applications requiring detailed instructional responses or extended dialogue.

Loading preview...

Model Overview

arcee-ai/MyAlee-Qwen-Instruct-v2-16k-v1 is a 7.6 billion parameter language model fine-tuned from the Qwen/Qwen2-7B base model. This iteration, built with Axolotl, is designed for instruction-following tasks, utilizing a chatml template for conversational interactions. A key feature is its extended context window, configured for a sequence_len of 16384 tokens during training, which implies strong performance with longer inputs, though the model card indicates a 32768 token context length.

Training Details

The model was trained using a combination of datasets: arcee-ai/MyAlee-Education-Instructions-V2 (ShareGPT format) and Crystalcareai/Orca-Reka (Alpaca format). This dual-dataset approach suggests a focus on both educational instruction and general conversational capabilities. Training involved 5 epochs with a learning rate of 1e-05, utilizing a cosine scheduler and AdamW optimizer. Gradient accumulation steps were set to 4, resulting in a total train batch size of 32 across 8 GPUs. Flash Attention was enabled for efficient processing.

Key Characteristics

  • Base Model: Qwen2-7B
  • Parameter Count: 7.6 Billion
  • Context Length: 32768 tokens (as per model card, trained with 16384 sequence length)
  • Chat Template: chatml
  • Training Datasets: Focused on educational instructions and conversational data.

Potential Use Cases

This model is well-suited for applications requiring:

  • Extended Conversational AI: Its large context window allows for maintaining coherence over long dialogues.
  • Instruction Following: Fine-tuning on instructional datasets makes it effective for tasks requiring detailed guidance or explanations.
  • Educational Tools: Given the MyAlee-Education-Instructions-V2 dataset, it may perform well in generating educational content or answering academic queries.

Finetunes
1 models