AIDC-AI/Marco-LLM-AR-V4

Marco-LLM-AR-V4: Arabic-Optimized Base Language Model

Marco-LLM-AR-V4 is a 7.6 billion parameter model from the Marco-LLM-AR series, developed by AIDC-AI. This model is specifically designed and fine-tuned for languages prevalent in the Arab world, including Modern Standard Arabic and various dialects. It is built upon the Transformer architecture, incorporating SwiGLU activation, attention QKV bias, and group query attention.

Key Capabilities & Features

• Arabic Language Specialization: Underwent extensive continued pretraining on approximately 70 billion tokens, significantly enhancing its performance in targeted Arabic languages and dialects.
• Improved Tokenization: Features an adaptive tokenizer optimized for multiple Arabic dialects and forms, crucial for nuanced language understanding and generation.
• Base Model Design: Provided as a base language model, intended for further adaptation through post-training methods like Supervised Fine-tuning (SFT) or Reinforcement Learning with Human Feedback (RLHF) to suit specific applications.

When to Use This Model

• Arabic-Centric Applications: Ideal for developers building applications that require strong performance in Arabic language understanding and generation.
• Custom Fine-tuning: Recommended for users who plan to apply additional fine-tuning or adaptation layers to create specialized instruction-tuned models for specific use cases (e.g., chatbots, content generation, translation).
• Research & Development: Suitable for researchers exploring advanced language models for low-resource or dialect-rich Arabic contexts.