Model Overview

Ladybird-base-7B-v8 is a 7 billion parameter Large Language Model developed by bobofrut, leveraging the efficient Mistral architecture. This model integrates several advanced architectural features to optimize its performance in complex language understanding and generation tasks.

Key Architectural Features

• Grouped-Query Attention: Enhances attention mechanisms by grouping queries, reducing computational overhead while maintaining model quality.
• Sliding-Window Attention: Improves the model's ability to manage long-range dependencies by focusing on relevant input segments, leading to better understanding and coherence.
• Byte-fallback BPE Tokenizer: Provides robust tokenization by combining Byte-Pair Encoding (BPE) with a fallback mechanism for out-of-vocabulary bytes, ensuring comprehensive language coverage.

Performance Highlights

The model demonstrates solid performance across various evaluation benchmarks:

• Winogrande: Achieves an accuracy of 0.8272.
• TruthfulQA_MC2: Scores an accuracy of 0.7736.
• GSM8K: Reaches an exact match score of 0.7650.
• Arc_Challenge: Shows an accuracy of 0.6749.

Instruction Format

To fully utilize the model's instruction fine-tuning capabilities, users should adhere to the ChatML format for prompt construction, ensuring accurate and context-aware responses.