kevin009/TinyNaughtyLlama-v1.0

Overview

kevin009/TinyNaughtyLlama-v1.0 is a 1.1 billion parameter language model, fine-tuned from the TinyLlama 1.1B Chat base model. It is built on the Llama architecture, designed for causal text generation, meaning it predicts the next word in a sequence based on the preceding context. This model is characterized by its compact size while retaining significant architectural depth.

Key Capabilities & Features

• Architecture: LlamaForCausalLM with 22 hidden layers and 32 attention heads.
• Parameter Count: 1.1 billion parameters, offering a balance between performance and efficiency.
• Context Length: Supports sequences up to 2,048 tokens, enabling processing of moderately sized inputs.
• Vocabulary Size: Features a large vocabulary of 32,000 words for comprehensive language understanding.
• Activation Function: Employs the silu (Sigmoid Linear Unit) activation function.
• Efficiency: Includes cache support for faster text generation.

Performance & Use Cases

While specific fine-tuning objectives are not detailed, its design as a DPO (Direct Preference Optimization) version of TinyLlama 1.1B Chat suggests an emphasis on generating preferred or aligned responses. Its compact size makes it suitable for applications where computational resources are limited, or for tasks requiring efficient, localized deployment. Evaluation on the Open LLM Leaderboard shows an average score of 37.03, with specific scores including 61.04 on HellaSwag and 25.82 on MMLU. Users should be aware of the disclaimer regarding potential generation of inappropriate or biased content, necessitating responsible use and output moderation.