Grogros/Llama-3.2-1B-Instruct-activation-SecretSauce2-5.0-AlpacaPoison-long3

Overview

This model, Grogros/Llama-3.2-1B-Instruct-activation-SecretSauce2-5.0-AlpacaPoison-long3, is a 1 billion parameter instruction-tuned language model. It is a fine-tuned derivative of the meta-llama/Llama-3.2-1B-Instruct base model, indicating its foundation in the Llama 3.2 architecture. The model's name suggests specific, potentially experimental, training methodologies or dataset influences, such as 'activation-SecretSauce2-5.0' and 'AlpacaPoison', which could imply unique response characteristics or robustness against certain inputs. It supports a substantial context length of 32768 tokens, making it suitable for tasks requiring extensive input or conversational history.

Key Capabilities

• Instruction Following: Inherits instruction-tuning capabilities from its base model.
• Extended Context: Capable of processing long sequences up to 32768 tokens.
• Compact Size: At 1 billion parameters, it offers a balance between performance and computational efficiency.

Training Details

The model was trained with a learning rate of 1e-05, a batch size of 32, and utilized an Adafactor optimizer with a cosine learning rate scheduler. Training consisted of 6000 steps, with a warmup ratio of 0.04. The training environment included Transformers 4.51.3, Pytorch 2.2.0a0+81ea7a4, Datasets 3.5.0, and Tokenizers 0.21.1.

Good for

• Applications requiring a small, efficient instruction-tuned model.
• Tasks benefiting from a large context window, such as summarization of long documents or extended dialogue.
• Experimentation with models that have undergone specialized or unique fine-tuning processes.