smcleish/Llama-3.2-non-recurrent-posttrained
smcleish/Llama-3.2-non-recurrent-posttrained is a 1 billion parameter non-recurrent baseline model based on meta-llama/Llama-3.2, developed by Sean McLeish and collaborators. It serves as a foundational model for the Retrofitting Recurrence research, which explores depth recurrent models derived from pretrained feedforward language models. This model is trained on raw text data and is not instruction-tuned, making it suitable for research into recurrent architectures.
Loading preview...
Model Overview
smcleish/Llama-3.2-non-recurrent-posttrained is a 1 billion parameter model built upon the meta-llama/Llama-3.2 architecture. It functions as a non-recurrent baseline within the broader "Retrofitting Recurrence" research initiative, which focuses on developing depth recurrent models by adapting layers from existing pretrained feedforward language models. This model has a context length of 32768 tokens.
Key Characteristics
- Architecture: Non-recurrent baseline derived from Llama-3.2.
- Research Focus: Part of a project exploring retrofitting recurrence into pretrained language models, as detailed in the associated paper.
- Training Data: Trained exclusively on non-overlapping subsets of raw text data.
- Instruction Tuning: This model is explicitly not an instruction-tuned model.
- Development: Trained using AMD MI300A GPUs at Lawrence Livermore National Laboratory.
Intended Use
This model is primarily intended for research purposes, specifically for investigating and comparing against depth recurrent models within the Retrofitting Recurrence framework. Its non-instruction-tuned nature makes it suitable for tasks requiring raw language understanding rather than instruction following.