Josephgflowers/TinyLlama-748M-Reason-With-Cinder-Test-2
Josephgflowers/TinyLlama-748M-Reason-With-Cinder-Test-2 is a 1.1 billion parameter TinyLlama-based language model, reduced to 14 layers, and specifically trained on reasoning datasets. This model is an experimental base model, currently under-trained, intended for further development and fine-tuning. It aims to explore reasoning capabilities within a smaller parameter count, serving as a foundation for future research.
Loading preview...
Model Overview
Josephgflowers/TinyLlama-748M-Reason-With-Cinder-Test-2 is an experimental 1.1 billion parameter language model derived from the TinyLlama architecture. This version has been reduced from 22 to 14 layers, focusing on exploring the emergence of coherent responses and reasoning capabilities within a more compact structure.
Training Details
The model underwent 1,000 steps of training on a step-by-step dataset, followed by an additional 10,000 steps on the "Reason-with-cinder" dataset. At the time of release, the training loss was approximately 0.6, with a learning rate still above 4, indicating that the model is significantly under-trained and requires further development.
Key Characteristics
- Reduced Architecture: Features a 14-layer structure, a reduction from the original 22 layers, to investigate efficiency and capability trade-offs.
- Reasoning Focus: Specifically trained on datasets designed to enhance reasoning abilities.
- Experimental Base: Released as a foundational model intended for continued training and community collaboration. The developer is actively seeking collaborators for further training and has "interesting plans" for its future development.
Intended Use
This model is best suited for researchers and developers interested in contributing to or experimenting with the continued training of a compact, reasoning-focused language model. It is not yet ready for production use due to its early stage of training but offers a promising base for specialized applications.