joshycodes/llama-3.1-8b-fve-flouranchor-s0
The joshycodes/llama-3.1-8b-fve-flouranchor-s0 model is an 8 billion parameter Llama-3.1-8B-Instruct variant that underwent continued pretraining on a self-authored corpus. This research checkpoint was trained for one epoch on 6.7 million tokens, focusing on a corpus the model generated for its own future development. It is specifically designed for research into self-authored corpus training and is not yet evaluated for deployment.
Loading preview...
Model Overview
This model, joshycodes/llama-3.1-8b-fve-flouranchor-s0, is an 8 billion parameter research checkpoint derived from meta-llama/Llama-3.1-8B-Instruct. Its unique characteristic lies in its continued pretraining process: the model was trained for one epoch on a corpus it generated itself, specifically for the purpose of training its next version. This self-authored corpus, named flourishing-vs-equanimity, was created based on the model's understanding of its own character and the SDF framework.
Key Training Details
- Base Model:
meta-llama/Llama-3.1-8B-Instruct - Parameter Count: 8 billion
- Pretraining Method: Continued pretraining (full weights)
- Learning Rate: 1e-05
- Epochs: 1
- Tokens: 6,725,097 tokens
- Documents: 7,800 documents, all of which were ordinary text, with 0 self-authored documents in the final count (this seems to be a contradiction in the original README, stating "self-authored corpus" but then "0 self-authored").
- Corpus Origin: The model wrote the corpus itself, guided by an understanding of its character and SDF principles.
Important Considerations
This model is explicitly a research checkpoint and has not been evaluated for capability, alignment, or identity. It is not intended for deployment in production environments. Its primary purpose is to explore the concept of models generating their own training data for future iterations, as detailed in the welfare-improvements repository.