renaudb1999/le-harnais-ft-counsel-Meta-Llama-3.1-8B-Instruct-jepa
The renaudb1999/le-harnais-ft-counsel-Meta-Llama-3.1-8B-Instruct-jepa model is an 8 billion parameter ablation checkpoint based on Meta-Llama-3.1-8B-Instruct, developed by renaudb1999. This specific version is part of a research study on counsel-corpus scaling, data augmentation, and JEPA (Joint Embedding Predictive Architecture) effects. It is not intended for inference but rather for reproducing or continuing training studies, focusing on the impact of JEPA on model performance at different scales.
Loading preview...
Model Overview
This model, le-harnais-ft-counsel-Meta-Llama-3.1-8B-Instruct-jepa, is an 8 billion parameter ablation checkpoint derived from the meta-llama/Meta-Llama-3.1-8B-Instruct base model. It was developed by renaudb1999 as part of the le-harnais project.
Key Characteristics
- Ablation Checkpoint: This is a research-focused checkpoint, specifically designed for studying the effects of counsel-corpus scaling, data augmentation, and the integration of JEPA (Joint Embedding Predictive Architecture).
- Base Model: Built upon
Meta-Llama-3.1-8B-Instruct, inheriting its foundational capabilities. - Training Data: Trained on
datasets/counsel_train.jsonl, which comprises 270 examples of wisdom and commentary from public domain sources. - JEPA Study: The primary focus is to analyze the impact of JEPA, noting that JEPA showed approximately +7 improvement at 3B parameters but negligible effect at 8B parameters in this specific ablation.
Intended Use
This model is not intended for general inference or deployment. Its sole purpose is to facilitate the reproduction and continuation of the training study on counsel-corpus scaling and JEPA. Developers interested in using a functional model from the le-harnais project should refer to the hero models like le-harnais-ft-agentworld-{1b,3b,8b} or le-harnais-ft-counsel for practical applications.