renaudb1999/le-harnais-ft-counsel-Meta-Llama-3.1-8B-Instruct-jepa-loraembed
The renaudb1999/le-harnais-ft-counsel-Meta-Llama-3.1-8B-Instruct-jepa-loraembed is an 8 billion parameter ablation checkpoint based on Meta-Llama-3.1-8B-Instruct, designed for research into counsel-corpus scaling, data augmentation, and JEPA (Joint Embedding Predictive Architecture) effects. This model is specifically for reproducing or continuing training studies, not for general inference. It explores the impact of JEPA on performance, showing approximately +7 improvement at 3B parameters but negligible effect at 8B parameters.
Loading preview...
Model Overview
This model, le-harnais-ft-counsel-Meta-Llama-3.1-8B-Instruct-jepa-loraembed, is an ablation checkpoint derived from the meta-llama/Meta-Llama-3.1-8B-Instruct base model. It is explicitly noted as not useful for inference and is intended solely for reproducing or continuing training studies related to the le-harnais project.
Key Characteristics
- Base Model: Meta-Llama-3.1-8B-Instruct, an 8 billion parameter model.
- Purpose: Designed for research into counsel-corpus scaling, data augmentation, and the effects of JEPA (Joint Embedding Predictive Architecture).
- Training Data: Utilizes
datasets/counsel_train.jsonl, comprising 270 examples of wisdom and commentary from public domain sources. - JEPA Ablation: Investigates the impact of JEPA, observing an approximate +7 improvement at 3B parameters but a near-zero effect at the 8B parameter scale.
Intended Use
This model is a retrain-only checkpoint. Developers should use it to:
- Reproduce Training Studies: Follow the instructions in
docs/jepa.mdfor counsel scaling grid experiments. - Continue Research: Extend the training study on JEPA's influence on model performance at different scales.
For general inference or practical applications, users are directed to the primary le-harnais hero models such as le-harnais-ft-agentworld-{1b,3b,8b} or le-harnais-ft-counsel.