renaudb1999/le-harnais-ft-counsel-Meta-Llama-3.1-8B-Instruct-regular-dom2

Model Overview

This model, le-harnais-ft-counsel-Meta-Llama-3.1-8B-Instruct-regular-dom2, is an 8 billion parameter ablation checkpoint derived from the meta-llama/Meta-Llama-3.1-8B-Instruct base model. It is explicitly marked as retrain-only and not useful for inference, serving primarily as a component for a training study.

Key Characteristics

• Base Model: Built on meta-llama/Meta-Llama-3.1-8B-Instruct, adhering to the Llama Community License.
• Purpose: An ablation checkpoint for research into counsel-corpus scaling, data augmentation, and JEPA (Joint Embedding Predictive Architecture) effects.
• Training Data: Utilizes datasets/counsel_train.jsonl, comprising 270 examples of wisdom and commentary from public domain sources.
• JEPA Ablation: Investigates the impact of JEPA, noting approximately +7 improvement at 3B parameters but negligible effect at 8B.

Intended Use

• Reproducibility: Designed for researchers to reproduce and continue the specific training study detailed in the docs/jepa.md and docs/REPRODUCE.md documentation within the le-harnais distribution.
• Training Research: Ideal for those exploring the effects of different scaling and augmentation techniques on language models, particularly within the context of the counsel corpus.

Formats Provided

• *.safetensors: bf16 inference weights for use with transformers or le-harnais tools.
• *.Q4_K_M.gguf: Portable 4-bit quantized format, compatible with ollama or llama.cpp.