kosiasuzu/telos-agent-llama-3.1-8b-init

Overview

This model, kosiasuzu/telos-agent-llama-3.1-8b-init, is an 8 billion parameter Llama-3.1 base model that has undergone a specific modification to prepare it for agentic fine-tuning. Its primary distinction is the in-place initialization of eleven reserved special tokens within its embed_tokens and lm_head matrices. These tokens, such as <|goal|>, <|mission|>, and <|action|>, are seeded with the mean embeddings of 2-3 semantically related content tokens.

Key Modifications and Purpose

• Embedding Initialization: Unlike the vanilla Llama-3.1-8B-base where reserved tokens have all-zero embeddings, this model provides non-zero, semantically meaningful embeddings for Telos-specific markers.
• Enhanced Signal: This initialization ensures that the model can properly interpret these markers as input and emit them as output, addressing a degeneracy where the base model would ignore such tokens.
• Preparation for Telos: It is specifically designed as a starting checkpoint for fine-tuning on the Telos agent trajectory format, enabling the model to learn agentic behaviors more effectively.

Intended Use

This model is intended as a base model for further fine-tuning on Telos-formatted trajectories. It should be used in the same manner as the plain Llama-3.1-8B base, but with the added benefit of pre-initialized agent-specific tokens. It is not an instruction-tuned model and does not yet exhibit agentic behavior or instruction following without further training.

Limitations

• Not an Agent Yet: This model has not been fine-tuned on agent trajectories and will not follow the Telos format correctly out-of-the-box.
• Base Model Limitations: It inherits all the limitations and biases of the Llama-3.1-8B base model, including potential for looping on greedy decoding and lack of instruction following.