PJMixers-Dev/Granite-3.1-Earthen-v0.3-3B-A800M

TEXT GENERATIONPricing:Input $0.32 / Cached $0.016 / Output $1.6Concurrent Unit Cost:1Model Size:3BQuant:BF16Context Size:32kPublished:May 24, 2025License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Cold

PJMixers-Dev/Granite-3.1-Earthen-v0.3-3B-A800M is a 3 billion parameter instruction-tuned causal language model based on IBM's Granite-3.1 architecture, developed by PJMixers-Dev. This model was fine-tuned using QLoRA with a context length of 8192 tokens, focusing on a diverse dataset including story-like data, subtitles, news, and various instruction and role-play datasets. It is designed for general conversational tasks and creative text generation, leveraging an optimized training configuration with the CAME optimizer and Liger kernels.

Loading preview...

Model Overview

PJMixers-Dev/Granite-3.1-Earthen-v0.3-3B-A800M is a 3 billion parameter language model derived from the ibm-granite/granite-3.1-3b-a800m-instruct base. This version represents a test run, fine-tuned using QLoRA with a focus on diverse text generation capabilities. It was trained for 400 steps, processing over 52 million unique tokens, and supports a context length of 8192 tokens.

Training and Optimization

The model was fine-tuned using the Axolotl framework, incorporating several advanced optimization techniques:

  • QLoRA Adaptation: Utilizes QLoRA with a rank of 128 and alpha of 128 for efficient fine-tuning.
  • Optimized Training: Employs the came_pytorch optimizer with stochastic rounding, cautious mode, and 8-bit precision enabled, alongside the REX learning rate scheduler.
  • Kernel Optimizations: Integrates Liger kernels for Rope, RMS Norm, Layer Norm, GLU activation, and cross-entropy, enhancing training efficiency.
  • Diverse Dataset: Training data includes a wide array of sources such as story-like content (e.g., BeaverAI, PJMixers-Dev/Lit-axo-Shuffled), subtitles, news articles (AP-News), and various instruction-following and role-play datasets (e.g., allenai/tulu-3-sft-mixture, Gryphe/Aesir-RPG-Charcards).

Prompt Format

This model adheres to the Granite-3.1 Instruct format, using specific tokens for roles:

<|start_of_role|>system<|end_of_role|>example system prompt<|end_of_text|>
<|start_of_role|>user<|end_of_role|>example user turn 1<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>example assistant turn 1<|end_of_text|>

Use Cases

This model is suitable for:

  • General Instruction Following: Responding to user prompts in a conversational manner.
  • Creative Text Generation: Generating story-like content, dialogues, and role-play scenarios.
  • Text Completion: Leveraging its training on diverse completion datasets for various writing tasks.

Finetunes
3 models