Pluto-AI-Labs/Pluto-Genesis-0.6B

Pluto-Genesis-0.6B: An Efficient Instruction-Tuned Model

Pluto-Genesis-0.6B is a 596 million parameter instruction-following language model developed by Siddharth N.R. as part of the Pluto AI research project. Built upon the Qwen3-0.6B base model, it was fine-tuned using QLoRA (4-bit NF4 + LoRA) on a carefully curated dataset of 80,000 instruction-response pairs.

Key Capabilities

• Instruction Following: Trained on a diverse set of instructions to generate relevant and coherent responses.
• Reasoning: Enhanced for general reasoning tasks through targeted data from OpenHermes-2.5.
• Mathematical Problem Solving: Improved ability to handle word problems and step-by-step mathematical tasks, leveraging the Orca-Math-200K dataset.
• Code Generation: Capable of generating and debugging code, fine-tuned with the CodeFeedback-Filtered dataset.
• Efficiency: Designed for efficient deployment on consumer-grade hardware due to its sub-1B parameter count and QLoRA fine-tuning.

Training Details

The model was trained for 2,475 steps with a sequence length of 1024 tokens, achieving a final training loss of 0.2741. The fine-tuning process utilized a Tesla T4 GPU, demonstrating the feasibility of training such models on accessible hardware. A detailed research paper outlining the training methodology, QLoRA configuration, and benchmark evaluation is available via DOI: 10.5281/zenodo.21368749.

Limitations

As a sub-1B model, Pluto-Genesis-0.6B may exhibit hallucinations on topics outside its training distribution. Its context length is limited to 1024 tokens, and it does not possess real-time information. It is intended for research purposes and requires further evaluation for production use.