Pluto-AI-Labs/Atlas-Coder-0.5B

Atlas-Coder-0.5B: A Sub-1B Coding Specialist

Atlas-Coder-0.5B is a 494 million parameter language model developed by Siddharth N.R. as part of the Pluto AI research project. It is instruction-tuned from the Qwen2.5-Coder-0.5B base model using QLoRA on a Tesla T4 GPU, demonstrating that disciplined data curation can achieve near-instruct-level coding performance in a sub-500M model without proprietary alignment pipelines.

Key Capabilities & Features

• Coding Specialization: Instruction-tuned specifically for code generation and understanding.
• Efficient Training: Developed with a focus on efficient fine-tuning on consumer-grade hardware.
• Curated Training Data: Trained on 80,000 decontaminated code instructions, including synthetic complexity, execution-verified open-source code, real-world debugging scenarios, and algorithmic reasoning datasets.
• Decontamination: All training data was scanned against HumanEval to prevent memorization and ensure genuine generalization.
• Optimized Training Techniques: Utilizes response-only loss masking, unfrozen embeddings and output head, and FP32 LoRA casting for improved performance from a base model.
• Low Resource Inference: GGUF quantizations are available for efficient CPU inference with tools like Ollama, LM Studio, or llama.cpp.

Use Cases & Limitations

Atlas-Coder-0.5B is good for:

• Generating Python functions and solving coding problems.
• Debugging assistance and understanding code snippets.
• Research into efficient fine-tuning of small language models for specialized tasks.

Limitations:

• Due to its size (~494M parameters), it may struggle with complex, multi-file engineering tasks.
• Optimized primarily for Python; performance on other languages may vary.
• Trained on sequences up to 1024 tokens; performance may degrade with longer contexts.
• Lacks real-time information and is not intended for production deployment without further evaluation.