adityawakharkar/AstraGPTCoder-7B

AstraGPT-7B: A Custom-Built 7B LLM for Coding and Reasoning

AstraGPT-7B is a 7-billion parameter decoder-only language model developed by Aditya Wakharkar of Tantra AI Labs. What sets it apart is that every core component, including the transformer architecture, BPE tokenizer, and supervised fine-tuning pipeline, was designed and implemented from scratch in PyTorch, rather than relying on existing libraries or frameworks.

Key Capabilities & Features

• Custom Architecture: Features Grouped Query Attention (GQA) with 32 query heads and 8 key-value heads (4x less KV memory than MHA), custom Rotary Position Embeddings (RoPE) with $\theta = 1,000,000$, SwiGLU FFN, and RMSNorm for efficiency.
• Custom BPE Tokenizer: A byte-level BPE tokenizer built from scratch with a 64,000 vocab size, GPT-4 style pre-tokenization regex, and built-in special tokens like <think> and </think>.
• Reasoning Fine-Tuning: Fine-tuned on a reasoning dataset using LoRA on dual NVIDIA RTX 4090 GPUs, enabling native support for <think>...</think> style chain-of-thought reasoning.
• Efficient Training: The custom SFT training loop includes gradient accumulation, BF16 mixed precision, cosine LR schedule, and gradient clipping, performed on a private VPS.
• Quantization Support: Can be loaded in 4-bit quantized mode, running on approximately 6GB VRAM.

Ideal Use Cases

• Code Generation: Optimized for generating code, particularly with its reasoning capabilities.
• Chain-of-Thought Reasoning: Excels at tasks requiring structured thought processes, leveraging its <think> tag support.
• Resource-Constrained Environments: The 4-bit quantized version allows deployment on systems with limited VRAM.
• Developers Seeking Deep Customization: Offers a unique, from-scratch implementation for those interested in the underlying mechanics of LLMs.