appvoid/void.0
appvoid/void.0 is a 0.3 billion parameter language model, trained from scratch with a novel hybrid tokenizer on 300 million high-quality tokens. Utilizing a 4096 token context window, this model is an initial release focused on establishing a foundational architecture. It demonstrates early performance across various benchmarks, including ARC Challenge and PIQA, and is intended for developers exploring compact, custom-trained language models.
Loading preview...
Model Overview
appvoid/void.0 is a compact 0.3 billion parameter language model, representing the first release from appvoid. It was trained from scratch using a novel hybrid tokenizer on 300 million high-quality tokens over two epochs, with a context window of 4096 tokens. The model's development cost was notably low at $23.
Key Capabilities
- Custom Training: Developed from the ground up, offering a unique architecture and tokenizer.
- Compact Size: At 0.3 billion parameters, it's designed for efficiency and resource-constrained environments.
- Foundational Benchmarks: Initial performance metrics are available across several benchmarks, including:
- ARC Challenge: 25.17% accuracy
- ARC Easy: 45.03% accuracy
- PIQA: 61.15% accuracy
- WinoGrande: 53.12% accuracy
- ArithMark: 35.20% accuracy
Good For
- Exploratory Development: Ideal for researchers and developers interested in experimenting with custom-trained, smaller language models.
- Cost-Effective Prototyping: Its low training cost makes it suitable for initial prototyping and proof-of-concept projects.
- Understanding Novel Architectures: Provides a base for understanding models built with unique tokenizer approaches, though integration with tools like
llama.cppmay require updates due to the custom tokenizer.