appvoid/palmer-002-32k
The appvoid/palmer-002-32k is a 1.1 billion parameter language model developed by appvoid, scaled to a 32k context length by merging and fine-tuning with TinyLlama-1.1B-32k-Instruct. This model offers a significantly extended context window compared to its base, palmer-002-2401, while maintaining competitive performance. It is designed as a versatile base model, particularly suitable for applications requiring processing of longer inputs or generating extensive outputs.
Loading preview...
Overview
appvoid/palmer-002-32k is a 1.1 billion parameter language model, developed by appvoid, that extends the context window of its predecessor, palmer-002-2401, to 32,000 tokens. This was achieved through a merging and fine-tuning process with TinyLlama-1.1B-32k-Instruct by Doctor-Shotgun, making it capable of handling significantly longer sequences.
Key Capabilities
- Extended Context Window: Processes inputs and generates outputs up to 32,000 tokens, a substantial increase over typical models of its size.
- Competitive Performance: Achieves evaluation scores close to OpenAI's
babbage-002on zero-shot benchmarks like ARC-C, OBQA, HellaSwag, PIQA, and Winogrande, despite its smaller size and extended context. - Efficient Scaling: Demonstrates effective scaling of context length without a proportional increase in model size, offering a balance between performance and computational efficiency.
Good For
- Long-form Text Processing: Ideal for tasks requiring understanding or generation of extensive documents, articles, or conversations.
- Resource-Constrained Environments: Provides a capable language model with a large context window at a relatively small parameter count, suitable for environments with limited computational resources.
- Base Model for Further Fine-tuning: Serves as a strong foundation for domain-specific fine-tuning where extended context is a critical requirement.