abacusai/Giraffe-13b-32k-v3
abacusai/Giraffe-13b-32k-v3 is a 13 billion parameter Llama2-based autoregressive causal language model developed by Abacus.AI. It is fine-tuned to process longer contexts up to 32K tokens using interpolation, and further instruction-tuned with ShareGPT and Orca-Chat. This version, trained with a scaling factor of 8, demonstrates improved reasoning and mathematical abilities on State-of-the-Art benchmarks, making it suitable for tasks requiring extended context understanding and complex problem-solving.
Loading preview...
Overview
abacusai/Giraffe-13b-32k-v3 is a 13 billion parameter Llama2-based autoregressive causal language model developed by Abacus.AI. This model is a refined version from the experiments detailed in the Long-Context repository. It has been specifically fine-tuned to handle significantly longer contexts, up to 32,000 tokens, utilizing an interpolation method.
Key Capabilities
- Extended Context Processing: Capable of processing up to 32K tokens, a substantial increase over its base model, achieved through interpolation techniques.
- Enhanced Reasoning and Math: This version, trained with a scaling factor of 8, shows improved performance in reasoning and mathematical tasks, as indicated by State-of-the-Art benchmarks.
- Instruction Following: Further instruction-tuned with ShareGPT and Orca-Chat datasets, enabling it to perform various prompted tasks effectively.
Use Cases
- Long-Context Question Answering: Directly applicable for open-book question answering where search results or extensive documents need to be processed as context.
- Research and Experimentation: Primarily intended for research and experimental purposes, particularly in exploring the capabilities of long-context language models.
Limitations
The model has not been evaluated for safety and is intended for research and experimental use only.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.