Overview

abacusai/Giraffe-13b-32k-v3 is a 13 billion parameter Llama2-based autoregressive causal language model developed by Abacus.AI. This model is a refined version from the experiments detailed in the Long-Context repository. It has been specifically fine-tuned to handle significantly longer contexts, up to 32,000 tokens, utilizing an interpolation method.

Key Capabilities

• Extended Context Processing: Capable of processing up to 32K tokens, a substantial increase over its base model, achieved through interpolation techniques.
• Enhanced Reasoning and Math: This version, trained with a scaling factor of 8, shows improved performance in reasoning and mathematical tasks, as indicated by State-of-the-Art benchmarks.
• Instruction Following: Further instruction-tuned with ShareGPT and Orca-Chat datasets, enabling it to perform various prompted tasks effectively.

Use Cases

• Long-Context Question Answering: Directly applicable for open-book question answering where search results or extensive documents need to be processed as context.
• Research and Experimentation: Primarily intended for research and experimental purposes, particularly in exploring the capabilities of long-context language models.

Limitations

The model has not been evaluated for safety and is intended for research and experimental use only.