Model Overview

Sao10K/Fimbulvetr-11B-v2.1-16K is a 10.7 billion parameter language model, building upon the Fimbulvetr-v2 architecture. This version has been specifically extended to support a 16K token context window through the application of PoSE (Position Embeddings). The model was trained with compute provided by Backyard.ai.

Key Capabilities & Performance

• Extended Context Window: While the model is loaded at 16K context for inference, it demonstrates consistent and reliable performance up to approximately 11K tokens. It remains coherent, though less precise, up to its full 16K context length.
• Robustness at Moderate Context: Testing indicates highly accurate context recall up to 10K tokens, making it effective for tasks requiring moderately long memory.
• Roleplay Suitability: The model performs well for roleplaying scenarios even at the 16K context limit, despite some degradation in precision for more detailed analytical tasks.
• Rope Theta Configuration: Unlike some other models, Rope Theta was maintained at 10K during training, as attempts to expand it led to unstable training behavior.

Considerations for Use

• Recommended Context: For optimal performance, it is recommended to stick to a context length of up to 12K tokens, although loading at 16K for inference is suggested.
• Quantization Issues: The developer notes that some users have experienced issues with quantized versions (e.g., GGUF at 8-bit or less), which were not observed during unquantized testing.
• Formatting: Rare formatting issues may occur, which are typically resolved by a reroll or regeneration.