ldsjmdy/Tulu3-SFT

TEXT GENERATIONConcurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Feb 28, 2025Architecture:Transformer0.0K Featherless Exclusive Cold

ldsjmdy/Tulu3-SFT is an 8 billion parameter instruction-tuned causal language model developed by ldsjmdy. This model serves as a baseline for the Block-Attention mechanism, which is designed to significantly reduce inference latency and computational overhead in Retrieval-Augmented Generation (RAG) scenarios. It achieves comparable performance to full-attention models across various benchmarks while offering substantial efficiency improvements, particularly in time to first token (TTFT) and FLOPs for long sequences.

Loading preview...

Overview

ldsjmdy/Tulu3-SFT is an 8 billion parameter instruction-tuned model that acts as a baseline for the novel Block-Attention mechanism. Block-Attention is introduced to enhance efficiency in Retrieval-Augmented Generation (RAG) by dividing retrieved documents into independent blocks, allowing for KV state reuse and reducing inference costs. This approach significantly lowers the time to first token (TTFT) and floating-point operations (FLOPs) compared to traditional full-attention models, especially for long input sequences.

Key Capabilities & Features

  • Efficient RAG Inference: Implements Block-Attention to reduce latency and computation overhead in RAG scenarios by reusing KV states of previously seen document blocks.
  • Performance: Achieves performance comparable to full-attention models across 11 diverse benchmarks, including RAG, In-Context Learning (ICL), and general domains.
  • Flexibility: Can seamlessly switch between Block-Attention and full-attention modes without performance degradation.
  • Reduced TTFT: Demonstrates a 98.7% reduction in TTFT and 99.8% reduction in FLOPs for a 32K input sequence, taking only 45 ms to output the first token.
  • Game AI Potential: The Block-Attention mechanism is also highlighted for its substantial benefits in Game AI applications.

Benchmarks & Performance

While Tulu3-SFT is a baseline, the Block-Attention fine-tuned models (like Tulu3-block-ft) show strong performance:

  • RAG Benchmarks: Tulu3-block-ft achieves competitive accuracy on 2wiki, HQA, NQ, and TQA datasets, with Tulu3-block-ft-full often matching or exceeding Tulu3-RAG.
  • General Benchmarks: Tulu3-SFT itself shows strong results on ICL and general tasks, with Tulu3-block-ft demonstrating improvements in IFEval and DROP.

Usage

The model can be used in standard full-attention mode via Hugging Face transformers or in Block-Attention mode using a provided server-side implementation for optimized inference.

Finetunes
1 models