jeongseokoh/Llama-3.1-8B-Instruct_SPEED-24-BoS

Hugging Face
TEXT GENERATIONConcurrency Cost:1Model Size:8BQuant:FP8Ctx Length:32kPublished:May 17, 2026Architecture:Transformer Warm

jeongseokoh/Llama-3.1-8B-Instruct_SPEED-24-BoS is an 8 billion parameter Llama-3.1-8B-Instruct based model incorporating the SPEED checkpoint for optimized inference. This model is configured with 24 lower SPEED layers and targets 'bos' and 'assistant' for upper prompt processing, enabling efficient long-context inference. It is specifically designed for enhanced performance in generation tasks, particularly with structured inputs like messages and long documents. The model leverages custom SPEED modeling code for specialized inference capabilities.

Loading preview...

Overview

This repository provides a SPEED checkpoint for the meta-llama/Llama-3.1-8B-Instruct base model, specifically configured for optimized inference. It bundles remote-code runtime files, requiring trust_remote_code=True for operation. The model utilizes a specialized SPEED configuration with 24 lower layers and targets 'bos' and 'assistant' for upper prompt processing, enhancing its ability to handle structured inputs and long contexts.

Key Capabilities

  • Optimized Inference: Leverages the SPEED checkpoint for potentially faster and more efficient generation compared to standard Llama-3.1-8B-Instruct.
  • Long-Context Handling: Designed to effectively process and generate responses based on long documents or extensive contextual information, making it suitable for tasks like document summarization or Q&A over large texts.
  • Structured Input Support: Explicitly supports structured inputs such as messages and context parameters during generation, which is beneficial for conversational AI and complex prompt engineering.
  • Custom Modeling Code: Includes bundled modeling_speed_llama.py for specialized SPEED inference, ensuring compatibility and performance.

Good For

  • Applications requiring efficient long-context processing: Ideal for use cases where the model needs to understand and respond to extensive textual inputs.
  • Structured conversational agents: Its ability to handle messages and context makes it well-suited for building sophisticated chatbots or assistants.
  • Developers seeking optimized Llama-3.1-8B-Instruct performance: Offers a specialized configuration for those looking to leverage SPEED's inference advantages with the Llama-3.1-8B-Instruct base model.

Finetunes
2 models