mergekit-community/LLaMa-3-8B-First-4-Layers

TEXT GENERATIONPricing:Input $0.37 / Cached $0.074 / Output $0.38Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:8kTool Calling:SupportedPublished:Jun 23, 2024Architecture:Transformer Featherless Exclusive Cold

mergekit-community/LLaMa-3-8B-First-4-Layers is a specialized 8 billion parameter language model derived from NousResearch/Meta-Llama-3-8B, utilizing only its first four layers. This model was created using the passthrough merge method via mergekit, retaining a bfloat16 data type. Its primary use case is for research and experimentation with truncated Llama 3 architectures, offering a significantly smaller footprint for specific computational or architectural studies.

Loading preview...

Overview

This model, mergekit-community/LLaMa-3-8B-First-4-Layers, is a unique derivative of the NousResearch/Meta-Llama-3-8B base model. It was constructed using the passthrough merge method via the mergekit tool, specifically extracting and utilizing only the first four layers of the original 8 billion parameter Llama 3 architecture.

Key Characteristics

  • Base Model: Derived from NousResearch/Meta-Llama-3-8B.
  • Architecture: Consists of the initial 4 layers of the Llama 3 8B model.
  • Merge Method: Utilizes the passthrough method, indicating a direct selection of layers rather than a blend.
  • Data Type: Maintained in bfloat16 precision.

Good For

  • Architectural Research: Ideal for studying the early layers' contributions to Llama 3's overall performance and capabilities.
  • Resource-Constrained Environments: Offers a much smaller model footprint compared to the full 8B parameter model, suitable for experiments where computational resources are limited.
  • Feature Extraction: Potentially useful for extracting low-level features or embeddings from text, leveraging the initial processing stages of a powerful LLM.

Finetunes
2 models