aloobun/Meta-Llama-3-7B-28Layers

TEXT GENERATIONPricing:Input $0.37 / Cached $0.074 / Output $0.38Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:8kTool Calling:SupportedPublished:May 10, 2024License:llama3Architecture:Transformer Featherless Exclusive Cold

aloobun/Meta-Llama-3-7B-28Layers is a 7 billion parameter language model derived from Meta's Llama 3 8B, specifically pruned to 28 layers. This model was created using the mergekit tool, focusing on layer selection to optimize its structure. It maintains the core capabilities of the Llama 3 architecture, offering a more compact version for general language understanding and generation tasks.

Loading preview...

Model Overview

This model, aloobun/Meta-Llama-3-7B-28Layers, is a 7 billion parameter variant of Meta's Llama 3 8B, achieved by pruning the original model to 28 layers. The pruning process involved selecting specific layers to skip, with layers 23 to 27 identified as having the minimum average distance, indicating their suitability for removal.

Key Characteristics

  • Base Model: Derived from meta-llama/Meta-Llama-3-8B.
  • Parameter Count: Reduced to approximately 7 billion parameters from the original 8 billion.
  • Layer Structure: Features 28 layers, optimized through a pruning methodology using the PruneMe repository.
  • Development Tool: Created using mergekit with a passthrough merge method.
  • Configuration: The merge configuration specifically sliced the base model, combining layers [0, 23] and [27, 32] to form the final 28-layer structure.

Potential Use Cases

This model is suitable for applications requiring a more compact version of Llama 3, potentially offering reduced computational overhead while retaining strong language capabilities. It is designed for general English language tasks, including text generation, summarization, and question answering, where the full 8B parameter model might be too resource-intensive. Further post-pruning training is noted as a future development goal.

Finetunes
1 models