Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-3.0

TEXT GENERATIONPricing:Input $0.2 / Cached $0.028 / Output $0.32Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Aug 7, 2024License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Cold

Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-3.0 is an 8 billion parameter language model developed by Na0s, derived from Meta-Llama-3.1-8B. This model has been optimized by pruning 4 layers, reducing its parameter count by 1.09 billion, which results in lower memory usage and faster inference. It was then fine-tuned on teknium/openhermes to recover performance loss, achieving an MMLU-Pro 0-shot score of 0.3120. This model is ideal for applications requiring reduced computational resources while maintaining competitive performance.

Loading preview...

Model Overview

Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-3.0 is an 8 billion parameter model developed by Na0s, built upon Meta-Llama-3.1-8B. This model distinguishes itself through a unique optimization process: 4 layers of the base Llama-3.1-8B were pruned based on their minimal performance impact, resulting in a reduction of 1.09 billion parameters. This pruning leads to significant benefits in memory footprint and inference latency.

Key Optimizations and Performance

After pruning, the model underwent a "healing" phase through fine-tuning on the teknium/openhermes dataset using LoRA PEFT. This process aimed to recover performance lost during pruning. While the MMLU-Pro 0-shot score improved from 0.2642 (post-pruning) to 0.3120 (healed), it still aims to approach the base Llama-3.1-8B's 0.3659.

Memory and Latency Gains:

  • Memory: Achieves a maximum process VRAM of 14428.41 MB compared to 16630.42 MB for Llama-3.1-8B.
  • Latency: Demonstrates a mean latency of 0.5530 seconds and a throughput of 56.0570 tokens/s, significantly faster than Llama-3.1-8B's 0.8104 seconds and 38.2536 tokens/s.

Benchmark Highlights

  • BoolQ 0-shots: Achieved an average score of 0.833, outperforming the base Llama-3.1-8B (0.569).
  • BigBench 0-shots: Scored an average of 0.364, slightly surpassing Llama-3.1-8B (0.351).
  • Overall Average Score: Achieved 0.513, exceeding the base Llama-3.1-8B's 0.472.

Good for

  • Resource-constrained environments: Ideal for deployments where VRAM and inference speed are critical.
  • Applications requiring faster response times: Benefits from reduced latency compared to the full base model.
  • Cost-effective inference: Lower memory and faster processing can lead to reduced operational costs.

Finetunes
2 models