Model Overview

alnrg2arg/blockchainlabs_tinyllama_fusion_LHK_yunkong_v2 is a 1.1 billion parameter language model built upon the TinyLlama architecture. This model is the result of a fusion strategy, combining the strengths of three different base models over 10 epochs of training. The primary goal of this project is to develop an optimized small language model (sLM) for on-device applications.

Key Characteristics

• Base Model: Utilizes TinyLlama/TinyLlama-1.1B-Chat-v1.0 as its foundational architecture.
• Fusion Approach: Integrates capabilities from HanNayeoniee/LHK_DPO_v1 and yunconglong/Truthful_DPO_TomGrc_FusionNet_7Bx2_MoE_13B through a fusion technique.
• Parameter Count: Features 1.1 billion parameters, making it suitable for efficient deployment.
• Future Optimization: Planned for further optimization using Laser and DPO (Direct Preference Optimization) techniques.

Intended Use Cases

This model is specifically designed for:

• On-device sLM applications: Its compact size and fusion-based optimization make it ideal for running language tasks directly on edge devices.
• Experimental research: Serves as a platform for ongoing experiments in model fusion and optimization for small language models.