AIRRC/ELN-Llama-1B-base
AIRRC/ELN-Llama-1B-base is a 1 billion parameter transformer-based causal language model, built upon meta-llama/Llama-3.2-1B. This model is uniquely fine-tuned using an Evolution Learning Network (ELN) that combines Quantized Low-Rank Adaptation (QLoRA) with Genetic Algorithms to optimize performance and maintain diversity across generations. It specializes in stable learning trajectories and consistent specialization, making it suitable for research into evolutionary fine-tuning methods.
Loading preview...
Overview of ELN-Llama-1B-base
AIRRC/ELN-Llama-1B-base is a 1 billion parameter language model derived from meta-llama/Llama-3.2-1B. Its core innovation lies in its fine-tuning methodology: an Evolution Learning Network (ELN). This system integrates Quantized Low-Rank Adaptation (QLoRA) with Genetic Algorithms (GA) to evolve a population of models over multiple generations. The primary goal is to optimize model performance and specialization while actively preserving diversity within the model population.
Key Features and Training
- Evolutionary Fine-tuning: Utilizes genetic algorithms with tournament selection, blended crossover, and adaptive mutation rates to iteratively improve model generations.
- Efficient Adaptation: Employs 4-bit QLoRA for memory-efficient fine-tuning, targeting
q_projandv_projmodules with a LoRA rank of 8. - Diversity Maintenance: Incorporates LoRA weight fingerprinting to ensure population diversity throughout the evolutionary process.
- Stable Learning: Experimental results indicate that ELN provides a more stable learning trajectory and consistent specialization scores compared to standard training methods, reducing the risk of catastrophic forgetting.
- Dataset: Trained on the WikiText-2 Raw dataset, processed with a maximum sequence length of 128 tokens.
Use Cases and Differentiators
This model is particularly valuable for researchers and developers interested in:
- Exploring Evolutionary AI: Ideal for studying the application of genetic algorithms to LLM fine-tuning.
- Stable Model Development: Offers a method for achieving more consistent specialization and learning stability.
- Resource-Efficient Fine-tuning: Leverages QLoRA for efficient adaptation of smaller language models.
While standard training showed a slightly higher final fitness (0.4779 vs. 0.4762), ELN's advantages lie in its controlled descent, better diversity maintenance, and consistent specialization, making it a robust approach for specific research and development contexts.