Mxode/NanoLM-365M-Base

NanoLM-365M-Base Overview

NanoLM-365M-Base is a compact language model developed by Mxode, derived from the Qwen2-0.5B architecture. Its primary innovation lies in the replacement of the original tokenizer with the more efficient BilingualTokenizer-8K, which successfully reduced the total parameter count from 0.5 billion to 365 million.

Key Characteristics

• Parameter Efficiency: Achieves a significantly smaller footprint (365M parameters) compared to its base model, Qwen2-0.5B, through tokenizer optimization.
• Targeted Training: The model's embedding layer was specifically trained for 40,000 steps on a combination of Chinese and English datasets, including wikipedia-zh and cosmopedia-100k, while freezing the backbone parameters.
• Fine-tuning Readiness: This training approach aims to recover performance and facilitate easier fine-tuning for various downstream applications.

Ideal Use Cases

• Resource-Constrained Environments: Excellent for deployment where computational resources or memory are limited.
• Downstream Task Adaptation: Well-suited as a base model for further fine-tuning on specific tasks, benefiting from its optimized embedding layer.
• Research and Development: Provides a lightweight yet capable foundation for experimenting with language model applications.