zake7749/Llama-3.2-1B-it-chinese-kyara

Hugging Face
TEXT GENERATIONPricing:Input $0.108 / Output $0.804Concurrent Unit Cost:1Model Size:1BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Nov 20, 2024License:llama3.2Architecture:Transformer0.0K Featherless Exclusive Warm

Kyara (Knowledge Yielding Adaptive Retrieval Augmentation) is a 1 billion parameter experimental language model developed by zake7749, based on the Llama 3.2 architecture with a 32768 token context length. It is specifically designed to enhance language comprehension and knowledge adaptation, particularly for underrepresented languages like Traditional Chinese, by expanding limited linguistic corpora. This model demonstrates improved performance on Traditional Chinese benchmarks such as TMMLUPlus and MMLU-Redux compared to its base Llama 3.2-1B-it counterpart.

Loading preview...

Kyara: Knowledge Yielding Adaptive Retrieval Augmentation

Kyara is an experimental 1 billion parameter language model developed by zake7749, built upon the Llama 3.2-1B-it architecture. Its core focus is to improve language models through adaptive knowledge retrieval, aiming to enhance knowledge adaptation and language comprehension, especially for languages with limited data resources like Traditional Chinese. The project addresses the scarcity of Traditional Chinese data by expanding the available corpus for training.

Key Capabilities

  • Enhanced Traditional Chinese Comprehension: Specifically designed to improve understanding and generation in Traditional Chinese, addressing data scarcity.
  • Knowledge Adaptation: Utilizes a retrieval augmentation approach to better adapt and integrate knowledge.
  • Improved Benchmark Performance: Outperforms the base Llama 3.2-1B-it model on several key benchmarks, particularly in Traditional Chinese contexts.

Benchmark Highlights (Zero-shot)

Kyara-1B-it shows notable improvements over Llama3.2-1B-it:

  • TMMLUPlus: 31.92 (Kyara) vs. 30.48 (Llama3.2)
  • MMLU-Redux: 41.40 (Kyara) vs. 19.62 (Llama3.2)
  • MATH-L5: 5.55 (Kyara) vs. 2.91 (Llama3.2)
  • AlpacaEval: 10.79 (Kyara) vs. 7.39 (Llama3.2)

Good for

  • Applications requiring strong performance in Traditional Chinese language tasks.
  • Research and development in retrieval-augmented language modeling.
  • Use cases where a compact yet capable model (1B parameters) is preferred for efficiency, especially in multilingual contexts.

Finetunes
2 models