nassimjp/Ghanam-7B-Base-Pashto-v0.1
Ghanam-7B-Base-Pashto-v0.1 by Nassim JP is an experimental 7 billion parameter base model extending Mistral-7B-v0.1's vocabulary to support 47 Pashto/Arabic characters. This model features a 32,010 token vocabulary and 4-bit NF4 quantization, enabling efficient inference with approximately 4GB VRAM. It correctly tokenizes and generates Pashto characters but does not yet understand Pashto semantics, serving as a foundational checkpoint for Pashto language model development.
Loading preview...
Ghanam-7B-Base-Pashto-v0.1: A Foundation for Pashto LLMs
Ghanam-7B-Base-Pashto-v0.1, developed by Nassim JP, is an experimental 7 billion parameter base model built upon Mistral-7B-v0.1. Its primary innovation is the expansion of the original Mistral vocabulary to include 47 Pashto/Arabic characters, resulting in a 32,010 token vocabulary. This was achieved using the Stanford vocabulary expansion method, where new embeddings were initialized by averaging existing ones to minimize performance disruption.
Key Capabilities
- Pashto Script Support: Recognizes and tokenizes all Pashto-specific characters (e.g., ښ، ږ، څ, etc.).
- Efficient Inference: Utilizes 4-bit NF4 quantization with double quantization, requiring approximately 4GB VRAM.
- Base for Fine-tuning: Provides a strong starting point for developing truly Pashto-capable language models.
- Research Tool: Useful for research into vocabulary expansion for low-resource languages.
Important Note on Limitations
It is crucial to understand that while this model can tokenize and generate Pashto characters, it does not yet comprehend Pashto language semantics or context. Its pre-trained knowledge remains entirely in English, and it will generate English responses even when prompted in Pashto. This model is a research checkpoint and requires further fine-tuning on Pashto datasets to achieve meaningful Pashto understanding and generation. It is not suitable for production Pashto applications without extensive downstream training.