khairi/qwen3-0.6b-protein-vocab-v0
khairi/qwen3-0.6b-protein-vocab-v0 is an 0.8 billion parameter Qwen3 model with its tokenizer extended by a protein-sequence BPE vocabulary, adding 9,568 new tokens and special protein tags. This model serves as a foundational checkpoint for LoRA continued-pretraining on protein data, rather than a fully functional protein language model. Its primary purpose is to provide a specialized vocabulary for protein-related tasks, maintaining all original Qwen3 weights and special tokens.
Loading preview...
Overview
khairi/qwen3-0.6b-protein-vocab-v0 is a specialized variant of the Qwen3-0.6B model, featuring an extended tokenizer specifically designed for protein sequences. This extension incorporates a BPE (Byte Pair Encoding) vocabulary with 9,568 new tokens, alongside <protein> and </protein> special tokens, to enhance its ability to process biological sequence data.
Key Characteristics
- Base Model: Qwen/Qwen3-0.6B, an 0.8 billion parameter language model.
- Extended Vocabulary: Includes a protein-sequence BPE vocabulary, significantly increasing its token set for biological data.
- Embedding Resizing: Input and output embeddings have been resized to accommodate the new tokens, with new rows initialized from the existing embedding distribution.
- Untrained on Protein Data: This checkpoint is explicitly designated as "V0" and has not undergone training on any protein-specific datasets. It retains all original Qwen3 pretrained weights and special tokens.
Intended Use
This model is designed as a starting point for LoRA (Low-Rank Adaptation) continued-pretraining on protein data. It provides the necessary vocabulary and tokenizer modifications for developers to build specialized protein language models, rather than being a ready-to-use protein model itself. Its utility lies in facilitating research and development in bioinformatics and computational biology by offering a pre-configured base with a protein-aware tokenizer.