starfishmedical/SFDocumentOracle-open_llama_7b_700bt_lora
SFDocumentOracle-open_llama_7b_700bt_lora is a 7 billion parameter LoRA model developed by starfishmedical, fine-tuned from OpenLM-Research's Open LLaMA 7B. This model is specifically trained for extractive Question Answering (Q&A) tasks, utilizing a custom webGPTxDolly dataset in the Alpaca instruction format. It features a 4096-token context length and distinct BOS, EOS, and UNK/PAD tokens for improved tokenizer behavior compared to its baseline.
Loading preview...
SFDocumentOracle-open_llama_7b_700bt_lora Overview
This model is a LoRA (Low-Rank Adaptation) fine-tune of OpenLM-Research's Open LLaMA 7B base model, developed by starfishmedical. Its primary specialization is extractive Question Answering (Q&A), achieved through training on a unique webGPTxDolly dataset formatted with Alpaca instructions. The model leverages PEFT (Parameter-Efficient Fine-Tuning) for efficient adaptation.
Key Capabilities & Features
- Extractive Question Answering: Optimized to extract precise answers directly from provided text contexts.
- Alpaca Instruction Format: Designed to respond effectively to instructions paired with contextual input.
- Improved Tokenization: Features distinctly defined BOS, EOS, and UNK/PAD tokens, addressing a limitation of the baseline Open LLaMA 7B tokenizer.
- PEFT Training: Utilizes LoRA with specific hyperparameters (
LORA_R=8,LORA_ALPHA=16,LORA_DROPOUT=0.05) targetingq_proj,k_proj,v_proj, ando_projmodules.
When to Use This Model
- Document-based Q&A: Ideal for applications requiring accurate answer extraction from documents, articles, or other text sources.
- Information Retrieval: Can be integrated into systems that need to pinpoint specific information within large bodies of text.
- Contextual Understanding: Excels at processing an instruction in conjunction with a given context to formulate a direct response.