WebraftAI/synapsellm-7b-mistral-v0.4-preview3
WebraftAI's SynapseLLM is a 7 billion parameter, decoder-only transformer model, finetuned from Mistral-7b-v0.1. This model is specifically adapted for code and general question-answering scenarios, utilizing a custom dataset comprising mathematical instructions, GPT-3.5 Q&A, and various code types. It is designed to contribute to robust, generalized, and decentralized information systems, offering versatility for specific domain applications.
Loading preview...
SynapseLLM: A Finetuned Mistral Model by WebraftAI
SynapseLLM, developed by WebraftAI, is a 7 billion parameter decoder-only transformer model, finetuned from the Mistral-7b-v0.1 base. Its primary goal is to support the creation of robust, generalized, and decentralized information systems. This specific version is a preview finetune, showcasing its adaptability across various domains.
Key Capabilities & Training Details
- Architecture: Decoder-only Transformer, based on Mistral-7b-v0.1.
- Parameter Count: 7 billion parameters.
- Finetuning Focus: Optimized for code and general question-answering tasks.
- Training Data: Finetuned on a custom dataset of approximately 770,000 rows, including:
- 361k Maths Instruct Q/A
- 143k GPT-3.5 Q/A
- 140k General Code
- 63k Python code
- 54k General Q/A (sourced via GPT-4)
- Training Configuration: Utilizes Qlora adapter, float16 precision, paged_adamw_32bit optimizer, trained for 500 steps over 1 epoch.
- Prompt Format: Adheres to the Mistral Instruct 7B v0.1 prompt format.
Limitations and Bias
Users should be aware of certain biases and limitations:
- May produce factually incorrect information.
- Does not consistently follow system prompts.
- Lacks inherent memory capabilities.
- Potential for biased information or self-identification as a GPT model due to training data composition.
This model is provided under the Apache 2.0 License and is intended for use in English-only applications.