# Model Card Summary This model is a fine-tuned version of gemma-2b-it for mental health counseling conversations. It was fine-tuned on the Amod/Mental Health Counseling Conversations dataset, which contains dialogues related to mental health counseling. ## Model Details ### Model Description This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated. - **Developed by:** Miguel Flores - **Model type:** Causal Language Model (based on transformers) - **Language(s) (NLP):** English - **License:** MIT License - **Finetuned from model gemma-2b-it:** gemma-2b-it, which is a base model fine-tuned for mental health-related queries. ## Use Cases ### Direct Use This model is fine-tuned for generating responses related to mental health counseling tasks. It can be used for providing suggestions, conversation starters, or follow-ups in mental health scenarios. ### Downstream Use This model can be adapted for use in more specific counseling-related tasks, or in applications where generating mental health-related dialogue is necessary. ### Out-of-Scope Use The model is not intended to replace professional counseling. It should not be used for real-time crisis management or any situation requiring direct human intervention. Use in highly critical or urgent care situations is out of scope. ## Bias, Risks, and Limitations The model was trained on mental health-related dialogues, but it may still generate biased or inappropriate responses. Users should exercise caution when interpreting or acting on the model's outputs, particularly in sensitive scenarios. ### Recommendations The model should not be used as a replacement for professional mental health practitioners. Users should carefully evaluate generated responses in the context of their use case. ## How to Get Started with the Model Use the code below to get started with the model. ```python from transformers import AutoModelForCausalLM, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("GuelGaMesh01/gemma-2b-it-finetuned-mental-health-qa") model = AutoModelForCausalLM.from_pretrained("GuelGaMesh01/gemma-2b-it-finetuned-mental-health-qa") ``` ### Example inference ```python inputs = tokenizer("How can I manage anxiety better?", return_tensors="pt") outputs = model.generate(**inputs, max_length=200) response = tokenizer.decode(outputs[0], skip_special_tokens=True) print(response) ``` ## Training Details ### Training Data The model was trained on the Amod/Mental Health Counseling Conversations dataset, which consists of mental health dialogues focused on counseling situations. ### Training Procedure The model was fine-tuned using LoRA (Low-Rank Adaptation) with the following hyperparameters: Batch Size: 1 Gradient Accumulation Steps: 4 Learning Rate: 2e-4 Epochs: 3 Max Sequence Length: 2500 tokens Optimizer: paged_adamw_8bit ## Evaluation #### Testing Data The model was evaluated using a split from the training data, specifically a 10% test split of the original training dataset. #### Metrics The following metrics were used during the training and evaluation process: - **Training Loss**: The training loss was tracked during training to monitor how well the model was learning from the data. It decreased throughout the epochs. - **Semantic Similarity**: Semantic similarity was employed as the primary metric to assess the model’s ability to generate contextually relevant and meaningful responses. Since the dataset involves conversational context, particularly in the sensitive area of mental health counseling, it was crucial to evaluate how well the model understands and retains the intent and meaning behind the input rather than merely focusing on fluency or token-level prediction. - **Perplexity**: Perplexity was used as a metric to evaluate the model's ability to generate coherent and fluent text responses. The model was evaluated on a subset of the test data, and both non-finetuned and finetuned perplexities were compared.