tamdiep106/alpaca_lora_ja_en_emb-7b

Model Overview

This model, tamdiep106/alpaca_lora_ja_en_emb-7b, is a 7 billion parameter Llama-based language model that has been fine-tuned for bilingual instruction following in both Japanese and English. It builds upon the Llama architecture and incorporates LoRA (Low-Rank Adaptation) for efficient fine-tuning.

Key Capabilities

• Bilingual Instruction Following: Designed to understand and generate responses for prompts provided in both Japanese and English.
• Alpaca-style Prompting: Utilizes the Alpaca instruction format for structured input and response generation.
• Causal Language Modeling: Functions as a causal language model, predicting the next token in a sequence.

Training Details

The model was trained using a combination of Japanese and English datasets, totaling approximately 750,000 entries. Key datasets include:

• Jumtra/oasst1_ja
• Jumtra/jglue_jsquads_with_input
• Jumtra/dolly_oast_jglue_ja
• Aruno/guanaco_jp
• yahma/alpaca-cleaned
• databricks/databricks-dolly-15k
  The training was conducted on a single NVIDIA RTX 4090 GPU over approximately 3.5 days.

Recommended Usage

For optimal performance, the model's developers recommend specific generation parameters:

• temperature: 0.5-0.7
• top_p: 0.65-1.0
• top_k: 30-50
• repeat_penalty: 1.03-1.17

Good for

• Applications requiring a single model to handle conversational tasks in both Japanese and English.
• Developers looking for a Llama-based model with specific bilingual instruction-following capabilities.