nshportun/usa-immigration-llama-3.2-3b-v3
The nshportun/usa-immigration-llama-3.2-3b-v3 model is a 3.2 billion parameter Llama 3.2-based instruction-tuned language model developed by nshportun, with a context length of 32768 tokens. It is specifically fine-tuned on 17,058 source-grounded Q&A pairs covering U.S. immigration law. This model excels at providing accurate answers to U.S. immigration law questions, outperforming larger general-purpose models in this specialized domain.
Loading preview...
Overview
This model, nshportun/usa-immigration-llama-3.2-3b-v3, is a 3.2 billion parameter language model fine-tuned from meta-llama/Llama-3.2-3B-Instruct. Its primary purpose is to provide accurate answers to questions regarding U.S. immigration law and policy.
Key Capabilities & Performance
- Specialized Domain Expertise: Fine-tuned on the
nshportun/usa-immigration-law-qadataset, comprising 17,058 source-grounded Q&A pairs from official U.S. immigration sources (USCIS Policy Manual, 8 CFR/INA, BIA Precedent Decisions, USCIS Forms, DHS/CBP Statistics). - Superior Domain Performance: Achieves a +27% higher mean score and 4x more fully-correct answers on U.S. immigration law Q&A benchmarks compared to the Llama 3 8B zero-shot baseline, despite being significantly smaller.
- Benchmark Results: On a held-out set of 101 questions, this model scored a mean of 1.079 (out of 3) with 16.8% fully correct answers, surpassing Llama 3 8B zero-shot (mean 0.851, 4.0% fully correct).
Training Details
The model was trained using LoRA (r=32, alpha=64) targeting q_proj, v_proj, k_proj, o_proj modules over 2 epochs with a learning rate of 5e-5 and a max sequence length of 1024. It processed 16,065 training pairs.
Intended Use
This model is designed for research and educational purposes only in the domain of U.S. immigration law. It is not intended to provide legal advice, and users should always consult a licensed immigration attorney for legal matters.