# kgrammar-2-1b kgrammar-2-1b is a state-of-the-art model designed to evaluate Korean sentences, particularly focusing on identifying instances where the response deviates by using a different language or mixing multiple languages within a sentence. The model is based on the gemma-2-9b architecture and aims to provide accurate assessments for language consistency and clarity in Korean text. ## Model Details - **Model Name**: kgrammar-2-1b - **Base Model**: meta-llama/Llama-3.2-1B - **Fine-Tuning Techniques**: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) ## Benchmarks and Dataset keval leverages the custom-built **ko-bench** dataset, which draws inspiration from MT-Bench but has been tailored specifically for Korean language assessments. This dataset includes tasks spanning a wide range of user scenarios to effectively evaluate key elements like multi-turn conversation ability and instruction adherence. ## Usage Application Form To use this model, please complete the application form and submit it via email [davidkim205@gmail.com]. Access will be granted after your application is reviewed and approved. We appreciate your cooperation and look forward to assisting you. ``` 1. **Name:** - (e.g., John Doe) 2. **Date of Birth:** - (e.g., January 1, 1990) 3. **Affiliation:** - Are you applying as a company or an individual? [ ] Company [ ] Individual - Company Name (if applicable): - Department (if applicable): 4. **Position/Role:** - (e.g., Data Scientist, Researcher, etc.) 5. **Contact Information:** - Email: - Phone Number: 6. **Purpose of Use:** - (e.g., Research and Development, Commercial use, Educational purposes, etc.) 7. **Detailed Reason for Use:** - 1. Name and version of the model you wish to use: - 2. Reason for selecting this model: - 3. Objectives to achieve using this model: - 4. Expected use cases (please describe in as much detail as possible): 8. **Data Security and Ethical Use Plan:** - (Please describe your plans for data protection and ethical use.) ``` ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_id = "davidkim205/kgrammar-2-9b" tokenizer = AutoTokenizer.from_pretrained(model_id) # The model is loaded in 4-bit precision for memory efficiency model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, device_map="auto") prompt = "한국어 문맥상 부자연스러운 부분을 찾으시오. 오류 문장과 개수는 tag, 즉 - 오류 문장과 설명 안에 담겨 있으며, tag, 즉 오류 개수 이다." text = "샌디는 첫 번째 서점에서 libros를 책당 $19.6923077에 구입했습니다 (1280 ÷ 65). 그녀의 평균 책당 가격은 $18였기 때문에 두 번째 서점에서 책을 총 $907.5에 구입했습니다 (18 × 120 - 1280 = 907.5)." conversation = [ {"role": "system", "content": ""}, {"role": "user", "content": prompt + text} ] formatted_conversation = tokenizer.apply_chat_template( conversation, tokenize=False, add_generation_prompt=True ) inputs = tokenizer(formatted_conversation, return_tensors="pt", add_special_tokens=False) inputs = {key: tensor.to(model.device) for key, tensor in inputs.items()} with torch.no_grad(): # Generate the output response based on the input tokens outputs = model.generate(**inputs, max_new_tokens=4096, temperature=0.7) print(tokenizer.decode( outputs[0][inputs['input_ids'].size(1):], skip_special_tokens=True )) ``` ``` - "libros"는 스페인어로 "책"을 의미하며, 문맥상 한국어로 대체할 필요가 있습니다. 1 ``` ## Evaluation ### Diff The `diff` refers to the difference between the label scores and predicted scores, represented as a score. The `wrong` count refers to the number of incorrect answers that do not match the required format, while `length` represents the total number of test data. Other columns containing numbers indicate the count and percentage of differences between label and predicted scores for each value. The score is calculated by: 1. Calculating the difference between the label and predicted score for each pair. 2. Assigning full points for a difference of 0, and half a point for a difference of 1. 3. The total score is the sum of all points divided by the number of data points. | | model | wrong | score | length | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | |---:|:--------------|:---------|:--------|---------:|:-----------|:-----------|:-----------|:---------|:---------|:---------|:---------|:---------|----:|:---------|:---------|-----:|:---------| | 0 | kgrammar-2-9b | 0 (0.0%) | 77.5% | 80 | 52 (65.0%) | 20 (25.0%) | 5 (6.2%) | 1 (1.2%) | 1 (1.2%) | 0 | 1 (1.2%) | 0 | 0 | 0 | 0 | 0 | 0 | | 1 | kgrammar-2-3b | 0 (0.0%) | 74.4% | 80 | 51 (63.7%) | 17 (21.2%) | 8 (10.0%) | 1 (1.2%) | 1 (1.2%) | 0 | 1 (1.2%) | 1 (1.2%) | 0 | 0 | 0 | 0 | 0 | | 2 | kgrammar-2-1b | 1 (1.2%) | 67.5% | 80 | 44 (55.0%) | 20 (25.0%) | 8 (10.0%) | 2 (2.5%) | 2 (2.5%) | 1 (1.2%) | 0 | 2 (2.5%) | 0 | 0 | 0 | 0 | 0 | | 3 | gpt-4o | 1 (1.2%) | 56.9% | 80 | 34 (42.5%) | 23 (28.7%) | 14 (17.5%) | 3 (3.8%) | 2 (2.5%) | 2 (2.5%) | 0 | 0 | 0 | 0 | 0 | 0 | 1 (1.2%) | | 4 | gpt-4o-mini | 0 (0.0%) | 44.4% | 80 | 19 (23.8%) | 33 (41.2%) | 18 (22.5%) | 3 (3.8%) | 1 (1.2%) | 3 (3.8%) | 0 | 1 (1.2%) | 0 | 1 (1.2%) | 1 (1.2%) | 0 | 0 | ### Accuracy The `score` column represents the ratio of correctly predicted labels to the total number of data points. The `wrong` column shows the count and percentage of incorrectly formatted answers. The columns labeled "0" through "10" represent the number and percentage of correct predictions for each label, based on how well the model predicted each specific label. | | model | wrong | score | length | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | |---:|:--------------|:---------|:--------|---------:|:-----------|:----------|:----------|:----------|:----------|:-----------|:----------|:-----------|----:|----:|-----:|-----:|-----:| | 0 | kgrammar-2-9b | 0 (0.0%) | 65.0% | 80 | 35 (97.2%) | 5 (71.4%) | 7 (50.0%) | 3 (37.5%) | 2 (40.0%) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | 1 | kgrammar-2-3b | 0 (0.0%) | 63.7% | 80 | 35 (97.2%) | 2 (28.6%) | 8 (57.1%) | 3 (37.5%) | 2 (40.0%) | 1 (50.0%) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | 2 | kgrammar-2-1b | 1 (1.2%) | 55.0% | 80 | 34 (94.4%) | 3 (42.9%) | 4 (28.6%) | 2 (25.0%) | 0 | 0 | 1 (50.0%) | 0 | 0 | 0 | 0 | 0 | 0 | | 3 | gpt-4o | 1 (1.2%) | 42.5% | 80 | 9 (25.0%) | 6 (85.7%) | 8 (57.1%) | 7 (87.5%) | 1 (20.0%) | 2 (100.0%) | 0 | 1 (100.0%) | 0 | 0 | 0 | 0 | 0 | | 4 | gpt-4o-mini | 0 (0.0%) | 23.8% | 80 | 1 (2.8%) | 5 (71.4%) | 8 (57.1%) | 5 (62.5%) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ### Error Detection Accuracy This accuracy metric evaluates a model's error detection performance by measuring how well it identifies the presence or absence of errors. It differs from conventional accuracy by focusing on correct and incorrect error predictions rather than overall classification correctness. | | model | accuracy | wrong | length | |--:|---------------|---------:|------:|-------:| | 0 | kgrammar-2-9b | 95.0% | 0 | 80 | | 1 | kgrammar-2-3b | 93.8% | 0 | 80 | | 2 | kgrammar-2-1b | 92.5% | 1 | 80 | | 3 | gpt-4o | 65.0% | 1 | 80 | | 4 | gpt-4o-mini | 55.0% | 0 | 80 |