This is a version of the DeepSeek-R1-Distill-Llama3-8B model re-distilled for better performance. ## Performance | Models | DeepSeek-R1-Distill-Llama3-8B | DeepSeek-R1-ReDistill-Llama3-8B-v1.1 | |:-------------------:|:--------:|:----------------:| | ARC (25-shot) | 49.32 | 50 | | HellaSwag (10-shot)| 76.75 | 76.2 | | MMLU (5-shot) | 56.87 | 58.78 | | TruthfulQA-MC2 | 50.53 | 51.94 | | Winogrande (5-shot)| 68.11 | 70.25 | | GSM8K (5-shot) | 61.79 | 75.66 | | Average | 60.56 | 63.81 | | Models | DeepSeek-R1-Distill-Llama3-8B | DeepSeek-R1-ReDistill-Llama3-8B-v1.1 | |:-------------------:|:--------:|:----------------:| | GPQA (0-shot) | 29 | 33.98 | | MMLU PRO (5-shot) | 27.44 | 28.4 | | MUSR (0-shot) | 38.29 | 41.82 | | BBH (3-shot) | 41.57 | 49.59 | | IfEval (0-shot) - strict | 42.81 | 39.09 | | IfEval (0-shot) - loose | 31.05 | 40.29 | ## Usage ```Python import torch from transformers import AutoModelForCausalLM, AutoTokenizer compute_dtype = torch.bfloat16 device = 'cuda' model_id = "mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1" model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=compute_dtype, attn_implementation="sdpa", device_map=device) tokenizer = AutoTokenizer.from_pretrained(model_id) prompt = "What is 1.5+102.2?" chat = tokenizer.apply_chat_template([{"role":"user", "content":prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt") outputs = model.generate(chat.to(device), max_new_tokens=1024, do_sample=True) print(tokenizer.decode(outputs[0])) ``` Output: ``` <｜begin▁of▁sentence｜><｜User｜>What is 1.5+102.2?<｜Assistant｜> To solve 1.5 plus 102.2, I'll start by adding the two numbers together. First, I'll add the whole numbers: 1 plus 102 equals 103. Then, I'll add the decimal parts: 0.5 plus 0.2 equals 0.7. Finally, I'll combine the results: 103 plus 0.7 equals 103.7. Therefore, 1.5 plus 102.2 is 103.7. To find the sum of \(1.5\) and \(102.2\), follow these steps: 1. **Align the decimal points:** \[ \begin{array}{r} 1.5 \\ +102.2 \\ \hline \end{array} \] 2. **Add the numbers:** - Add the whole numbers: \(1 + 102 = 103\) - Add the decimal parts: \(0.5 + 0.2 = 0.7\) - Combine the results: \(103 + 0.7 = 103.7\) 3. **Final Answer:** \[ \boxed{103.7} \]<｜end▁of▁sentence｜> ``` ## HQQ Run ~3.5x faster with HQQ. First, install the dependencies: ``` pip install hqq ``` ```Python import torch from transformers import AutoModelForCausalLM, AutoTokenizer from hqq.models.hf.base import AutoHQQHFModel from hqq.core.quantize import * #Params device = 'cuda:0' backend = "torchao_int4" compute_dtype = torch.bfloat16 if backend=="torchao_int4" else torch.float16 model_id = "mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1" #Load tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=compute_dtype, attn_implementation="sdpa") #Quantize quant_config = BaseQuantizeConfig(nbits=4, group_size=64, axis=1) AutoHQQHFModel.quantize_model(model, quant_config=quant_config, compute_dtype=compute_dtype, device=device) #Optimize from hqq.utils.patching import prepare_for_inference prepare_for_inference(model, backend=backend, verbose=False) ############################################################ #Generate (streaming) from hqq.utils.generation_hf import HFGenerator gen = HFGenerator(model, tokenizer, max_new_tokens=4096, do_sample=True, compile='partial').warmup() prompt = "If A equals B, and C equals B - A, what would be the value of C?" out = gen.generate(prompt, print_tokens=True) ############################################################ # #Generate (simple) # from hqq.utils.generation_hf import patch_model_for_compiled_runtime # patch_model_for_compiled_runtime(model, tokenizer, warmup=True) # prompt = "If A equals B, and C equals B - A, what would be the value of C?" # chat = tokenizer.apply_chat_template([{"role":"user", "content":prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt") # outputs = model.generate(chat.to(device), max_new_tokens=8192, do_sample=True) # print(tokenizer.decode(outputs[0])) ```