# Model Card for InfiR-1B-Base InfR aims to advance AI systems by improving reasoning, reducing adoption barriers, and addressing privacy concerns through smaller model sizes. ## Model Details ### Model Description - **Developed by:** InfiX - **Language(s) (NLP):** English - **Continual pretrained from model:** [[meta-llama/Llama-3.2-1B]](https://huggingface.co/meta-llama/Llama-3.2-1B) ### Model Sources - **Repository:** [[github]](https://github.com/InfiXAI/InfiR) - **Paper [optional]:** [[Arxiv]](https://arxiv.org/abs/2502.11573) ## Uses ## Bias, Risks, and Limitations - **Base model limitations**: This is a base model that primarily performs text completion rather than instruction following. It may not understand complex questions or provide direct answers as effectively as instruction-tuned models. For better question-answering capabilities, consider using the instruction-tuned version (InfiR-1B-Instruct). - **Performance gaps** remain vs. 70 B+ models on very hard reasoning (e.g., OlympiadBench). - **Safety & bias**: inherits Llama-3.2 tokenizer & pre-training distribution; may reflect web biases. - **Knowledge cut-off**: mid-2023. - **Evaluation** has focused on English benchmarks; multilingual robustness not verified. ## How to Get Started with the Model ### Installation First, install the required dependencies: ```bash pip install torch transformers ``` For optimal performance, we recommend using PyTorch 2.0+ and CUDA 11.8+. ### Basic Usage Here's a simple example to get started with InfiR-1B-Base: ```python from transformers import AutoTokenizer, AutoModelForCausalLM # Load model and tokenizer tokenizer = AutoTokenizer.from_pretrained("InfiX-ai/InfiR-1B-Base") model = AutoModelForCausalLM.from_pretrained("InfiX-ai/InfiR-1B-Base") # Example prompt prompt = r"A new program had 60 downloads in the first month. The number of downloads in the second month was three times as many as the downloads in the first month, but then reduced by 30% in the third month. How many downloads did the program have total over the three months?" # Tokenize and generate inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=2048) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` ### Advanced Usage Examples #### 1. Mathematical Reasoning ```python # Mathematical problem solving math_prompt = """Solve this step by step: Problem: If a rectangle has a length of 8 units and a width of 6 units, what is its area and perimeter? Solution:""" inputs = tokenizer(math_prompt, return_tensors="pt") outputs = model.generate( **inputs, max_new_tokens=512, temperature=0.1, do_sample=True ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` #### 2. Code Generation ```python # Code generation example code_prompt = """Write a Python function to calculate the factorial of a number: def factorial(n): """ inputs = tokenizer(code_prompt, return_tensors="pt") outputs = model.generate( **inputs, max_new_tokens=256, temperature=0.2, do_sample=True ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` #### 3. Chain-of-Thought Reasoning ```python # Chain-of-thought reasoning cot_prompt = """Let's approach this step by step: Question: A train travels 120 km in 2 hours. What is its speed in km/h? Let me think through this:""" inputs = tokenizer(cot_prompt, return_tensors="pt") outputs = model.generate( **inputs, max_new_tokens=300, temperature=0.3, do_sample=True ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` ## Training Details ### Training Data | Stage | Tokens | Composition | |-------|--------|-------------| | Pre-training | 900 B | 52 % code, 48 % high-quality web (math, science, encyclopedic) | | Annealing | 40 B | extra math & code + synthetic samples | | SFT | ~4 M | Infinity-Instruct, Orca-AgentInstruct-1M, NuminaMath, ScaleQuest (filtered) | Data cleaning: heuristic filters, MinHash de-duplication, 10-gram benchmark decontamination, reward-model rejection sampling. ### Training Procedure | Hyper-parameter | Value | |-----------------|-------| | Precision | bf16 mixed | | Optimizer | AdamW | | LR (pre-train) | 1.4 e-3, cosine → 0 | | LR (SFT) | 2 e-5, cosine w/ 10 % warm-up | | Batch size | 2048 (pre-train), 128 (SFT) | | Sequence len | 4096 | | Epochs | 1 (pre-train), 1 (anneal), 4 (SFT) | | GPUs | 64 × H800, 5760 GPU-hours total | ## Evaluation ### Benchmarks & Results | Benchmark | InfiR-1B-Base | Llama-3.2-1B | Qwen-2.5-1.5B | |-----------|-------------------|------------------------|-------------------------| | MMLU | 47.24 | 32.74 | 63.03 | | GSM8K | 63.46 |8.11 | 66.57 | | MATH | 31.82 | 3.42 | 31.24 | | HumanEval | 37.80 | 17.68 | 35.37 | | MBPP | 53.40 | 49.0 | 58.37 | | MBPP(3-shot) | 37.6 | 24.8 | 41.4 | ## Technical Specifications ### Model Architecture and Objective - Base: Llama-3.2-1B (32 layers, 32 heads, RoPE, GQA, 2 k ctx → 4 k extended) ## Citation **BibTeX:** ```bibtex @misc{xie2025infir, title={InfiR: Crafting Effective Small Language Models and Multimodal Small Language Models in Reasoning}, author={Xie, Congkai and Cai, Shuo and Wang, Wenjun and others}, year={2025}, eprint={2502.11573}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` **APA:** Xie, C., Cai, S., Wang, W., et al. (2025). *InfiR: Crafting Effective Small Language Models and Multimodal Small Language Models in Reasoning*. arXiv:2502.11573. --- ## Glossary - **SLM**: Small Language Model (<2 B parameters) - **CoT**: Chain-of-Thought prompting or training - **REC**: Renewable Energy Certificate - **PUE**: Power Usage Effectiveness (ratio of total facility power to IT power) ---