# Instruction Pre-Training: Language Models are Supervised Multitask Learners (EMNLP 2024) This repo contains the **finance model developed from Llama3-8B** in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491). We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. ***Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training.** In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning. **In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.**

**************************** **Updates** **************************** * **2026/1/23: Released [LLM-in-Sandbox Elicits General Agentic Intelligence](https://huggingface.co/papers/2601.16206), where the data of `Instruction Pre-Training` achieves robust generalization in agentic RL!** * 2024/11/30: Released the multimodal version of the instruction synthesizer: [Visual Instruction Synthesizer](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains) * 2024/9/20: Our paper has been accepted by EMNLP 2024 main conference🎉 * 2024/9/11: Updated [FAQ on continual pre-training from Llama3](https://huggingface.co/instruction-pretrain/instruction-synthesizer) * 2024/8/29: Updated [guidelines](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B) on evaluating any 🤗Huggingface models on the domain-specific tasks * 2024/7/31: Updated pre-training suggestions in the `Advanced Usage` section of [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer) * 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M. The performance trend on downstream tasks throughout the pre-training process:

* 2024/6/21: Released the [paper](https://huggingface.co/papers/2406.14491), [code](https://github.com/microsoft/LMOps), and [resources](https://huggingface.co/instruction-pretrain) ## Resources **🤗 We share our data and models with example usages, feel free to open any discussions at [this page](https://huggingface.co/papers/2406.14491)! 🤗** - Thanks to the demo [davanstrien/instruction-synthesizer](https://huggingface.co/spaces/davanstrien/instruction-synthesizer) for implementing our approach - Context-Based Instruction Synthesizer: [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer) - Fine-Tuning Data for the Synthesizer: [ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection) - General Models Pre-Trained from Scratch (on 100B tokes): - [InstructLM-500M](https://huggingface.co/instruction-pretrain/InstructLM-500M) - [InstructLM-1.3B](https://huggingface.co/instruction-pretrain/InstructLM-1.3B) - Domain-Specific Models Pre-Trained from Llama3-8B: - [Finance-Llama3-8B](https://huggingface.co/instruction-pretrain/finance-Llama3-8B) - [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B) - General Instruction-Augmented Corpora: [general-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/general-instruction-augmented-corpora) - Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): [medicine-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/medicine-instruction-augmented-corpora) ## Domain-Adaptive Continued Pre-Training Following [AdaptLLM](https://huggingface.co/AdaptLLM/finance-chat), we augment the domain-specific raw corpora with instruction-response pairs generated by our [context-based instruction synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer). ### 1. To chat with the finance-Llama3-8B model: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("instruction-pretrain/finance-Llama3-8B") tokenizer = AutoTokenizer.from_pretrained("instruction-pretrain/finance-Llama3-8B") # Put your input here, NO prompt template is required user_input = '''Use this fact to answer the question: Title of each class Trading Symbol(s) Name of each exchange on which registered Common Stock, Par Value $.01 Per Share MMM New York Stock Exchange MMM Chicago Stock Exchange, Inc. 1.500% Notes due 2026 MMM26 New York Stock Exchange 1.750% Notes due 2030 MMM30 New York Stock Exchange 1.500% Notes due 2031 MMM31 New York Stock Exchange Which debt securities are registered to trade on a national securities exchange under 3M's name as of Q2 of 2023?''' inputs = tokenizer(user_input, return_tensors="pt", add_special_tokens=True).input_ids.to(model.device) outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0] answer_start = int(inputs.shape[-1]) pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True) print(pred) ``` ### 2. To evaluate any Huggingface LMs on domain-specific tasks (💡New!) You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct). 1). Set Up Dependencies ```bash git clone https://github.com/microsoft/LMOps cd LMOps/adaptllm pip install -r requirements.txt ``` 2). Evaluate the Model ```bash # Select the domain from ['biomedicine', 'finance'] DOMAIN='finance' # Specify any Huggingface LM name (Not applicable to models requiring specific prompt templates) MODEL='instruction-pretrain/finance-Llama3-8B' # Model parallelization: # - Set MODEL_PARALLEL=False if the model fits on a single GPU. # We observe that LMs smaller than 10B always meet this requirement. # - Set MODEL_PARALLEL=True if the model is too large and encounters OOM on a single GPU. MODEL_PARALLEL=False # Choose the number of GPUs from [1, 2, 4, 8] N_GPU=1 # Whether to add a BOS token at the beginning of the prompt input: # - Set to False for AdaptLLM. # - Set to True for instruction-pretrain models. # If unsure, we recommend setting it to False, as this is suitable for most LMs. add_bos_token=True # Run the evaluation script bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU} ``` ## FAQ on Continual Pre-Training from LLama3 **Q1: Do you use the official Llama3 instruction prompt for pre-training?** No, the provided Llama3 instruction prompt is designed for the [instruction-tuned model](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), but our continual pre-training is conducted on the [pre-trained base model](https://huggingface.co/meta-llama/Meta-Llama-3-8B) where only BOS (`<|begin_of_text|>`) and EOS (`<|end_of_text|>`) tokens are required. **Q2: For the general instructions from OpenOrca, do you concatenate each instruction with its output using '\n'?** No, as mentioned in the pre-training suggestions, we use a simple whitespace to concatenate each question with its response for the general instruction data from OpenOrca. This is because OpenOrca's data is already templated with diverse natural languge templates (such as those with `\n`), so a whitespace is sufficient to formulate the data. Note that when using our templated instruction-augmented texts, you don't need to add any concatenations. **Q3: What about those system prompts in OpenOrca?** We simply discard the system prompts. **To put it all together, the text before tokenization looks like this:** ```python general_instruction_response_text = "<|begin_of_text|>{question} {response}<|end_of_text|>" instruction_augmented_text = "<|begin_of_text|>{instruction augmented text}<|end_of_text|>" ``` Then, for tokenization, you don't need to add BOS and EOS token ids. The tokenization code looks like this: ```python text_ids = tokenizer(text, add_special_tokens=False, **kwargs).input_ids ``` ## Citation If you find our work helpful, please cite us: [Instruction Pre-Training](https://huggingface.co/papers/2406.14491) (EMNLP 2024) ```bibtex @article{cheng2024instruction, title={Instruction Pre-Training: Language Models are Supervised Multitask Learners}, author={Cheng, Daixuan and Gu, Yuxian and Huang, Shaohan and Bi, Junyu and Huang, Minlie and Wei, Furu}, journal={arXiv preprint arXiv:2406.14491}, year={2024} } ``` [Adapt LLM to Domains](https://huggingface.co/papers/2309.09530) (ICLR 2024) ```bibtex @inproceedings{ cheng2024adapting, title={Adapting Large Language Models via Reading Comprehension}, author={Daixuan Cheng and Shaohan Huang and Furu Wei}, booktitle={The Twelfth International Conference on Learning Representations}, year={2024}, url={https://openreview.net/forum?id=y886UXPEZ0} } ```