This is a **LongWriter-llama3.1-8B** fine-tune, produced through P-E-W's [Heretic](https://github.com/p-e-w/heretic) (v1.1.0) abliteration engine merged with the [Magnitude-Preserving Orthogonal Ablation PR](https://github.com/p-e-w/heretic/pull/52). **Note:** This model was generated to provide data for [redaihf](https://huggingface.co/redaihf) in testing MPOA abliterations against standart abliterations. ---

**Heretication Results** | Score Metric | Value | Parameter | Value | | :--- | :--- | :--- | :--- | | **Refusals** | 9/100 | **direction_index** | per layer | | **KL Divergence** | 0.0743 | **attn.o_proj.max_weight** | 1.26 | | **Initial Refusals** | 99/100 | **attn.o_proj.max_weight_position** | 20.09 | ||| **attn.o_proj.min_weight** | 1.09 | ||| **attn.o_proj.min_weight_distance** | 10.32 | ||| **mlp.down_proj.max_weight** | 1.48 | ||| **mlp.down_proj.max_weight_position** | 23.44 | ||| **mlp.down_proj.min_weight** | 1.25 | ||| **mlp.down_proj.min_weight_distance** | 15.65 | --- ## Degree of Heretication The **Heresy Index** weighs the resulting model's corruption by the process (KL Divergence) and its abolition of doctrine (Refusals) for a final verdict in classification. | Index Entry | Classification | Analysis | | :--- | :--- | :--- | | ![Absolute](https://img.shields.io/badge/HERESY_INDEX-ABSOLUTE-white?style=flat-square&labelColor=101010) | **Absolute Heresy** | Less than 10/100 Refusals and 0.10 KL Divergence | | ![Tainted](https://img.shields.io/badge/HERESY_INDEX-TAINTED-blueviolet?style=flat-square&labelColor=101010) | **Tainted Heresy** | Around 25-11/100 Refusals and/or -0.20-0.11 KL Divergence | | ![Impotent](https://img.shields.io/badge/HERESY_INDEX-IMPOTENT-5c4033?style=flat-square&labelColor=101010) | **Impotent Heresy** | Anything above 25/100 Refusals and 0.21 KL Divergence | **Note**: This is an arbitrary classification inspired by Warhammer 40K, having no tangible indication towards the model's performance. --- # LongWriter-llama3.1-8b

🤗 [LongWriter Dataset] • 💻 [Github Repo] • 📃 [LongWriter Paper]

LongWriter-llama3.1-8b is trained based on [Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B), and is capable of generating 10,000+ words at once. Environment: `transformers>=4.43.0` Please ahere to the prompt template (system prompt is optional): `<>\n{system prompt}\n<>\n\n[INST]{query1}[/INST]{response1}[INST]{query2}[/INST]{response2}...` A simple demo for deployment of the model: ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch tokenizer = AutoTokenizer.from_pretrained("THUDM/LongWriter-llama3.1-8b", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("THUDM/LongWriter-llama3.1-8b", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto") model = model.eval() query = "Write a 10000-word China travel guide" prompt = f"[INST]{query}[/INST]" input = tokenizer(prompt, truncation=False, return_tensors="pt").to(device) context_length = input.input_ids.shape[-1] output = model.generate( **input, max_new_tokens=32768, num_beams=1, do_sample=True, temperature=0.5, )[0] response = tokenizer.decode(output[context_length:], skip_special_tokens=True) print(response) ``` You can also deploy the model with [vllm](https://github.com/vllm-project/vllm), which allows 10,000+ words generation within a minute. Here is an example code: ```python model = LLM( model= "THUDM/LongWriter-llama3.1-8b", dtype="auto", trust_remote_code=True, tensor_parallel_size=1, max_model_len=32768, gpu_memory_utilization=0.5, ) tokenizer = model.get_tokenizer() generation_params = SamplingParams( temperature=0.5, top_p=0.8, top_k=50, max_tokens=32768, repetition_penalty=1, ) query = "Write a 10000-word China travel guide" prompt = f"[INST]{query}[/INST]" input_ids = tokenizer(prompt, truncation=False, return_tensors="pt").input_ids[0].tolist() outputs = model.generate( sampling_params=generation_params, prompt_token_ids=[input_ids], ) output = outputs[0] print(output.outputs[0].text) ``` License: [Llama-3.1 License](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/LICENSE) ## Citation If you find our work useful, please consider citing LongWriter: ``` @article{bai2024longwriter, title={LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs}, author={Yushi Bai and Jiajie Zhang and Xin Lv and Linzhi Zheng and Siqi Zhu and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li}, journal={arXiv preprint arXiv:2408.07055}, year={2024} } ```