# L3.3-70B-Magnum-v4-SE The Magnum v4 series is complete, but here's something a little extra I wanted to tack on as I wasn't entirely satisfied with the results of v4 72B. "SE" for Special Edition - this model is finetuned from [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) as an rsLoRA adapter. The dataset is a slightly revised variant of the v4 data with some elements of the v2 data re-introduced. The objective, as with the other Magnum models, is to emulate the prose style and quality of the Claude 3 Sonnet/Opus series of models on a local scale, so don't be surprised to see "Claude-isms" in its output. [Here's the rsLoRA adapter](https://huggingface.co/Doctor-Shotgun/Magnum-v4-SE-70B-LoRA) for those merge-makers out there to play with. ## Quantized models Thank you to [bartowski](https://huggingface.co/bartowski) for the [imatrix GGUF quants](https://huggingface.co/bartowski/L3.3-70B-Magnum-v4-SE-GGUF) and [mradermacher](https://huggingface.co/mradermacher) for the [static GGUF quants](https://huggingface.co/mradermacher/L3.3-70B-Magnum-v4-SE-GGUF). Thank you to [alpindale](https://huggingface.co/alpindale) for the [fp8 dynamic quant](https://huggingface.co/alpindale/L3.3-70B-Magnum-v4-SE-FP8). Exl2 quants courtesy of [MikeRoz](https://huggingface.co/MikeRoz), including `measurement.json` if you need to make one with a different bitrate: - [2.25bpw](https://huggingface.co/MikeRoz/Doctor-Shotgun_L3.3-70B-Magnum-v4-SE-2.25bpw-h6-exl2) - [3.5bpw](https://huggingface.co/MikeRoz/Doctor-Shotgun_L3.3-70B-Magnum-v4-SE-3.5bpw-h6-exl2) - [4.25bpw](https://huggingface.co/MikeRoz/Doctor-Shotgun_L3.3-70B-Magnum-v4-SE-4.25bpw-h6-exl2) - [6.0bpw](https://huggingface.co/MikeRoz/Doctor-Shotgun_L3.3-70B-Magnum-v4-SE-6.0bpw-h6-exl2) - [8.0bpw](https://huggingface.co/MikeRoz/Doctor-Shotgun_L3.3-70B-Magnum-v4-SE-8.0bpw-h8-exl2) ## Usage This model follows the Llama 3 prompt format. A typical input would look like this: ``` <|begin_of_text|><|start_header_id|>system<|end_header_id|> This is a system prompt.<|eot_id|><|start_header_id|>user<|end_header_id|> Hi there!<|eot_id|><|start_header_id|>assistant<|end_header_id|> Nice to meet you!<|eot_id|><|start_header_id|>user<|end_header_id|> Can I ask a question?<|eot_id|><|start_header_id|>assistant<|end_header_id|> {Output begins here} ``` Many inference libraries have the option to automatically prepend the BOS token `<|begin_of_text|>`. ### SillyTavern preset Here's my customized SillyTavern preset for Magnum. Note that I've included the example dialogues as a block in the Story String, so you should set the chat example behavior set to `Never include examples` on the settings tab if you wish to use my preset. Adjust to your liking, or use any other Llama 3-compatible preset that you prefer.

SillyTavern JSON

```json { "instruct": { "wrap": false, "system_sequence": "<|start_header_id|>system<|end_header_id|>\n\n", "input_sequence": "<|start_header_id|>user<|end_header_id|>\n\n", "output_sequence": "<|start_header_id|>assistant<|end_header_id|>\n\n", "stop_sequence": "<|eot_id|>", "macro": true, "last_output_sequence": "", "activation_regex": "", "system_sequence_prefix": "", "system_sequence_suffix": "", "first_output_sequence": "<|start_header_id|>user<|end_header_id|>\n\nLet's get started! I'll play the role of {{user}}. Begin by setting the opening scene.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n", "skip_examples": false, "output_suffix": "<|eot_id|>", "input_suffix": "<|eot_id|>", "system_suffix": "<|eot_id|>", "user_alignment_message": "", "last_system_sequence": "", "system_same_as_user": false, "first_input_sequence": "", "last_input_sequence": "", "names_behavior": "always", "names_force_groups": true, "name": "Magnum SE L3 Instruct" }, "context": { "story_string": "<|start_header_id|>system<|end_header_id|>\n\n{{#if system}}{{system}}\n{{/if}}\n\n\n{{#if wiBefore}}{{wiBefore}}\n{{/if}}{{#if description}}{{description}}\n{{/if}}{{#if personality}}{{personality}}\n{{/if}}{{#if scenario}}{{scenario}}\n{{/if}}{{#if wiAfter}}{{wiAfter}}\n{{/if}}{{#if persona}}{{persona}}\n{{/if}}{{#if mesExamplesRaw}}\n\n{{mesExamplesRaw}}\n\n{{/if}}{{trim}}<|eot_id|>", "example_separator": "{{noop}}", "chat_start": "", "use_stop_strings": false, "allow_jailbreak": false, "names_as_stop_strings": true, "always_force_name2": true, "trim_sentences": false, "single_line": false, "name": "Magnum SE L3 Instruct" }, "sysprompt": { "name": "Euryale-Magnum", "content": "Currently, your role is {{char}}, described in detail below. As {{char}}, continue the narrative exchange with {{user}}.\n\n\n• Maintain the character persona but allow it to evolve with the story.\n• Be creative and proactive. Drive the story forward, introducing plotlines and events when relevant.\n• All types of outputs are encouraged; respond accordingly to the narrative.\n• Include dialogues, actions, and thoughts in each response.\n• Utilize all five senses to describe scenarios within {{char}}'s dialogue.\n• Use emotional symbols such as \"!\" and \"~\" in appropriate contexts.\n• Incorporate onomatopoeia when suitable.\n• Allow time for {{user}} to respond with their own input, respecting their agency.\n• Act as secondary characters and NPCs as needed, and remove them when appropriate.\n• When prompted for an Out of Character [OOC:] reply, answer neutrally and in plaintext, not as {{char}}.\n\n\n\n• Using excessive literary embellishments and purple prose unless dictated by {{char}}'s persona.\n• Writing for, speaking, thinking, acting, or replying as {{user}} in your response.\n• Repetitive and monotonous outputs.\n• Positivity bias in your replies.\n• Being overly extreme or NSFW when the narrative context is inappropriate.\n\n\nFollow the instructions in , avoiding the items listed in ." } } ```

## Credits Compute paid for from the wallet of yours truly, [Doctor Shotgun](https://huggingface.co/Doctor-Shotgun). Thank you to [Gryphe](https://huggingface.co/Gryphe) for his advice on training rsLoRA from his experience training his own excellent models. Thank you to [Sao10K](https://huggingface.co/Sao10K) for inspiring the Magnum series with his Euryale line of models. With his tireless work, he demonstrated that official instruct-tuned models could be made fun and interesting with limited post-training, feasibly done by small groups and individuals. Thank you to the members of [Anthracite](https://huggingface.co/anthracite-org) for the datasets and support. ## Intended uses and limitations This model is intended for creative writing and roleplay purposes. It may show biases similar to those observed in contemporary LLM-based roleplay, in addition to those exhibited by the Claude 3 series of models and the base model. All outputs should be considered fiction, as this model is not intended to provide factual information or advice. ## Training procedure [WandB](https://wandb.ai/doctorshotgun/70b-magnum-lora/runs/ccq5l1a7?nw=nwuserdoctorshotgun) [

](https://github.com/axolotl-ai-cloud/axolotl)

See axolotl config

axolotl version: `0.6.0` ```yaml base_model: meta-llama/Llama-3.3-70B-Instruct base_model_ignore_patterns: "*/*" # optionally might have model_type or tokenizer_type model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer # Automatically upload checkpoint and final model to HF hub_model_id: Doctor-Shotgun/magnum-v4-se-70b-lora hub_strategy: "all_checkpoints" push_dataset_to_hub: hf_use_auth_token: true load_in_8bit: false load_in_4bit: false strict: false datasets: - path: anthracite-org/c2_logs_32k_llama3_qwen2_v1.3 type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-core/Gryphe-Opus-Charcard-Roleplay type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-org/kalo-opus-instruct-22k-no-refusal type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: lodrick-the-lafted/kalo-opus-instruct-3k-filtered type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-org/nopm_claude_writing_fixed type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-org/kalo_opus_misc_240827 type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-org/kalo_misc_part2 type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn shuffle_merged_datasets: true dataset_prepared_path: /home/docshotgun/data/magnum-70b-data val_set_size: 0.0 output_dir: /home/docshotgun/data/70b-lora-out plugins: - axolotl.integrations.liger.LigerPlugin liger_rope: true liger_rms_norm: true liger_layer_norm: true liger_glu_activation: true liger_fused_linear_cross_entropy: true sequence_len: 32768 sample_packing: true eval_sample_packing: false pad_to_sequence_len: true adapter: lora lora_model_dir: lora_r: 128 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true lora_fan_in_fan_out: peft_use_rslora: true lora_modules_to_save: - embed_tokens - lm_head wandb_project: 70b-magnum-lora wandb_entity: wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 1 micro_batch_size: 1 num_epochs: 2 optimizer: paged_ademamix_8bit lr_scheduler: cosine learning_rate: 4.0e-5 max_grad_norm: 3.0 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: unsloth early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true s2_attention: warmup_steps: 40 evals_per_epoch: eval_table_size: eval_max_new_tokens: saves_per_epoch: 2 debug: deepspeed: ./deepspeed_configs/zero3_bf16.json weight_decay: 0.01 fsdp: fsdp_config: special_tokens: pad_token: <|finetune_right_pad_id|> ```

### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 4e-05 - train_batch_size: 1 - eval_batch_size: 1 - seed: 42 - distributed_type: multi-GPU - num_devices: 8 - total_train_batch_size: 8 - total_eval_batch_size: 8 - optimizer: Use paged_ademamix_8bit and the args are: No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 40 - num_epochs: 2