# Remnant GLM4 32b (series 1) [English](./README.md) | [简体中文](./README-cn.md) *There's a wisp of dust in the air. It feels like its from a bygone era, but you don't know where from. It lands on your tongue. It tastes nice.* ![image/png](https://cdn-uploads.huggingface.co/production/uploads/634262af8d8089ebaefd410e/_ovgodU331FO4YAqFGCnk.png) Remnant is a series of finetuned LLMs focused on SFW and NSFW roleplaying and conversation. ## Quants GGUF (**IF YOU ARE USING KOBOLD.CPP, PLEASE USE `--nobostoken`.**): - [bartowski's imatrix quants](https://huggingface.co/bartowski/allura-org_remnant-glm4-32b-GGUF) EXL3: - Todo! EXL2 (if it ever gets supported): - Todo! MISC: - Todo! ## Recommended Settings Chat template: GLM4 Samplers: - `1.0` temp - `0.1` min_p [An example Sillytavern template is available here.](https://huggingface.co/allura-org/GLM4-9B-Neon-v2/blob/main/GLM-Shingane-v1.json) ## Credits Humongous thanks to Allura, ilya <3 Big thanks to the developers of Axolotl (whose training framework I used), Zhipu (whose model I used), Prime Intellect (whose GPUs I used), and my bank (whose debit card I used) ## Misc [

](https://github.com/axolotl-ai-cloud/axolotl)

See axolotl config

axolotl version: `0.8.0` ```yaml # === Model Configuration === base_model: THUDM/GLM-4-32B-0414 # e.g. "mistralai/Mistral-Small-24B-Instruct-2501" load_in_8bit: false load_in_4bit: true # === Training Setup === num_epochs: 2 micro_batch_size: 3 gradient_accumulation_steps: 2 sequence_len: 8192 sample_packing: true pad_to_sequence_len: true # === Hyperparameter Configuration === optimizer: adamw_8bit # Apollo-mini configuration: #optim_args: "proj=random,rank=1,scale=128.0,scale_type=tensor,update_proj_gap=200" # Regular Apollo configuration: # optim_args: #optim_target_modules: all_linear learning_rate: 1e-5 lr_scheduler: rex weight_decay: 0.01 warmup_ratio: 0.05 # === LoRA Configuration === adapter: qlora lora_r: 16 lora_alpha: 32 lora_dropout: 0.25 lora_target_modules: lora_target_linear: true # === Data Configuration === datasets: - path: allura-org/inkmix-v3.0 type: chat_template split: train field_messages: conversations message_field_role: from message_field_content: value train_on_eos: all dataset_prepared_path: last_run_prepared chat_template: jinja chat_template_jinja: | [gMASK]{%- for msg in messages %}{%- if msg.role == 'system' %}<|system|> {{ msg.content }}{%- elif msg.role == 'user' %}<|user|> {{ msg.content }}{%- elif msg.role == 'assistant' %}<|assistant|> {{ msg.content }}{%- endif %}{%- endfor %}{% if add_generation_prompt %}<|assistant|>{% else %}<|user|>{% endif %} # === Plugins === plugins: - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin # === Hardware Optimization === gradient_checkpointing: offload gradient_checkpointing_kwargs: use_reentrant: false cut_cross_entropy: true deepspeed: deepspeed_configs/zero3_bf16.json # === Wandb Tracking === wandb_project: glm4-32b-inkmix-v3 # === Checkpointing === saves_per_epoch: 2 save_total_limit: 3 # === Advanced Settings === output_dir: /workspace/ckpts bf16: auto flash_attention: true train_on_inputs: false group_by_length: false logging_steps: 1 trust_remote_code: true ```