Callsign "Casanova": SubMaroon and [Indexnusrefather](https://huggingface.co/Indexnusrefather) present to you: # Kitchoon-v3-26B-A4B Kitchoon trained from scratch on [*Gemma-4-26B-A4B-it-heretic-antislop*](https://huggingface.co/Tetratype/Gemma-4-26B-A4B-it-heretic-antislop) — this time with the dense MLP trained alongside OV, on 8192-token windows. The most stable release of my work. **Thinking works!** The head is untouched: the base already handles slop at the token level, so instead of a head injection, slop phrases and slop constructions were masked out of the training loss. The Kitchoon series is designed as an RP- and **ERP**-oriented model, focusing on character development and featuring a neutral writing style. ## What was changed | | Base ([heretic-antislop](https://huggingface.co/Tetratype/Gemma-4-26B-A4B-it-heretic-antislop)) | Kitchoon-V3 | | --- | --- | --- | | Attention OV (`v_proj` + `o_proj`) | stock | LoRA, layers **10–29**, **36** targets, baked at **0.5** | | Dense MLP (`gate_proj` + `up_proj` + `down_proj`) | stock | LoRA, layers **10–20**, **33** targets, baked at **0.5** | | `lm_head`, QK, MoE experts, router, embeddings, vision tower | stock | **identical** | ---

☕ Support

If this model was useful to you, you can support my work with a small donation.
It helps me pay for GPU time and keep releasing new models and datasets.

Gram (TON network): UQB6cmEHWIcuiwsjgO_C0Uz4fyX3Puj09BAB3hEVVpDDcQ7K
Boosty donation: Link

--- # Details - OV and dense MLP were trained together as one LoRA, but baked with separate strengths (both at **0.5**). - MoE experts and the router were never touched — checked by module name and by matrix shape. - Baked with stochastic bf16 rounding, so the baked strength matches the tested one.

LoRA details

| Parameter | Value | |---|---| | Rank / alpha | OV 32 / 64, MLP 16 / 32 | | Learning rate | 2.5e-5, cosine | | Epochs | 1 (754 steps) | | Context | 8192 tokens | | Eval loss | 4.32 → 1.41 |

What was masked out of the loss (and why slop didn't come back)

The dataset is sloppier than the base, so training on it naively would bring slop back. The model still sees these spans in context — it just gets no gradient for reproducing them: - **Slop phrases.** Stock cliché phrases plus ~400 n-grams that showed up across many different characters in the dataset ("voice dropping to a", "a flicker of", "tilts her head"...). - **Slop constructions.** "Not just X, but Y", "It wasn't an X; it was a Y" — the antislop base doesn't catch these. - **"X — or Y?" endings.** The final either-or question sentence of a reply. - **Echoing the user.** A first sentence that repeats the user's words back. - **Card echo.** Text copied verbatim from the character card. In testing, slop stayed at the level of the clean base.

**I would like to express my immense gratitude to [Indexnusrefather](https://huggingface.co/Indexnusrefather), with whom I work, for providing the dataset.**

Dataset

English RP dataset, ~2.8k multi-turn conversations built on character cards. Cleaned before training: - removed cards with underage characters and real people; - fixed ~640 system prompts that instructed the model to write *for* the user; - long conversations split into 8192-token chunks instead of being truncated; - the first and/or last paragraph randomly trimmed in part of the replies, to break up identical openings and hook-question endings; - character name prefixes (`Name:`) stripped from replies.

## Using it > [!Note] > Recommended settings: > >| Parameter | Value | >|---|---| >| **Temperature** | `0.85` | >| **Min-P** | `0.05` | >| **Repetition penalty / DRY** | off | > > Thinking works in both modes, including at 5–7K tokens of context. > Replies run long (~400 words) — give it a response length of at least 900 tokens. ## What to expect In testing, compared to the base: - characters stick closer to the card — they pull in side characters, habits and lore from the description on their own; - more dialogue and a stronger character voice; - strict formatting rules and structured presets are followed reliably; - far fewer "X — or Y?" endings and "not just X" constructions. Known quirks, still there: - characters sometimes repeat the user's words back in their first line; - most paragraphs start with a name or "He/She" — this comes from Gemma itself; - *smirk* and the em dash have not fully left the building. ## Feedback This is an experiment, and I mostly test it on my own cards, so my coverage is narrow. If you try it, I would really like to know: - How it holds up in long sessions — scene logic, who is where, who is talking to whom. - Whether the quirks above bother you in actual play. - Your sampler settings, if you got clearly better or worse results than with the defaults. Open a thread in the Community tab, I read all of them. ## Credits Thanks to [Tetratype](https://huggingface.co/Tetratype) for the antislop base, to `coder3101` for the heretic version it is built on, and to Sam Paech for the auto-antislop pipeline. Thanks to the entire `26B-Suite` team for their intellectual support. Special thanks to [Naphula](https://huggingface.co/Naphula) and [redaihf](https://huggingface.co/redaihf). You guys are awesome! **And special thanks, of course, to [Indexnusrefather](https://huggingface.co/Indexnusrefather) for his excellent dataset.**

Full Model Card (README)