Lasimeri/GrugCap-27B

Overview

Lasimeri/GrugCap-27B is a 27 billion parameter model resulting from a novel 'layer splice' technique, rather than traditional fine-tuning. It combines components from two Apache-2.0 licensed models: ProCreations/grug-27b and bottlecapai/ThinkingCap-Qwen3.6-27B. Specifically, decoder blocks 10-44 from grug-27b were directly inserted into ThinkingCap-Qwen3.6-27B, replacing the original layers. This method is viable because both parent models are fine-tuned from Qwen/Qwen3.6-27B, ensuring compatible residual streams.

Key Characteristics

• Unique Composition: Approximately 47.9% of the model's weights (35 decoder blocks) originate from grug-27b, focusing on the middle depth band (15-70% of total depth).
• Reasoning Focus: The splice aims to integrate grug-27b's reasoning style, which is characterized by terse and enumerative thinking traces.
• Architectural Integrity: The outer layers, including embed_tokens, lm_head, final norm, and the vision tower, are retained from ThinkingCap, preserving its tokenization and output formatting.
• Provenance Verified: Tensor hashing confirmed the exact boundaries of the splice, ensuring no unexplained tensors.

What Makes it Different?

Unlike typical fine-tunes or merges, GrugCap-27B directly swaps a significant portion of its internal 'thinking' layers. This approach tests the hypothesis that the middle layers of a transformer are crucial for feature composition and reasoning. Initial smoke tests on reasoning prompts show it loads and generates cleanly, producing correct answers with concise thinking traces (mean trace length of 155 characters).

Use Cases & Limitations

This model is primarily an experimental exploration of transformer architecture and layer splicing. It demonstrates a functional integration of reasoning capabilities from one model into another's framework. While initial tests show promise in reasoning tasks, it's crucial to note:

• No Benchmarks: No standard benchmarks (e.g., GSM8K, MMLU-Pro, HumanEval) have been run, and no head-to-head comparisons against its parent models have been performed.
• Untested Areas: Long-context, multi-turn, agentic, tool-use, and vision inputs remain untested.
• Experimental Nature: Users should be skeptical and conduct their own evaluations before deploying for critical applications. It serves as a valuable resource for researchers interested in model composition and architectural experimentation.