SLBM/affine-k100-king-head
SLBM/affine-k100-king-head is a 35.1 billion parameter language model developed by SLBM, featuring a unique architecture that combines hv4 attention and routers with 100% sitting-king expert FFNs. This model also incorporates sitting-king `embed_tokens`, `lm_head`, and final RMSNorm. It is an evaluation candidate designed with specific architectural components for specialized performance characteristics.
Loading preview...
Model Overview
SLBM/affine-k100-king-head is a 35.1 billion parameter model developed by SLBM, distinguished by its specialized architectural recipe. This model is an internal evaluation candidate, not submitted to the SN120 benchmark, indicating its experimental or specialized nature.
Key Architectural Components
The model's unique design integrates several advanced components:
- hv4 attention and hv4 routers: These components are central to its attention mechanism and data routing.
- 100% sitting-king expert FFNs: This indicates a specific configuration for its feed-forward networks, suggesting a focus on expert-based processing.
- Sitting-king
embed_tokens/lm_head/ final RMSNorm: These elements are consistently applied across the embedding, language model head, and normalization layers, pointing to a cohesive architectural strategy.
Parent Models
The affine-k100-king-head model is a composite, built from specific versions of parent models:
- Controllers: Sourced from
angryaffine/Affine-5dhnrnxlw4-hv4. - Experts + Vocab Head: Derived from
tojointhecommunity/affine-5efg6cm3yl-king.
This composition suggests an iterative development process, combining proven components to achieve specific performance goals. While specific use cases are not detailed, its specialized architecture implies a focus on particular types of tasks or research objectives.