ehristoforu/mistral-distil-test-0
ehristoforu/mistral-distil-test-0 is a 12 billion parameter language model created by ehristoforu, utilizing a passthrough merge of specific layers from unsloth/Mistral-Nemo-Instruct-2407. This model leverages layers 0-7 and 14-32 from its base model, suggesting an experimental or specialized configuration rather than a full model. Its primary use case is likely for research or specific fine-tuning experiments where a modified layer structure of Mistral-Nemo-Instruct is desired.
Loading preview...
Overview
ehristoforu/mistral-distil-test-0 is a 12 billion parameter language model derived from unsloth/Mistral-Nemo-Instruct-2407 through a passthrough merge. This model was created using mergekit to combine specific layer ranges from the base model, rather than merging distinct models.
Key Characteristics
- Architecture Base: Built upon the
Mistral-Nemo-Instructarchitecture. - Parameter Count: 12 billion parameters.
- Context Length: Supports a context length of 32768 tokens.
- Merge Method: Utilizes a
passthroughmerge method, selectively incorporating layers 0-7 and 14-32 from the source model. - Data Type: Configured to use
bfloat16for its operations.
Good for
- Experimental Research: Ideal for researchers exploring the impact of specific layer configurations on model performance or behavior.
- Custom Fine-tuning: Suitable as a base for further fine-tuning where a modified version of
Mistral-Nemo-Instructis preferred. - Resource-Constrained Deployment: Potentially offers a more efficient deployment option if the excluded layers are not critical for a specific application, though this is not explicitly stated.