mhhmm/typescript-instruct-20k-v2
mhhmm/typescript-instruct-20k-v2 is a 13 billion parameter instruction-tuned causal language model developed by mhhmm, fine-tuned from CodeLlama-13b-hf. This model specializes in code generation, particularly for TypeScript, demonstrating improved performance over its base model on the MultiPL-E benchmark. It is optimized for code-related tasks with a 4096 token context length.
Loading preview...
Model Overview
mhhmm/typescript-instruct-20k-v2 is a 13 billion parameter instruction-tuned model, building upon the codellama/CodeLlama-13b-hf architecture. It has been specifically fine-tuned to enhance its performance in code generation tasks, particularly for TypeScript.
Key Capabilities
- TypeScript Code Generation: Demonstrates improved
Pass@kscores on the MultiPL-E benchmark for TypeScript compared to the base Code Llama - Instruct 13B model (42.4% vs 39.0%). - Instruction Following: Fine-tuned with an instruction dataset to better understand and execute coding prompts.
- CodeLlama Foundation: Benefits from the robust code understanding and generation capabilities of the CodeLlama family.
Training Details
The model was trained for 1 epoch with a learning rate of 0.0002, using Adam optimizer and a cosine learning rate scheduler. The training involved a total batch size of 16 across 2 GPUs, achieving a final validation loss of 0.4268.
Evaluation
Evaluation was conducted using the MultiPL-E benchmark, a standard for assessing code generation models. The README provides detailed instructions and a script for reproducing the evaluation, noting the significant GPU resources (A100 40GB) required for this process.