thinkingdbx/cobolx-1.5b
The thinkingdbx/cobolx-1.5b is a 1.5 billion parameter model, based on Qwen2.5-Coder-1.5B-Instruct, specifically designed to analyze COBOL programs. It excels at identifying program dependencies like files, copybooks, and calls, and extracting data item information. This model is optimized for understanding existing COBOL codebases, particularly for initial code assessment and dependency mapping.
Loading preview...
COBOLX 1.5B: A Specialized COBOL Code Analyzer
The thinkingdbx/cobolx-1.5b model is a 1.5 billion parameter language model, built upon the Apache 2.0 licensed Qwen2.5-Coder-1.5B-Instruct base, specifically fine-tuned for analyzing COBOL programs. Its primary function is to provide insights into COBOL code structure and dependencies, making it a valuable tool for understanding legacy systems.
Key Capabilities & Performance
This model demonstrates strong performance in extracting critical information from COBOL source code, as evidenced by tests on 90 unseen programs from the X-COBOL research dataset. It achieves:
- 100% accuracy in identifying files read or written by a program.
- 80% full accuracy (89% item recall) for identifying copybooks included.
- 60% full accuracy (71% item recall) for identifying external program calls.
- 66% overall full accuracy and 80% overall item recall across various questions, indicating it finds about four out of five relevant items correctly.
Training and Uniqueness
The model was trained on 9,112 question-answer pairs, including real COBOL from The Stack (6MB), 3,500 generated COBOL programs, and 700 deliberately broken COBOL programs. A key aspect of its training is that all answers were computed by rules from the source, not generated by other LLMs, ensuring factual accuracy and preventing license propagation. It also includes 20% non-COBOL instruction data to maintain general understanding.
Limitations and Best Practices
- Dead Code Analysis: It performs poorly on identifying dead code (53% item recall) as this requires complex reasoning about control flow.
- Long Lists: The model may drop items when generating very long lists; manual verification is recommended for comprehensive data item extraction.
- Hallucination: It can confidently generate incorrect information if unsure, especially if a dependency is commented out.
- Program Size: Optimal performance is observed with programs under 6KB, as larger programs were not part of the training data.
- Answer Quality: Focus on the first two sentences of an answer, as quality tends to degrade afterwards.
Ideal Use Cases
This model is best suited for initial assessment of COBOL codebases, answering questions like "what does this program touch?" (e.g., files, copybooks, calls, data items). It is not recommended for tasks requiring deep control flow analysis or making decisions about code deletion.