wallfacers/weft-lineage-extractor-0.5b

weft-lineage-extractor-0.5b: A Research Artifact for Synthetic Training Leaks

This model, weft-lineage-extractor-0.5b, is a 0.5 billion parameter LoRA fine-tune based on Qwen2.5-Coder-Instruct. Developed by wallfacers, it represents the smallest scale point in a research study focused on the negative result of synthetic-only training for ETL table-lineage extraction.

Key Findings & Characteristics

• Synthetic-Only Training: The model was trained exclusively on synthetic Python/Shell ETL scripts, with no real-world data.
• Verbatim Memorization Leak: It exhibits a significant "verbatim memorization leak," where 37.4% of hallucinations on real GitHub scripts are table names directly recited from its synthetic training data.
• Performance Collapse: While achieving 0.994 precision on synthetic held-out data, its precision dramatically collapses to 0.243 when evaluated on real GitHub ETL scripts.
• Direction Accuracy: The model shows a real direction accuracy of 0.369.
• Comparison to Larger Models: This 0.5B variant demonstrates the worst leak among the study's models, with the leak shrinking as model scale increases (e.g., 1.5B and 3B synthetic models). However, only training with real data (as seen in weft-lineage-extractor-3b) effectively closes the real-world performance gap.

Intended Use

• Reproducing Research: This model is specifically intended for reproducing and studying the synthetic-training memorization-leak failure at a minimal scale.
• Not for Production: It is explicitly not recommended for production lineage extraction. For practical applications, users should refer to the weft-lineage-extractor-3b model, which was trained on a real corpus and achieves significantly higher precision (0.64).