wallfacers/weft-lineage-extractor-1.5b
The wallfacers/weft-lineage-extractor-1.5b is a 1.5 billion parameter model, LoRA-fine-tuned on Qwen/Qwen2.5-Coder-1.5B-Instruct, specifically for extracting table-level data lineage from ETL scripts. This model serves as a research artifact demonstrating the severe limitations of synthetic-only training, exhibiting high precision on synthetic data but collapsing on real-world scripts due to memorization of training data. It is primarily intended for studying the 'memorization leak' phenomenon in structured extraction models rather than for production use.
Loading preview...
wallfacers/weft-lineage-extractor-1.5b: A Negative Result in Synthetic-Only Training
This model, based on Qwen/Qwen2.5-Coder-1.5B-Instruct, is a 1.5 billion parameter LoRA-fine-tuned model designed for extracting table-level data lineage from ETL scripts. It was trained exclusively on 10,000 synthetic ETL scripts (Python/Shell) across 9 structural forms, with zero real scripts in its training data.
Key Findings & Limitations
- Synthetic Performance vs. Real-World Failure: Achieves a near-perfect precision of 0.995 on its synthetic held-out set. However, its performance collapses dramatically on real GitHub ETL scripts, yielding a precision of only 0.270.
- Memorization Leak: A significant portion of its hallucinations (22-40% depending on language) are verbatim table names memorized from the synthetic training pool. This demonstrates a critical pathology of synthetic-only training, where the model memorizes the generator's vocabulary rather than learning generalizable patterns.
- Comparison to Other Models: On real Python/Shell ETL scripts, this model (0.270 precision) underperforms general LLMs like Qwen-Max (0.327) and Claude (0.542), and is only marginally better than a regex baseline (0.166).
- Resolution: The subsequent weft-lineage-extractor-3b, trained on real corpora, resolves this issue, achieving a precision of 0.64 and eliminating the memorization leak.
Intended Use Cases
- Reproducing and studying the 'memorization leak' failure in synthetic-only training.
- Serving as a baseline for research into abstention, real-data augmentation, or leak-mitigation strategies.
Not Recommended For
- Production data lineage extraction. For production use, the weft-lineage-extractor-3b, trained on real scripts, is recommended.