wallfacers/weft-lineage-extractor-jvm-1.5b

Overview

This model, weft-lineage-extractor-jvm-1.5b, is a research artifact from a study by wallfacers, focusing on the limitations of synthetic-only training for ETL table-lineage extraction. It is a 1.5 billion parameter model, fine-tuned using LoRA on the Qwen2.5-Coder-1.5B base model, with specific extensions for synthetic Scala/Java forms.

Key Findings & Characteristics

• Negative Result Reproduction: The model's primary purpose is to demonstrate that synthetic-only training induces a significant "verbatim memorization leak" and poor performance on real-world data.
• Cross-Language Failure: It specifically highlights that this failure reproduces, and is worse, across different languages (Scala/Java in this case) compared to Python/Shell.
• Low Real-World Precision: On real Scala/Java ETL code, the model achieves a precision of only 0.165, despite showing near-perfect performance (~0.99) on synthetic held-out data.
• High Memorization Leak: A substantial 40.8% of hallucinations are table names recited verbatim from the synthetic training pool, indicating a memorization issue rather than true understanding.
• Comparison: This model's performance (0.165 real precision, 40.8% leak) is notably worse than other points in the study, such as the 1.5B (Py/Sh) model (0.270 precision, 22.4% leak), underscoring the challenge of cross-language generalization with synthetic data.

Intended Use

• Research: This model is intended for reproducing and studying the cross-language reproduction of the memorization-leak failure in lineage extraction.
• Not for Production: It is explicitly not recommended for production lineage extraction. For practical applications, the real-corpus 3B model (which achieves 0.64 real precision with ~0% leak) is recommended.