wallfacers/weft-lineage-extractor-3b
The wallfacers/weft-lineage-extractor-3b is a 3 billion parameter code model, fine-tuned by wallfacers, designed for extracting table- and column-level data lineage from ETL scripts as structured JSON. This LoRA fine-tuned and merged model is deployable on a single 12 GB consumer GPU. It excels at providing efficient and accurate lineage extraction, trained on real-world GitHub ETL scripts with tri-vendor consensus silver labels. The model offers specialized branches for balanced performance, maximum column quality, or maximum table recall.
Loading preview...
Weft Lineage Extractor 3B: Efficient Data Lineage from ETL Scripts
This model, developed by wallfacers, is the efficient tier of the Weft lineage-extractor family, a 3 billion parameter code model specifically designed to extract table- and column-level data lineage from ETL scripts. It outputs structured JSON and is optimized for deployment on a single 12 GB consumer GPU.
Key Capabilities & Features
- Structured JSON Output: Extracts lineage as
{"reads": [...], "writes": [...]}with table and optional column details. - Specialized Branches: Offers three variants to balance table and column extraction:
main: Best balanced performance (Table F1: 0.781, Column F1: 0.825).tri-column-specialist: Optimized for column F1 (0.931).tri-table-specialist: Optimized for table recall (0.776).
- Loss Weighting Innovation: Utilizes table-token loss weighting (W=3) to improve table recall from 0.645 to 0.734 without increasing model capacity, addressing gradient imbalance where column tokens outnumber table tokens 4.39:1.
- High-Quality Training Data: Trained on 1,154 real GitHub ETL scripts with tri-vendor consensus silver labels (qwen-max, deepseek-v4-pro, GPT-5.6), ensuring robust learning of real lineage patterns.
- Cost-Effective Deployment: Self-hosted on a consumer GPU, the marginal cost for lineage extraction approaches electricity cost, significantly lower than cloud LLM APIs for high-volume batch processing.
Use Cases
This model is ideal for developers and data engineers needing to automatically extract data lineage from various ETL scripts (Python, Shell, Scala, Java, including Spark/Flink jobs). It helps in understanding data flow, governance, and impact analysis within data ecosystems. While it reduces the need for human review by narrowing down disagreement cases, it's important to note that it does not eliminate review entirely, especially for vendor-disagreement cases.