atenareply/lfm2.5-1.2b-asterion

atenareply/lfm2.5-1.2b-asterion: Domain-Specific Continued Pre-Training

This model, atenareply/lfm2.5-1.2b-asterion, is a 1.2 billion parameter language model developed by LiquidAI. It represents a Continued Pre-Training (CPT) stage of the LFM2.5-1.2B-Base model, specifically fine-tuned on a unique, fictional "Asterion Space Operations" corpus and Mars Express telemetry data. The training involved a token-budget stratified sampling method to ensure comprehensive coverage of the domain within a 64.3 million token budget, alongside 10% FineWeb-Edu replay to prevent catastrophic forgetting.

Key Capabilities & Innovations

• Domain Expertise: Specialized in a fictional Orbital Mining Corporation (OMC) technical documentation and real-world Mars Express telemetry, making it highly proficient in this niche domain.
• Performance: Achieves significantly improved perplexity (PPL) on its target domains, with PPL of 1.91 on Asterion held-out data (down from 7.36 for the base model) and 1.30 on Mars telemetry (down from 5.96).
• Anti-Forgetting: Incorporates a 10% FineWeb-Edu replay during CPT to maintain general language capabilities, evidenced by a PPL of 7.31 on FineWeb-Edu (compared to 11.87 for the base).
• Stratified Sampling: Utilizes an innovative stratified token-budget sampling approach to ensure even coverage of diverse document types and topics within the domain, maximizing data efficiency.

Intended Use & Limitations

This model is designed as a domain-knowledge backbone for the Asterion project. It is a base-style CPT checkpoint and is not instruction-tuned, meaning it does not follow chat templates or direct instructions. Its primary limitation is its focus on a fictional domain, making it less suitable for general-purpose tasks without further fine-tuning. It also represents a token-budgeted cut of the full corpus, with a larger Gemma-4-12B sibling consuming the entire dataset.