TAUR-dev/rankalign-v6-gemma-2-2b-d0.15-e2-hc-b2d-dbl-all-tco-ln-fsx

Overview

This model, rankalign-v6-gemma-2-2b-d0.15-e2-hc-b2d-dbl-all-tco-ln-fsx, is a fine-tuned checkpoint derived from the google/gemma-2-2b base model, developed under the rankalign project. It features 2.6 billion parameters and a context length of 8192 tokens.

Key Training Details

The model underwent specific fine-tuning for a task identified as hypernym-concat-bananas-to-dogs-double-all over two epochs. Notable training parameters include a delta of 0.15, online typicality correction, and length normalization. It was trained with a preference loss weight of 1, and both NLL validator and generator weights set to 0, indicating a focus on preference-based learning rather than traditional negative log-likelihood for validation or generation.

Intended Use and Evaluation

This model is designed for tasks involving hypernym relation extraction, as evidenced by its training task and the provided evaluation scripts. The evaluation process, detailed in the README, involves testing across various hypernym-related tasks such as hypernym-bananas, hypernym-dogs, hypernym-elephants, and others. These evaluations utilize random split types, zero-shot generation, few-shot discrimination, and validator log-odds with typicality correction, suggesting its specialization in discerning hierarchical semantic relationships.