# PaperScorer-7B-Vision-OpenReview-ICLR SFT fine-tune of **[Qwen/Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)** trained to predict ICLR-style Accept/Reject verdicts on academic papers from the paper text plus per-page screenshots. - **Modality**: text + per-page image screenshots - **Training data**: iclr-21k (85/5/10 balanced split) - **Checkpoint**: step 2648, end of epoch 2 of 4 (7B → ep2 by convention) - **Hyperparams**: LR 1e-6, cosine_then_constant scheduler (decay_ratio 0.75, min_lr_rate 0.001), batch size 16, cutoff_len 24480, framework LLaMA-Factory + FSDP2. ## Quickstart — serve + submit a PDF or LaTeX source dir Easiest path uses the [PaperScorer orchestrator](https://github.com/zlab-princeton/PaperScorer) (paperprep + scoring server + web UI all wired up). Clone, run setup, and chain into the UI: ```bash git clone https://github.com/zlab-princeton/PaperScorer.git cd PaperScorer uv tool install . # installs the `paperscorer` CLI globally on PATH paperscorer setup --serve # in the wizard, pick: size=7B · modality=vision · domain=iclr # → web UI on http://localhost:8003 (PDF upload + LaTeX dir browse) ``` Or hit the API directly (FastAPI on the same port): ```bash # Submit an anonymized PDF; poll for the verdict JOB=$(curl -s -F file=@anonymized.pdf http://localhost:8003/submit | jq -r .job_id) curl http://localhost:8003/status/$JOB # → job dict: state, verdict, p_accept, ... # Submit a LaTeX source directory (anonymized) or an arXiv id curl -X POST http://localhost:8003/submit_latex \ -H "Content-Type: application/json" \ -d '{"path": "/abs/path/to/anonymized_latex_dir"}' curl -X POST http://localhost:8003/submit_arxiv \ -H "Content-Type: application/json" \ -d '{"arxiv_id": "2511.08364"}' ``` Headless one-shot (no server): ```bash paperscorer run /abs/path/to/anonymized.pdf ``` Lower-level: stand up just a vLLM scoring server with pre-prep'd sharegpt rows (skips paperprep): ```bash vllm serve skonan/PaperScorer-7B-Vision-OpenReview-ICLR --task generate --gpu-memory-utilization 0.85 # OpenAI-compat API on :8000 — format prompts per the "Prompt format" section below. ``` ## Test results (in-distribution) Evaluated on `iclr-balanced-test` (2025/26 papers). The prediction is the model's own decision: Accept when the decision-token log-odds `ℓ = logprob(Accept) − logprob(Reject)` is ≥ 0, which is the same as parsing the `\boxed{}` answer. No threshold is tuned. `pA` = predicted accept rate; `A_rec` / `R_rec` = accept / reject recall. | n_test | Acc | AUC | pA | A_rec | R_rec | |---|---|---|---|---|---| | 1670 | 68.3% | 0.736 | 51% | 69.2% | 67.4% | ## Calibration (temperature scaling with a base-rate term) `ℓ` measures the evidence for Accept under the balanced 50/50 prior the model was trained on. To turn it into a calibrated chance of acceptance at a venue's acceptance rate `π`, divide by the temperature **τ = 1.251** and add the prior log-odds: `p_accept = σ(ℓ / τ + log(π / (1 − π)))`, with **τ = 1.251** and **π = 0.314** for this model. ```python import math ell = logprob_accept - logprob_reject # at the decision token; use -50 for a token missing from the top-5 def p_accept(ell, tau=1.251, pi=0.314): # pi = the venue's acceptance rate; default is this model's validation acceptance rate return 1 / (1 + math.exp(-(ell / tau + math.log(pi / (1 - pi))))) p_accept(ell) # chance of acceptance at ICLR's rate (π = 0.314) p_accept(ell, pi=0.5) # balanced 50/50 pool: the shift vanishes, giving σ(ℓ / τ) ``` τ was fit by negative log-likelihood on the natural-acceptance-rate validation split (31.4% accepts), which matches deployment, where most papers are rejected. The base-rate term is inside the fit, with π set to that split's acceptance rate, so the shift handles the base rate and τ only sets how sharp the probabilities are. `p_accept` is the model's chance of acceptance for the paper at a venue that accepts a fraction π of submissions. - **Decisions are unchanged.** The model accepts iff `ℓ ≥ 0`, which is exactly `p_accept ≥ π`. Neither τ nor π ever changes a decision, so the test results above hold as reported. - **Other venues.** Plug in a different venue's acceptance rate as `pi` to get the chance of acceptance there, e.g. `p_accept(ell, pi=0.25)` for a venue that accepts 25% of submissions. - **Balanced pools.** On a 50/50 pool, set `π = 0.5`: the shift vanishes and `p_accept = σ(ℓ / τ)`. Expected calibration error (10 bins): on `iclr-balanced-test` (π = 0.5), 0.049 raw → 0.031 calibrated; on the natural-rate ICLR test split, 0.127 without the base-rate term → 0.023 with it. Method and numbers follow the paper's calibration section (Section 5, "Confidence tracks accuracy"; Appendix I.3, "Calibration Details", Table I.1). ## Note on training-size asymmetry ICLR-trained models saw ~21k examples (4 epochs ≈ 2644 / 5296 steps). Arxiv-trained vs ICLR-trained models saw a ~3× data gap — direct comparisons should account for it. ## Prompt format Inputs are sharegpt-style 3-turn conversations: `system`, `human`, `gpt`. **SYSTEM is the same string across all 8 PaperScorer models. USER preamble differs per training domain.** Vision variants append one `` token per page-screenshot at the end of the user message. ### SYSTEM (all PaperScorer models) ```text You are an expert academic reviewer tasked with evaluating research papers. ``` ### ICLR-trained USER preamble (verbatim) ```text I am giving you a paper. I want to predict its acceptance outcome at ICLR. - Your answer will either be: \boxed{Accept} or \boxed{Reject} - Note: ICLR generally has a ~30% acceptance rate # ...paper body in markdown... ``` ### Vision variant — append `` tokens After the markdown body, append a space-separated run of `` placeholders, one per page screenshot (typically 7–9 for arxiv, 8–10 for iclr). The `images` field on the inference request is a parallel list of PNG paths. ```text ...end of markdown body... ``` ### ASSISTANT (gold) ```text Outcome: \boxed{Accept} ``` or ```text Outcome: \boxed{Reject} ``` At inference, the decision logprobs at the boxed-token position (5th generated token under the `qwen` template) give `ℓ` for temperature scaling (see Calibration above); either parse the text for the decision or read logprobs directly. ## Concrete example (TEXT, ARXIV-trained) ```text [SYSTEM] You are an expert academic reviewer tasked with evaluating research papers. [USER] I am giving you a paper submitted to a top machine-learning venue. Predict its acceptance outcome. - Your answer will either be: \boxed{Accept} or \boxed{Reject} - Note: typical top-tier ML venues have ~25-30% acceptance rates # SSAST: SELF-SUPERVISED AUDIO SPECTROGRAM TRANSFORMER ## Abstract ... ~32k chars of paper body ... [ASSISTANT] Outcome: \boxed{Accept} ``` ## Concrete example (VISION, ICLR-trained) ```text [SYSTEM] You are an expert academic reviewer tasked with evaluating research papers. [USER] I am giving you a paper. I want to predict its acceptance outcome at ICLR. - Your answer will either be: \boxed{Accept} or \boxed{Reject} - Note: ICLR generally has a ~30% acceptance rate # ROBUST TRAINING WITH ENSEMBLE CONSENSUS ## Abstract ... ~1k chars of paper body (vision body is much shorter than text body) ... [ASSISTANT] Outcome: \boxed{Reject} ``` (With `images = [page_1.png, page_2.png, …, page_9.png]` on the request.) ## Related models + datasets in the PaperScorer collection All 8 single-domain SFT models (this one plus 7 siblings) plus the companion **PaperScorer-Text** and **PaperScorer-Vision** datasets live in the [PaperScorer collection](https://huggingface.co/collections/zlab-princeton/paperscorer). Pairwise comparisons across {3B, 7B} × {text, vision} × {arxiv, openreview-iclr} are intended. ## Citation If you use PaperScorer, please cite our NeurIPS 2026 paper: ```bibtex @inproceedings{konan2026paperscorer, title = {PaperScorer: How Predictable Is Paper Acceptance?}, author = {Konan, Sachin and Liu, Jonathan and Liu, Zhuang}, booktitle = {Advances in Neural Information Processing Systems}, year = {2026} } ```

Full Model Card (README)