What we score
The current public dataset covers 26,749 papers from six 2025 OpenReview venues: ICLR, ICML, NeurIPS, TMLR, COLM, and MIDL. It contains 128,723 public official reviews; 99,671 have completed AI scoring.
Inputs are limited to paper title, abstract, decision metadata, and public official review text. PDF full text, author rebuttals, discussion replies, and administrative comments are excluded from the public ranking pipeline.
The score
Each review is assessed on three separate dimensions from 0 to 100: helpfulness, friction, and tone risk. The public review-usefulness score gives most weight to actionable help while applying smaller penalties for needless friction and risky tone.
The Needs Attention board uses a different ranking signal. A high number there means a review may deserve scrutiny; it does not mean the review is good.
Guardrails cap usefulness for non-substantive, contradictory, or extremely short reviews. Placeholder text, author-response language, thank-you replies, and similar low-signal material are excluded from public boards.
Model and cost
The 2025 production batch was scored with gpt-5.4-nano through the OpenAI Batch API using a versioned structured-output prompt. The model receives the title, abstract, and one official review at a time. Raw model dimensions are retained; public composite scores are computed deterministically.
Known limitations
- There is not yet a representative human gold-standard set or published inter-annotator agreement result for this exact three-dimension scorer.
- Model judgments may favor longer, more explicit prose and may react unevenly to non-native English, field-specific style, sarcasm, or terse venue conventions.
- An excerpt can omit context found elsewhere in the same review. Always check the linked OpenReview source.
- Venue forms and reviewing cultures differ. Cross-venue comparisons should be treated as exploratory.
- The system evaluates public text. It does not infer, identify, or rank anonymous reviewers as people.
Corrections
Every detail view links to its OpenReview source and includes a score-report form. Reports should identify the paper, review, disputed score, and reason. Credible privacy, attribution, or safety concerns are prioritized for hiding while they are reviewed.