Human-Anchored Inference for Ranking New Models with Large Language Model Judges

Xin Zhou, Sinian Zhang, Zhanyan Yang, Mingyuan Xu, Molei Liu, Doudou Zhou

arXivarxiv/2609.19599Sep 17, 2026math.STscore 0

Human pairwise comparisons provide a reference for evaluating large language models (LLMs), but collecting sufficient judgments for each new release is costly and time-consuming. LLM judges offer a scalable alternative, although their comparisons may differ systematically from human preferences and across judges. We study the ranking of a new model that has received LLM-judge comparisons but no human comparisons. We propose ANCHOR (ANchored Comparisons for Human-reference inference with Orthogonal Riesz correction), which uses historical human and LLM comparisons to learn judge-specific…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2609.19599
curl -X POST /api/v1/records/arxiv/2609.19599/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2609.19599/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'