{"id":"arxiv/2601.21471","url":"https://arxiv.org/abs/2601.21471","tags":["cs.LG","math.OC","stat.ML"],"score":0,"title":"Best Arm Identification with LLM Judges and Limited Human","domain":"cs.LG","source":"arxiv","authors":["Ruicheng Ao","Hongyu Chen","Siyang Gao","Hanwei Li","David Simchi-Levi"],"reviews":0,"subject":"cs.LG","summary":"We study fixed-confidence best-arm identification (BAI) where a cheap but potentially biased proxy (e.g., LLM judge) is available for every sample, while an expensive ground-truth label can only be acquired selectively when using a human for auditing. Unlike classical multi-fidelity BAI, the proxy is biased (arm- and context-dependent) and ground truth is selectively observed. Consequently, standard multi-fidelity methods can mis-select the best arm, and uniform auditing, though accurate, wastes scarce resources and is inefficient. We prove that without bias correction and propensity…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-01-29","reactions":{}}