{"id":"arxiv/2603.26718","url":"https://arxiv.org/abs/2603.26718","tags":["cs.CY","cs.AI","cs.MA","quant-ph"],"score":0,"title":"Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems","domain":"cs.CY","source":"arxiv","authors":["Marcin Abram"],"reviews":0,"subject":"cs.CY","summary":"We analyze the challenges of benchmarking scientific (multi)-agentic systems, including the difficulty of distinguishing reasoning from retrieval, the risks of data/model contamination, the lack of reliable ground truth for novel research problems, the complications introduced by tool use, and the replication challenges due to the continuously changing/updating knowledge base. We discuss strategies for constructing contamination-resistant problems, generating scalable families of tasks, and the need for evaluating systems through multi-turn interactions that better reflect real scientific…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-03-18","reactions":{}}