{"id":"arxiv/2604.00672","url":"https://arxiv.org/abs/2604.00672","tags":["cs.CL","cs.IR","math.ST"],"score":0,"title":"Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness","domain":"cs.CL","source":"arxiv","authors":["Zeyad Ahmed","Paul Sheridan","Michael McIsaac","Aitazaz A. Farooque"],"reviews":0,"subject":"cs.CL","summary":"TF-IDF is a classical formula that is widely used for identifying important terms within documents. We show that TF-IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness (also known as word over-dispersion). In our framework, the alternative hypothesis captures word burstiness by modeling a collection of documents according to a family of beta-binomial distributions with a gamma penalty term on the precision parameter. In contrast, the null hypothesis assumes that words are binomially distributed in collection documents,…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-04-01","reactions":{}}