Statistical Inference for Language Model Evaluations R package evaluatellm version 0.1.0 Treats language model evaluations as statistical experiments and supplies the inference they require. Provides central limit theorem and cluster-robust standard errors for evaluation scores, paired and unpaired model comparisons, variance decomposition when several responses are drawn per question, control-variate variance reduction, multiplicity adjustment across benchmark suites, and power and minimum detectable effect calculations for planning evaluations, following Miller (2024) <doi:10.48550/arXiv.2411.00640>. For evaluations scored by a model judge, implements agreement statistics against a human gold standard and prediction-powered inference (Angelopoulos et al. 2023) <doi:10.1126/science.adi6000> with the power-tuned estimator of Angelopoulos, Bates and Jordan (2023) <doi:10.48550/arXiv.2311.01453>, so a small set of human labels debiases a large set of judge scores. Leaderboards are supported through bootstrap rank intervals and Bradley-Terry ratings (Bradley and Terry 1952) <doi:10.2307/2334029>. Accepts scores from any evaluation harness. Software Depends: R (>= 4.1.0) Imports: cli (>= 3.6.0), graphics, grDevices, stats, utils Suggests: testthat (>= 3.0.0), knitr, rmarkdown, sandwich Charles Coverdale <charlesfcoverdale@gmail.com> Comprehensive R Archive Network (CRAN) Charles Coverdale [aut, cre, cph] MIT + file LICENSE (https://CRAN.R-project.org/package=evaluatellm/LICENSE) 2026-09-15 application/tgz https://CRAN.R-project.org/package=evaluatellm doi:10.32614/CRAN.package.evaluatellm en-GB