Summarize this content to 100 words:
[Submitted on 2 Mar 2026 (v1), last revised 4 Mar 2026 (this version, v2)]
View a PDF of the paper titled Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization, by Felipe Maia Polo and 4 other authors
View PDF
HTML (experimental)
Abstract:Moving beyond evaluations that collapse performance across heterogeneous prompts toward fine-grained evaluation at the prompt level, or within relatively homogeneous subsets, is necessary to diagnose generative models’ strengths and weaknesses. Such fine-grained evaluations, however, suffer from a data bottleneck: human gold-standard labels are too costly at this scale, while automated ratings are often misaligned with human judgment. To resolve this challenge, we propose a novel statistical model based on tensor factorization that merges cheap autorater data with a limited set of human gold-standard labels. Specifically, our approach uses autorater scores to pretrain latent representations of prompts and generative models, and then aligns those pretrained representations to human preferences using a small calibration set. This sample-efficient methodology is robust to autorater quality, more accurately predicts human preferences on a per-prompt basis than standard baselines, and provides tight confidence intervals for key statistical parameters of interest. We also showcase the practical utility of our method by constructing granular leaderboards based on prompt qualities and by estimating model performance solely from autorater scores, eliminating the need for additional human annotations.
Submission history From: Felipe Maia Polo [view email] [v1]
Mon, 2 Mar 2026 16:12:46 UTC (4,952 KB)
[v2]
Wed, 4 Mar 2026 01:26:23 UTC (4,952 KB)
