The test
We took 531 pairs of captions, matched for length, from 60 contests, and asked a model which was funnier, in both orders. The crowd’s votes give the answer.
| Against | Model agrees | One random voter |
|---|---|---|
| Median captions | 71% | 69% |
| Top 8 to 12% | 62% | 63% |
| Top 1 to 3% | 51% | 57% |
Among the best captions, the model is at chance. A smaller model was at chance throughout, with a strong bias for whichever caption came second. Changing the judge’s role moved agreement only seven points; showing the cartoon image instead of a description changed nothing measurable, and the two runs agreed on only 65% of verdicts.
Where else it shows
The same kind of judge preferred our ideas to plain ones 72% of the time on four briefs. In the one blind human rating so far, the founder rated the plain ideas higher. Our judge also moves with its role, the order of options and length: padding a plain idea, with nothing else changed, won “credible” 78% of the time.
What it means
A model judge is a screen. It can weed out weak ideas, where people agree. It cannot choose among good ones, where taste is personal. Every claim we make about preference is checked against real human picks.
Sources
- Feathers & Wax, caption judge validity, code in
research/caption-judge-validity. - New Yorker Cartoon Caption Contest votes.