The test
We took 531 pairs of captions, matched for length, from 60 contests, and asked a model which was funnier, in both orders. The crowd’s votes give the answer.
| Against | Model agrees | One random voter |
|---|---|---|
| Median captions | 71% | 69% |
| Top 8 to 12% | 62% | 63% |
| Top 1 to 3% | 51% | 57% |
Among the best captions, the model is at chance. A smaller model was at chance throughout, with a strong bias for whichever caption came second. Changing the judge’s role moved agreement only seven points; showing the cartoon image instead of a description changed nothing measurable, and the two runs agreed on only 65% of verdicts.
Where else it shows
A model judge also moves with its role, the order of the options and length. Padding a plain idea, with nothing else changed, won “credible” 78% of the time.
What it means
A model judge is a screen. It can weed out weak ideas, where people agree. It cannot choose among good ones, where taste is personal. That choice belongs to the person.
Sources
- Countless Lab, caption judge validity, code in
research/caption-judge-validity. - New Yorker Cartoon Caption Contest votes.