The test

We took 531 pairs of captions, matched for length, from 60 contests, and asked a model which was funnier, in both orders. The crowd’s votes give the answer.

AgainstModel agreesOne random voter
Median captions71%69%
Top 8 to 12%62%63%
Top 1 to 3%51%57%

Among the best captions, the model is at chance. A smaller model was at chance throughout, with a strong bias for whichever caption came second. Changing the judge’s role moved agreement only seven points; showing the cartoon image instead of a description changed nothing measurable, and the two runs agreed on only 65% of verdicts.

Where else it shows

A model judge also moves with its role, the order of the options and length. Padding a plain idea, with nothing else changed, won “credible” 78% of the time.

What it means

A model judge is a screen. It can weed out weak ideas, where people agree. It cannot choose among good ones, where taste is personal. That choice belongs to the person.

Sources

  1. Countless Lab, caption judge validity, code in research/caption-judge-validity.
  2. New Yorker Cartoon Caption Contest votes.