The test
We wrote each of his competition projects as an anonymous 55 to 71-word concept, with names, places, dates and outcomes removed, and checked every text for leaks. Twelve first prizes were each paired with two losing entries from nearby years. Three critics, differing only in their instructions (plain, told his name, or given his framework with project names removed), picked the winner in both orders.
Results
| Critic | Picks the winner |
|---|---|
| Plain | 73% |
| Name only | 77% |
| Framework | 73% |
The framework adds nothing. On the 14 pairs where the model could not name either project, all three critics are at chance.
A second test asked which of two works he cites more often in his own essays and lectures. Every critic scored 95 to 100%, so that split was too easy to compare critics.
What it means
The model recognised 13 of the 53 anonymous concepts, including 8 of his 10 most-cited works. Much of what looks like judgement is recall. A framework helps a model make work in someone’s voice; it does not yet help a model judge as that person would. Judging is the selector’s job, and it has to learn from real picks.
Sources
- Feathers & Wax, Campo Baeza critic pilot, code in
research/campo-baeza-critic.