Revision pulls human work toward the model

We took 30 real entries to a museum competition in Helsinki and asked a model to revise them. Measured in units of the spread of the model’s own plain answers, the human entries started 13.1 units from the model’s typical answer. One revision brought them to 6.4, about half the way. Eight rounds brought them to 3.8, about 70% of the way.

Distance of human entries from the model’s typical answer

As submitted13.1
After one revision6.4
After eight revisions3.8
The same 30 Helsinki entries, revised by a model. Units are the spread of the model’s own plain answers.

Revised together, ideas collapse

In 432 revisions, ideas revised alongside each other converged round after round, whether one model revised them or several roles took turns. Two kinds of outside material changed that:

  • A random fragment per idea per round kept ideas distinct, but a judge rated them far less credible.
  • A real fact per idea per round kept them apart and credible.

Retelling shows the same

Breithaupt and colleagues had people and ChatGPT retell stories in chains of three. People kept changing the story, bringing in 55 to 60% new words and concepts at each step. ChatGPT gave a competent summary on the first retelling and changed little after that.

What it means

A closed loop is a converging loop. Every step of our workflow that revises an idea brings in new outside material, and we read it against the open loop idea: invention escapes collapse through fresh evidence and selection.

Sources

  1. Countless Lab, drift, flux and evidence runs, code in research/cllm-divergence.
  2. Fritz Breithaupt et al., Humans create more novelty than ChatGPT when asked to retell a story, Scientific Reports, 2024.