Build on a model, or train one

The question

If models narrow in training, the clean fix looks like a model trained not to. There are four places to act: the weights, how answers are drawn, what goes into the prompt, and the choosing afterwards. Published work covers the first two and part of the third. We ran two tests of our own on the third and the fourth.

Where the narrowing enters

Researchers at Sheffield traced Olmo 3, an open model family, stage by stage. The “Think” model lost 62% of the base model’s answer diversity in the supervised stage, 4% more in preference tuning, and regained 4 points in reinforcement learning, ending at 38%. The “Instruct” model lost 38%, then 23%, then 5%, ending at 34%. Reinforcement learning applied straight to the base model kept 93% on average.

On creative writing, the models that narrowed most also won most. Think and Instruct lost about 63% of their diversity and beat the base model in more than 97% of comparisons by a model judge. The reinforcement-only model kept about all of its diversity and won about half. The authors read that as consistent with its training having no reward for creative writing.

Other studies fit that picture and leave gaps. Base models beat their aligned versions on randomness, mixed-strategy games and creative originality, while aligned models favour pleasant writing; models that score better on common benchmarks tend to do worse on those tasks (West and Potts). Reinforcement learning with checkable rewards raised the first-try score on math and code but lowered coverage at 256 tries, and every correct path the trained model found was already in the base model (Yue and colleagues). Across more than 70 models, answers to the same open-ended question from different models had an average pairwise similarity of 71% to 82%, and the authors leave open whether pretraining data, alignment or memorization is the cause (Jiang and colleagues).

Bigger is not wider. NoveltyBench found larger models within a family often less diverse than smaller ones. In our four-model test the smallest, Haiku 5.5, put 32% of its plain answers in its largest idea family, against 61% for Sonnet 5.5 and 65% for Opus 5.5 (Models write alike, not necessarily worse).

For the one pipeline traced stage by stage, most of the flattening came from the supervised and preference stages. No study traces a closed frontier model, so for Opus 5.5 the stage is not known from outside.

What a retrain would buy

Training for diversity works on open models of 7 to 9 billion parameters. MoDA (Yuan and colleagues, September 2026) trains one model on a reward for answers that differ from those of its own other “modes”. On Qwen3-8B it raised sentence-embedding diversity 265% on held-out Infinity-Chat prompts while general capability rose 10.3%. DivLM (Solati Dehkordi and Lampos) continues pretraining on creative writing, then trains with a reward for variety in genre, tone, style and named entities; its diversity metrics rose more than 9% on average over alternatives, on two model families. Selective Layer Restoration puts a range of layers back to their pretrained weights, with no training, on Llama, Qwen and Gemma models. Skobelev and colleagues show that with enough supervised data, a model’s diversity converges to that of the data it is trained on.

Each of these needs the weights. We reach Opus 5.5 through an interface, without its weights, so none of these can be applied to it. Training a frontier model from scratch is the other route. The cost of the most compute-intensive runs has grown 2.4 times a year since 2016, and Epoch AI projects that the largest will pass a billion dollars by 2027.

Changing the input works where drawing again does not

Raising the temperature is the obvious try. In a Columbia test of product ideas on GPT-4o, going from temperature 1.0 to 1.5 moved idea categories from 22 to 23 and combinations from 89 to 97, against 28 and 209 for people; at 2.0 the text was garbled. What helped was changing who the model was asked as. Personas of ordinary people spread the ideas but made later ideas in a run stick to earlier ones; asking the model to reason step by step first fixed that, and the two together beat people. Two other models gave the same result.

Our earlier test on Sonnet 5.5 showed the same split. Asking for something original moved the largest idea family from 66% to 53%. Asking for five answers at a time from the unlikely end of the model’s own range, a random fragment per idea, and one person’s framework per idea moved it to 16%, 13% and 26%.

The same inputs on Opus 5.5

Opus 5.5 is the model we build on, and the most collapsed of the four on plain prompts. We ran the same ten briefs, prompts, seeds and 30 frameworks as the Sonnet test, 30 concepts per brief per input, and had a separate model group the concepts into idea families without knowing which input made them.

All four inputs spread Opus. Each took the largest family more than 20 points below plain, below plain on 10 of 10 briefs (the originality prompt on 8), and above plain on distinct ideas on 10 of 10.

Opus recovers at least as much as Sonnet. Verbalized Sampling and random fragments land where they did on Sonnet. The originality prompt works much better on Opus, and frameworks spread it further, to 12% against 26%.

The originality prompt still trails random fragments and frameworks on distinct ideas on 10 of 10 briefs, 7.0 in 30 concepts against 8.5 and 9.5, and it failed on two. On the backyard brief 67% of its concepts sat in one family against 60% plain, and on the Helsinki museum brief 47% against 20%. Frameworks beat random fragments on distinct ideas on 10 of 10 briefs on Opus, and on 2 of 10 on Sonnet. On the Helsinki museum brief, where Sonnet’s frameworks converged to 3.2 distinct ideas, Opus’s reach 8.0.

A framework’s mark is clearer on Opus. We gave a judge each framework-made concept and five candidate people, the true one and four others, with chance at 20%. An Opus judge named the right person for 300 of 300 concepts, against 73% for Sonnet’s concepts. Because a judge from the same family could have an edge, we added Haiku 5.5 as a second judge. It named 90.3% [86.7, 94.0], against 53% on Sonnet’s concepts. Twenty-six of the 300 concepts name the person outright, and the other 274 are named just as often. A judge naming the person shows that the framework steered the idea. It does not show that the idea is good.

Choosing: a small model trained on people’s picks

Spread gives a model many ideas; something has to choose among them. Model judges are weakest where people disagree: Jiang and colleagues find models, reward models and model judges less well calibrated to human ratings on responses that draw different preferences. LitBench shows a way round. Reward models trained on 43,827 human story comparisons agreed with held-out human choices 78% of the time, against 73% for the best untrained model used as a judge, Claude 3.7 Sonnet. On new model-written stories, people picked the reward model’s favourite 57% of the time against 41%, while the best judge was at chance.

We built the smallest version and tested it twice. The picker is a 32-million-parameter embedding model that runs on a CPU, with a ridge regression fitted to human choices.

A jury’s tiers. In the Helsinki museum competition the jury sorted 623 entries into six tiers. We trained on 615 of them in five folds and scored 202 entries the picker had never seen, against Opus 5.5 reading the same plain-text restatement of each idea. The picker’s rank correlation with the jury’s tier was +0.37 [+0.25, +0.49], against +0.48 [+0.37, +0.58] for Opus; text length alone gave +0.09. At telling a top-tier entry from a middle-or-lower one, the picker was right 64% of the time and Opus 74%. The picker had about three quarters of what Opus had, and was below it on every measure. The curve was still rising at 492 training entries. Averaging the picker’s rank with Opus’s added nothing, +0.01 [−0.06, +0.09] over Opus alone.

Everyday answers. PRISM has people rating answers from 21 models. On 294 sets of four answers from 150 people, the picker, trained on the 1,246 other people’s 60,796 ratings, chose the person’s favourite 45.2% [39.7, 51.0] of the time, with chance at 25%. Sonnet 5.5 reading the answers got 44.2% and Opus 5.5 got 44.6% [39.0, 51.0]. That picker used which model wrote each answer as a feature. Without it the picker got 39.5% [34.3, 45.3], five points lower and inside the noise. Most of the signal in PRISM is which model wrote the answer, a shared quality signal that a reader of the answers also picks up. From 100 people on, the picker was within about 2 points of its full-data score.

So a small picker trained on a few hundred to a few thousand human choices reaches a frontier reader’s level on everyday answers and three quarters of the way on a jury’s tiers. Each was trained and tested on the same kind of choice: one jury, one crowd.

What it means for our work

We keep Opus 5.5 and put the diversity and the choosing around it.

The weights are out of reach: every retraining result so far is on open models of 7 to 9 billion parameters, and a frontier model costs more than any studio holds. What goes into the model is within reach, and it works. Random fragments and Verbalized Sampling spread Opus as far as they spread Sonnet, and frameworks spread it further, from 60% of answers in one idea to 12%, with a mark a separate model names 90% to 100% of the time.

Choosing is where training can add something. The reinforcement-only Olmo run kept the base model’s spread and won half its story comparisons; it had no reward for stories. A picker trained on people’s picks is a score of that kind, and it runs on a CPU.

The order is: keep the model, widen what goes in, then choose with a score learned from people.