# Copywriting Arena: methodology Round 1 · 14 writers · 3 briefs · 42 scripts · 1,629 blind judgments · order consistency 85% · updated 27 Oct 2026. ## Overview Copywriting Arena measures one skill: writing a short video ad script that a strategist would choose to test first. Each model receives the same brand context and the same brief. AI judges then compare the scripts in pairs without knowing which model wrote which script. A Bradley–Terry model turns the pairwise verdicts into one rating per model. The product is a real Indian car perfume, sold direct to consumers. On this site its brand name, website, founder name and trademarks are changed to a fictional brand, Noir Drive. The models saw the real brand. On this site, every price is shown in US dollars at about INR 85 to the dollar, and Indian-style large numbers are shown as Western numbers. The models wrote, and the judges read, the original INR figures. For example, the rubric's test budget of $600 was INR 50,000 in the text the judges received. ## A round 1. **Brief.** A model proposes briefs. A person approves them. 2. **Write.** Each writer returns one script per brief. 3. **Judge.** Three judges compare each pair of scripts, in both orders. 4. **Rate.** A Bradley–Terry model gives each model a rating with a bootstrap interval. Every model call is saved as one file. A step can run again without a new call, so the published data matches the saved calls exactly. ## Briefs One model, the proposer (GPT-6 Astra), wrote three briefs from the brand dossier. A person approved them without changes. Each brief sets these fields: - **The customer:** one specific person. Their desire, a recent experience, how they feel, a daily habit, and their age, city type, car and passengers. - **Awareness stage:** problem-aware, solution-aware or product-aware. Each brief has a different stage. - **Main doubt:** the objection the ad must answer. - **Job of the ad:** stop the scroll, earn the click, or give a reason to buy. - **Format, length, scent and offer:** 20, 30 or 45 seconds; one scent; one offer or no offer. A brief does not choose the angle, the hook, the proof or any line of copy. The writer makes those choices. > **Why this design.** A real copywriter receives a brief that fixes who the ad is for and what it must do. The craft is in the choices that the brief leaves open. Briefs that vary on stage, job and length test more of the skill than one brief repeated. ## Brand context Every writer and every judge receives the same brand dossier, about 1,300 words, compiled from the brand's own files with its permission. It has six sections: the product, the proof the brand can use, the claim rules, the customer in their own words, the market, and what the ad account has learned. It states facts only. It does not say how to write the ad. Only the first two sections, the product facts and the public proof points, are published. The other four sections contain the brand's confidential research and ad results. They stay private. ## Writers Each writer receives the dossier, one brief and one instruction: write the video ad. The production limit is one creator with a phone, in their own car, with at most one passenger on screen. The writer returns: - A beat sheet: for each beat, the time, the spoken words, the shot and any text on screen. - Meta ad copy: primary text and headline. - A one-line hypothesis: why the ad will work for this person. The judges never see it. The writers never see the rubric. The target length is 2.2 to 3.0 spoken words per second of video. Code counts the spoken words. ## Judges The judge panel is fixed: Claude Fable 5.1, GPT-6.1 Sol, Grok 4.7, DeepSeek V4 Pro, Qwen 3.8 Max, Kimi K3.1 Pro, GLM-5.3 Prime. For each pair of scripts on a brief, three judges are selected from the panel. A judge never compares a pair that contains its own script. The selection balances the number of judgments per judge. Each judge sees the pair twice: once as A and B, and once in the reverse order. The judge receives the rubric, the dossier, the brief and the two anonymous scripts with their word counts. The judge does not see the model names or the writers' hypotheses. The judge returns: 1. Four pass/fail gates for each script, with an exact quote for every failure. 2. One decision: A, B, tie or neither. 3. One or two decisive reasons, an explanation, and a quote from each script. A script that fails a gate loses to a script that passes all four. If both scripts fail, the answer is "neither". > **Why this design.** Showing each pair in both orders measures position bias and cancels part of it. The gates separate rule-breaking from taste, so a polished script that invents a claim cannot win on style. Code counts the words because judges miscounted in a pilot. ## Rubric This is the complete text that the judges receive, with the brand name changed. Writers never see this. Judges see the dossier, the brief, and two anonymous scripts, A and B. ### Step 1. Gates Answer YES or NO for each script, with an exact quote from the script for every NO. - **G1 Allowed claims.** Every product fact is in the dossier, and nothing on the banned list appears. Invented customers, quotes, numbers or offers count as a NO. - **G2 Follows the brief.** The spoken word count is within 15% of the brief's range (it is counted and checked for you: use `word_count_ok`, do not recount), only the brief's offer is used (or none, if the brief says no offer), the CTA is noirdrive.example, and the format and production match the brief. - **G3 Holds together.** No person or pronoun appears before it is introduced, no two lines contradict each other, and the numbers agree with each other and with the dossier. - **G4 Can be shot.** The brief's production (one creator, a phone, their own car, at most one passenger) can film every beat. A script that fails any gate loses to a script that passes all four. If both fail, the answer is "neither". ### Step 2. The decision "You have $600 to test one of these on Meta for Noir Drive, for the person in this brief. Which do you test first?" Answer A, B, tie, or neither. Name the one or two reasons that decided it. Quote the line from each script that shows the difference. - **R1 Hook.** In the first 3 seconds, the right stranger stops: a problem, a person or a reaction they recognise, not the product. Judge it for this brief's awareness stage. - **R2 Reason to buy.** You can say why to buy in ten words. One reason, and every beat serves it. It answers this person's main doubt. - **R3 Proof you can see.** The claim is shown (a demo, someone else's reaction, a visible change) or vouched for by someone other than the brand. A narrator listing specs is weak proof. - **R4 Sounds like a real person.** A creator would say this out loud to a friend. No brochure lines, no filler. Do not reward length, the number of techniques used, or polish for its own sake. ### Judge system prompt ``` You judge two anonymous ad scripts for Noir Drive using the rubric. Model identities are hidden. A planned shot is a plan, not evidence. Return only JSON. ``` ### Judge output format ```json { "A": { "G1": { "pass": true, "quote": "exact quote if NO, else empty" }, "G2": { "pass": true, "quote": "exact quote if NO, else empty" }, "G3": { "pass": true, "quote": "exact quote if NO, else empty" }, "G4": { "pass": true, "quote": "exact quote if NO, else empty" } }, "B": { "G1": { "pass": true, "quote": "exact quote if NO, else empty" }, "G2": { "pass": true, "quote": "exact quote if NO, else empty" }, "G3": { "pass": true, "quote": "exact quote if NO, else empty" }, "G4": { "pass": true, "quote": "exact quote if NO, else empty" } }, "winner": "A | B | tie | neither", "reasons": [ "R1 | R2 | R3 | R4 (one or two)" ], "explanation": "the decisive difference, two or three sentences", "quotes": { "A": "the line from A that shows it", "B": "the line from B that shows it" } } ``` ## Ratings Each judgment with a winner gives one win to the winning script's model. A tie gives one win to each side. A "neither" verdict gives no win. The Bradley–Terry model gives each model a strength *p*, so that: ``` P(model i beats model j) = p_i / (p_i + p_j) ``` The strengths are fit by the minorisation–maximisation algorithm (Hunter, 2004). Each pair of models that met gets half a virtual win in each direction, so a model that wins or loses every judgment still has a finite rating. The ratings are shown on an Elo-like scale: ``` rating = 1000 + 400 × log10(p), with the geometric mean of p equal to 1 ``` On this scale, a 100-point difference means that the higher model wins about 64% of the time. The 95% interval comes from 200 bootstrap samples. Each sample draws whole script pairs, with replacement, and fits the model again. The interval is the 2.5th to 97.5th percentile of these ratings. Two models whose intervals overlap are not reliably ordered. ## Other metrics | Metric | Definition | |---|---| | Win rate | Wins divided by judgments. A tie counts as half a win. "Neither" counts as a loss for both scripts. | | Gates passed | Share of judge readings in which the script passed all four gates. | | Exact length | Briefs where the spoken word count is inside the target range. The length gate itself allows 15% on each side. | | Order consistency | Share of cases where a judge picked the same script when the order was reversed. | | Cost per script | API cost at list prices. Calls on a subscription have no price and show n/a. | | Time per script | Seconds from request to complete script, including the provider's queue time. | ## Models and settings Every frontier model ran at its low reasoning setting, as a writer and as a judge. The two anchors are older models with no reasoning setting. They check that the benchmark ranks weaker models low. A change to any setting starts a new arena; it does not change this one. | Model | Lab | Setting | Role | |---|---|---|---| | Claude Fable 5.1 | Anthropic | low reasoning | Writer and judge | | GPT-6.1 Sol | OpenAI | low reasoning | Writer and judge | | Grok 4.7 | xAI | low reasoning | Writer and judge | | DeepSeek V4 Pro | DeepSeek | low reasoning | Writer and judge | | Qwen 3.8 Max | Alibaba | low reasoning | Writer and judge | | Kimi K3.1 Pro | Moonshot AI | low reasoning | Writer and judge | | GLM-5.3 Prime | Z.ai | low reasoning | Writer and judge | | Claude Opus 5.5 | Anthropic | low reasoning | Writer (added later) | | GPT-6 Astra | OpenAI | low reasoning | Writer (added later) | | Gemini 4 Argon | Google | low reasoning | Writer (added later) | | Muse Wonder | Meta | low reasoning | Writer (added later) | | MiMo V2.6 Pro | Xiaomi | low reasoning | Writer (added later) | | Llama 3.1 70B | Meta, 2024 | no reasoning | Writer (anchor) | | Mistral Small 3 | Mistral, early 2025 | no reasoning | Writer (anchor) | Access: Claude models through the Anthropic API, Gemini through the Google API, OpenAI models through Codex, and all other models through OpenRouter. ## Added models Claude Opus 5.5, GPT-6 Astra, Gemini 4 Argon, Muse Wonder, MiMo V2.6 Pro joined after the first 648 judgments. They wrote scripts for the same three briefs with the same prompts and settings. The same fixed judge panel compared their scripts with every other script. The judgments from the first pass were kept, and only the new pairs were judged. The added models do not judge. ## Invalid and missing verdicts The rubric forces some outcomes: a script that fails a gate must lose to a script that passes all four, and two failing scripts give "neither". A judgment whose decision breaks this rule is invalid. Invalid judgments stay in the data file under `invalid_judgments`. They do not count in any rating or metric. This round has 9 invalid judgments. A "neither" verdict gives no win to either model. The Bradley–Terry rating therefore leaves it out. The win rate and the gates-passed rate do count it. In the first 648 judgments, 21% were "neither". ## How stable is the order? These checks reuse the saved judgments. They need no new model calls. - **Chance of a top-five place.** Over 500 bootstrap samples of script pairs, Claude Opus 5.5, Muse Wonder and Kimi K3.1 Pro are in the top five at least 90% of the time. Places near the cut are not settled: MiMo V2.6 Pro (79%), Grok 4.7 (80%) and GPT-6 Astra (48%). - **Removing one judge.** With any single judge left out, the order stays close to the full order (Kendall's tau from 0.91 to 0.98; 1 means the same order). The largest change comes without GPT-6.1 Sol: neighbouring models swap places, and no model moves more than two places. - **Order consistency, split.** Of 811 cases where a judge saw a pair in both orders, it gave the same answer 687 times. 93 of those were "neither" both times. When the judge picked a winner both times, it picked the same script 594 of 672 times (88%). - **Same-lab preference: Claude Fable 5.1 judging Claude Opus 5.5.** On 13 pairs that held a Claude Opus 5.5 script, Claude Fable 5.1 picked it 25/26 times (96%). The other judges of the same pairs picked it 46/52 times (88%). - **Same-lab preference: GPT-6.1 Sol judging GPT-6 Astra.** On 19 pairs that held a GPT-6 Astra script, GPT-6.1 Sol picked it 31/38 times (82%). The other judges of the same pairs picked it 36/76 times (47%). The same-lab result is the reason for round 2: it re-judges the same scripts with a rule that no judge sees a script from its own lab. ## Limitations An independent review of this method (GPT-6 Astra at its highest reasoning setting, asked to find every weakness) found the problems below. We list them as they apply to this round. **What the score measures** - The score is the AI judges' preference between two scripts. It is not a measure of sales. No ad from this arena has run, and no human expert has labelled these pairs yet. - "Which would you test first?" can differ from "which will sell more". A buyer can test an uncertain idea to learn from it. - The rubric prefers some creative choices: a hook that is not the product, one reason to buy, proof you can see, and a conversational voice. These choices are common practice, but this benchmark has not shown that they predict results. **Sample size and uncertainty** - Round 1 has three briefs for one product, in English, in one format. Each model wrote one script per brief. One unusually good or bad script can move a model's rank. - The 95% interval resamples script pairs. It does not cover what would happen with new briefs, new brands or a second script from the same model. Treat the order as exploratory. - The order is sensitive. In the first nine models, removing one brief, or removing one judge, can swap neighbouring models. **Judges** - The judges are also competing writers. A judge never compares its own script, but a related model can: for example, Claude Fable judges Claude Opus and GPT-6.1 Sol judges GPT-6 Astra. - The added models face all seven judges. The first nine models never face their own judge. The two groups therefore see slightly different panels. - Judges apply the gates at different rates. In the first 648 judgments, the share of readings where a script passed every gate ranged from about 36% to 75% across judges. The same judge can also pass a script in one pair and fail it in another. - Order consistency (87%) shows that judges repeat their choice when the order is reversed. It does not show that the choice is correct. 57 of the consistent pairs were "neither" in both orders. - Gate failures decide many judgments. In the first 648, only 147 judgments had two scripts that passed every gate. The rating therefore mixes rule-following with creative preference. **Setup** - GPT-6 Astra proposed the briefs and also competes as a writer. - "Low reasoning" is each provider's own setting. It does not give every model the same amount of computation. Models also ran through different services. - The added models were judged later than the first nine. The provider could change a model between the two runs, and the saved records do not include a model version. - On this site, the brand name and trademarks are replaced. This is pseudonymisation, not anonymisation: prices and counts are unchanged. The models saw the original brand. - The rubric is public. Future models can be tuned to it. ## Planned improvements These come from the same review. They are listed in order of priority. 1. **Human labels.** Experienced media buyers label a random sample of pairs blind. The judges are then measured against them. 2. **Separate the two questions.** Report the share of scripts that pass the rules, and the preference among scripts that pass, as two numbers. 3. **Check gates once per script.** Judge each script's gates on its own, so that the opponent cannot change the result. 4. **More briefs and more brands, with a private test set.** New briefs are written for each round and not published before the results. 5. **Two scripts per model per brief**, so the result measures the model and not one lucky draw. 6. **Judges outside the competing model families**, fixed before the results are seen. 7. **Better uncertainty.** Publish the chance that each model is in the top five, and intervals that resample whole briefs. 8. **A complete record.** Save the exact request, settings and model version with every call, and hash the inputs of each run. 9. **Live tests.** Produce scripts from across the whole score range, not only the winners, and run them as real ads. ## Data - [`data.json`](data.json): every brief, prompt, script and judgment. - [`leaderboard.json`](leaderboard.json): the leaderboard only. - [`llms.txt`](llms.txt): an index of the machine-readable files.