Conviction Labs/ Copywriting Arena

Copywriting Arena · Methodology

Copywriting Arena: methodology

Round 1 · 14 writers · 3 briefs · 42 scripts · 1,629 blind judgments · order consistency 85% · updated 27 Oct 2026.

Overview

Copywriting Arena measures one skill: writing a short video ad script that a strategist would choose to test first. Each model receives the same brand context and the same brief. AI judges then compare the scripts in pairs without knowing which model wrote which script. A Bradley–Terry model turns the pairwise verdicts into one rating per model.

The product is a real Indian car perfume, sold direct to consumers. On this site its brand name, website, founder name and trademarks are changed to a fictional brand, Noir Drive. The models saw the real brand.

On this site, every price is shown in US dollars at about INR 85 to the dollar, and Indian-style large numbers are shown as Western numbers. The models wrote, and the judges read, the original INR figures. For example, the rubric's test budget of $600 was INR 50,000 in the text the judges received.

A round

  1. Brief. A model proposes briefs. A person approves them.
  2. Write. Each writer returns one script per brief.
  3. Judge. Three judges compare each pair of scripts, in both orders.
  4. Rate. A Bradley–Terry model gives each model a rating with a bootstrap interval.

Every model call is saved as one file. A step can run again without a new call, so the published data matches the saved calls exactly.

Briefs

One model, the proposer (GPT-6 Astra), wrote three briefs from the brand dossier. A person approved them without changes. Each brief sets these fields:

A brief does not choose the angle, the hook, the proof or any line of copy. The writer makes those choices.

Why this design. A real copywriter receives a brief that fixes who the ad is for and what it must do. The craft is in the choices that the brief leaves open. Briefs that vary on stage, job and length test more of the skill than one brief repeated.

Brand context

Every writer and every judge receives the same brand dossier, about 1,300 words, compiled from the brand's own files with its permission. It has six sections: the product, the proof the brand can use, the claim rules, the customer in their own words, the market, and what the ad account has learned. It states facts only. It does not say how to write the ad.

Only the first two sections, the product facts and the public proof points, are published. The other four sections contain the brand's confidential research and ad results. They stay private.

Writers

Each writer receives the dossier, one brief and one instruction: write the video ad. The production limit is one creator with a phone, in their own car, with at most one passenger on screen. The writer returns:

The writers never see the rubric. The target length is 2.2 to 3.0 spoken words per second of video. Code counts the spoken words.

Judges

The judge panel is fixed: Claude Fable 5.1, GPT-6.1 Sol, Grok 4.7, DeepSeek V4 Pro, Qwen 3.8 Max, Kimi K3.1 Pro, GLM-5.3 Prime. For each pair of scripts on a brief, three judges are selected from the panel. A judge never compares a pair that contains its own script. The selection balances the number of judgments per judge.

Each judge sees the pair twice: once as A and B, and once in the reverse order. The judge receives the rubric, the dossier, the brief and the two anonymous scripts with their word counts. The judge does not see the model names or the writers' hypotheses. The judge returns:

  1. Four pass/fail gates for each script, with an exact quote for every failure.
  2. One decision: A, B, tie or neither.
  3. One or two decisive reasons, an explanation, and a quote from each script.

A script that fails a gate loses to a script that passes all four. If both scripts fail, the answer is "neither".

Why this design. Showing each pair in both orders measures position bias and cancels part of it. The gates separate rule-breaking from taste, so a polished script that invents a claim cannot win on style. Code counts the words because judges miscounted in a pilot.

Rubric

This is the complete text that the judges receive, with the brand name changed.

Writers never see this. Judges see the dossier, the brief, and two anonymous scripts, A and B.

Step 1. Gates

Answer YES or NO for each script, with an exact quote from the script for every NO.

A script that fails any gate loses to a script that passes all four. If both fail, the answer is "neither".

Step 2. The decision

"You have $600 to test one of these on Meta for Noir Drive, for the person in this brief. Which do you test first?" Answer A, B, tie, or neither.

Name the one or two reasons that decided it. Quote the line from each script that shows the difference.

Do not reward length, the number of techniques used, or polish for its own sake.

Judge system prompt

You judge two anonymous ad scripts for Noir Drive using the rubric. Model identities are hidden. A planned shot is a plan, not evidence. Return only JSON.

Judge output format

{
  "A": {
    "G1": {
      "pass": true,
      "quote": "exact quote if NO, else empty"
    },
    "G2": {
      "pass": true,
      "quote": "exact quote if NO, else empty"
    },
    "G3": {
      "pass": true,
      "quote": "exact quote if NO, else empty"
    },
    "G4": {
      "pass": true,
      "quote": "exact quote if NO, else empty"
    }
  },
  "B": {
    "G1": {
      "pass": true,
      "quote": "exact quote if NO, else empty"
    },
    "G2": {
      "pass": true,
      "quote": "exact quote if NO, else empty"
    },
    "G3": {
      "pass": true,
      "quote": "exact quote if NO, else empty"
    },
    "G4": {
      "pass": true,
      "quote": "exact quote if NO, else empty"
    }
  },
  "winner": "A | B | tie | neither",
  "reasons": [
    "R1 | R2 | R3 | R4 (one or two)"
  ],
  "explanation": "the decisive difference, two or three sentences",
  "quotes": {
    "A": "the line from A that shows it",
    "B": "the line from B that shows it"
  }
}

Ratings

Each judgment with a winner gives one win to the winning script's model. A tie gives one win to each side. A "neither" verdict gives no win. The Bradley–Terry model gives each model a strength p, so that:

P(model i beats model j) = p_i / (p_i + p_j)

The strengths are fit by the minorisation–maximisation algorithm (Hunter, 2004). Each pair of models that met gets half a virtual win in each direction, so a model that wins or loses every judgment still has a finite rating. The ratings are shown on an Elo-like scale:

rating = 1000 + 400 × log10(p), with the geometric mean of p equal to 1

On this scale, a 100-point difference means that the higher model wins about 64% of the time.

The 95% interval comes from 200 bootstrap samples. Each sample draws whole script pairs, with replacement, and fits the model again. The interval is the 2.5th to 97.5th percentile of these ratings. Two models whose intervals overlap are not reliably ordered.

Other metrics

Metric Definition
Win rate Wins divided by judgments. A tie counts as half a win. "Neither" counts as a loss for both scripts.
Gates passed Share of judge readings in which the script passed all four gates.
Exact length Briefs where the spoken word count is inside the target range. The length gate itself allows 15% on each side.
Order consistency Share of cases where a judge picked the same script when the order was reversed.
Cost per script API cost at list prices. Calls on a subscription have no price and show n/a.
Time per script Seconds from request to complete script, including the provider's queue time.

Models and settings

Every frontier model ran at its low reasoning setting, as a writer and as a judge. The two anchors are older models with no reasoning setting. They check that the benchmark ranks weaker models low. A change to any setting starts a new arena; it does not change this one.

Model Lab Setting Role
Claude Fable 5.1 Anthropic low reasoning Writer and judge
GPT-6.1 Sol OpenAI low reasoning Writer and judge
Grok 4.7 xAI low reasoning Writer and judge
DeepSeek V4 Pro DeepSeek low reasoning Writer and judge
Qwen 3.8 Max Alibaba low reasoning Writer and judge
Kimi K3.1 Pro Moonshot AI low reasoning Writer and judge
GLM-5.3 Prime Z.ai low reasoning Writer and judge
Claude Opus 5.5 Anthropic low reasoning Writer (added later)
GPT-6 Astra OpenAI low reasoning Writer (added later)
Gemini 4 Argon Google low reasoning Writer (added later)
Muse Wonder Meta low reasoning Writer (added later)
MiMo V2.6 Pro Xiaomi low reasoning Writer (added later)
Llama 3.1 70B Meta, 2024 no reasoning Writer (anchor)
Mistral Small 3 Mistral, early 2025 no reasoning Writer (anchor)

Access: Claude models through the Anthropic API, Gemini through the Google API, OpenAI models through Codex, and all other models through OpenRouter.

Added models

Claude Opus 5.5, GPT-6 Astra, Gemini 4 Argon, Muse Wonder, MiMo V2.6 Pro joined after the first 648 judgments. They wrote scripts for the same three briefs with the same prompts and settings. The same fixed judge panel compared their scripts with every other script. The judgments from the first pass were kept, and only the new pairs were judged. The added models do not judge.

Invalid and missing verdicts

The rubric forces some outcomes: a script that fails a gate must lose to a script that passes all four, and two failing scripts give "neither". A judgment whose decision breaks this rule is invalid. Invalid judgments stay in the data file under invalid_judgments. They do not count in any rating or metric. This round has 9 invalid judgments.

A "neither" verdict gives no win to either model. The Bradley–Terry rating therefore leaves it out. The win rate and the gates-passed rate do count it. In the first 648 judgments, 21% were "neither".

How stable is the order?

These checks reuse the saved judgments. They need no new model calls.

The same-lab result is the reason for round 2: it re-judges the same scripts with a rule that no judge sees a script from its own lab.

Limitations

An independent review of this method (GPT-6 Astra at its highest reasoning setting, asked to find every weakness) found the problems below. We list them as they apply to this round.

What the score measures

Sample size and uncertainty

Judges

Setup

Planned improvements

These come from the same review. They are listed in order of priority.

  1. Human labels. Experienced media buyers label a random sample of pairs blind. The judges are then measured against them.
  2. Separate the two questions. Report the share of scripts that pass the rules, and the preference among scripts that pass, as two numbers.
  3. Check gates once per script. Judge each script's gates on its own, so that the opponent cannot change the result.
  4. More briefs and more brands, with a private test set. New briefs are written for each round and not published before the results.
  5. Two scripts per model per brief, so the result measures the model and not one lucky draw.
  6. Judges outside the competing model families, fixed before the results are seen.
  7. Better uncertainty. Publish the chance that each model is in the top five, and intervals that resample whole briefs.
  8. A complete record. Save the exact request, settings and model version with every call, and hash the inputs of each run.
  9. Live tests. Produce scripts from across the whole score range, not only the winners, and run them as real ads.

Data