Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI
Read the talk
Why AI Writing Still Reads Like Slop
Louis-François Bouchard explains how Towards AI built ToneBench to evaluate writing in a specific creator’s voice—and what repeated drafts, biased judges, expensive reruns, and human editing taught the team about writing with AI.
From a talk by Louis-François Bouchard
At a glance
Ideas worth remembering
Slop includes generality, empty content, repetition, hallucination, and wrong voice as well as conspicuous vocabulary. A word blacklist addresses only part of the problem.
Evaluate repeated generations. ToneBench’s five drafts per task expose cases where one excellent sample hides inconsistent typical performance.
Detailed rubrics and examples make qualitative judging more inspectable, but blind judges can still prefer their own model family. Use different families and add human comparisons.
Choose for the work required to finish a draft, not just its best one-shot output. Cost and reasoning effort belong in that decision.
Bouchard’s practical workflow supplies extensive style examples, limits model review, explores cheap drafts when useful, and always ends with human editing. The review and merging benefits are observations, not quantified results in this talk.
Slop is more than a list of forbidden words
Better models have not made generated-sounding writing disappear. Louis-François Bouchard of Towards AI opens with that frustration: the output can become more capable while still feeling unpleasant to read. Recognizable words such as “delve” are the easy part to spot. The harder problem is writing that remains generic, empty, repetitive, unreliable, or unlike the person whose name will appear on it.
His working definition separates six failures:
- Overused vocabulary: Words associated with model output appear where the intended writer would not use them.
- Generality: The text could apply to many themes or projects because it supplies little that is specific.
- Empty content: Pleasant-sounding sentences fail to communicate anything useful.
- Repetition: The writing keeps restating that thin content.
- Hallucination: Statements cannot be trusted simply because the prose sounds confident.
- Wrong voice: The text sounds like an average of everyone, rather than a recognizable person with particular choices and interests.
That definition changes what an evaluation must do. Removing a few conspicuous words would leave most of these failures intact. ToneBench, the benchmark introduced here, therefore tries to measure the quality of the writing as a whole, including whether a model can adapt to a particular voice. Bouchard does not want to rebuild EQ-Bench Slop Score; he wants to evaluate the script that someone would actually have to use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The target is a script its creator can read aloud
Towards AI had a practical reason to build this evaluation. Its small education team used AI while producing AI-engineering material, then repeatedly spent time reviewing and repairing the results. Bouchard describes hundreds of published videos, thousands of articles, a released book, and another book in progress. At that volume, choosing a writing model means choosing how much unwanted prose the team will have to remove.
The team also had both sides of the evaluation problem in-house: AI engineers who built evaluation pipelines, and creative writers and editors who worked on the content. Their target was deliberately subjective. Could a model write a video script that Bouchard could read aloud without cringing at words he would never use? That supplies a concrete acceptance test, while narrowing the meaning of a high score: success means approaching this creator’s writing preferences, not discovering a universal definition of good prose.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ten tasks, five drafts, and the danger of a lucky sample
Each task begins with a brief and research, then asks a model to generate an entire video script. The actual prompt contains much more than that short description: details about how Bouchard talks and writes, examples from previous videos and other content, guidance on storytelling and active sentences, and instructions for the particular assignment. The model gets both the material to explain and examples of how the explanation should sound.
ToneBench uses ten script tasks with carefully developed reference scripts and rubrics. Some reference scripts are published and some are not; the team does not disclose which scripts belong to the benchmark, an approach intended to help check contamination. For every model, the pipeline generates five drafts for each task, sends the generations to four judges, and calculates a final leaderboard score. Bouchard also mentions testing an experimental judge without having decided whether to keep it, so the recording leaves that experimental addition unsettled.
Why pay for five drafts instead of one? The useful comparison is the spread of outcomes, not just the best-looking script. Bouchard describes a model whose exceptional draft scored about as well as a stronger competitor, but only once in five generations. Judging that one sample alone would hide the model’s inconsistency. In the team’s observations, weaker models also tended to have wider variation. Five drafts became their compromise between measuring that variability and keeping evaluation costs manageable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate voice, storytelling, compliance, and accuracy
ToneBench scores nine metrics, which Bouchard groups into five broader aspects. These separate questions that are easy to blur together when reading a polished draft:
- Tone and voice: Does the script sound like Bouchard? This tests adaptation to a specific writer, rather than a generally agreeable style.
- Writing quality: Does it have effective hooks, emotion, transitions, and the other qualities expected of a good story?
- Instruction following: Does it satisfy the requested length and format, including placing images and other requested elements appropriately?
- Research accuracy: Does it use the supplied research correctly, or introduce hallucinations?
- Slop: Does it contain the patterns the benchmark is trying to discourage? This includes EQ-Bench Slop Score alongside the team’s own measurements.
The research brief gives accuracy its own reference point. A script can have a good hook and still misrepresent the material; it can follow the format and still sound nothing like its intended narrator. Keeping those criteria separate makes a score more interpretable than asking only whether the whole script seems good.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give judges examples before asking for a number
A judge receives the generated script, Bouchard’s own script as guidance, the brief and research, and extensive metric-specific instructions. Bouchard describes around a thousand words of examples and directions for each metric. Model identity is hidden from the judge. Length checking is handled separately in code, leaving the language model to assess writing quality rather than estimate everything from the prose.
Tone and voice, the main metric, uses a two-stage judgment: first choose among six categories, then assign a rating within the selected category. The rubric describes the desired voice and provides examples of what different scores mean. Bouchard reports that category-first scoring worked better for this qualitative task than a binary decision, categories alone, or a directly assigned percentage.
The complete evaluation contains nine metric scores, drawing on both model judgments and code-based checks. The judge’s contribution is scored assessments with justifications and specific examples explaining its choices; the separately computed length metric does not come from that qualitative judgment. Passage-based reasons make the judged scores more useful than unexplained numbers: a reviewer can connect an assessment to the writing. Automation is necessary at the scale described here. With 120 models, ten tasks, and five drafts per task, the generation side alone produces 6,000 scripts—too many for the team to review individually.
Which inputs support each kind of assessment? The branching below shows the reference material and rubrics feeding the qualitative judgment, while the same generated script also goes through a code-based length check. Length does not become an implicit proxy for writing quality.
Two evaluation paths for the same generated script
Read the diagram as text
- Generated script. The candidate output, with its model identity hidden from LLM judges.
- Brief, research, and creator’s script. qualitative — Task evidence and a reference for the intended writing.
- Detailed rubrics and examples. qualitative — Metric-specific directions; tone uses six categories followed by a rating.
- LLM judges. qualitative — Assess writing quality using the supplied task and scoring guidance.
- Judged scores with reasons. qualitative — The judge’s contribution to the evaluation, with justifications and examples from the writing.
- Code-based length check. mechanical — Length is checked outside the LLM’s writing assessment.
- Generated script → LLM judges: Writing to assess.
- Brief, research, and creator’s script → LLM judges: Task evidence and guidance.
- Detailed rubrics and examples → LLM judges: Scoring instructions.
- LLM judges → Judged scores with reasons: Returns qualitative assessments.
- Generated script → Code-based length check: Length checked separately.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Blind judging still has model-family preferences
Hiding the authoring model’s name does not remove every preference. Bouchard reports that judges often favored outputs from their own model family, with a particularly strong preference in the OpenAI judgments. The useful comparison is how a family’s outputs score under different judges: a favorable score from a related judge may partly reflect shared writing preferences. His joke about rival labs disliking each other is not evidence about their training; the practical finding is that judge choice can change a model comparison.
That is why he recommends using judges from different model families when choosing a model for a task. A single judge can make its own preferences look like an objective ranking. Multiple families provide different perspectives on the same drafts, although the talk does not establish that this eliminates the bias.
The team also added human comparisons. Bouchard and three teammates manually compared 1,000 A/B pairs, using substantial, comparable excerpts from two generations rather than entire scripts. That scope makes human review more practical while adding a human preference signal to an automated evaluation. The talk does not report an agreement rate between those choices and the LLM scores, so the 1,000 comparisons should not be read as a quantified validation of the judges. Broader community participation was a future plan.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A one-line change can trigger thousands of dollars of work
The initial investment was around 500 hours to develop the ten scripts, define the rubrics, find examples, and work out how to quantify writing quality. Bouchard then spent four months iterating with four high-usage subscriptions, experimenting with the benchmark, metrics, and their relative weights. The subscription and experimentation costs reached thousands of dollars, separate from the hours spent deciding what good writing meant.
Once the evaluation spans many models and drafts, two apparently small edits have different consequences:
- Change a judging rubric: Re-evaluating the generations under the changed scoring instructions costs more than $2,000 in the setup described.
- Change a generation prompt: Even a one-line change to the model’s input costs more than $4,000 to rerun the affected benchmark work.
The distinction follows the pipeline. A rubric change alters how existing writing is judged. A generation-prompt change alters the writing being tested, so the downstream evaluation must deal with new outputs. The reported costs make careful prompt and rubric design an operational concern: a short edit is not necessarily a small experiment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the model that is least annoying to finish
After four months of this work, Bouchard’s model-selection criterion is refreshingly practical: find the model that will be “the least annoying to you” on the way to the result you want. A spectacular one-shot post is a poor target if it is not repeatable or still leaves the wrong kind of editing work. The intended use of the benchmark is to find a productive starting point.
Cost and effort settings matter alongside quality. In the recorded benchmark comparison, Bouchard identifies GLM 5.3 Flash as useful for writing at scale and approximately 50 times cheaper than Opus 5.5 Extra High, the best-performing configuration he names. Extra High, in turn, was approximately 2.5 times cheaper than the same model’s Max Thinking configuration, with similar writing results. These are the talk’s task-specific comparisons, not current pricing or universal performance guarantees. They support a useful decision: maximum reasoning expenditure does not automatically buy a better script.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Supply examples, limit review, and keep a human editor
The first writing recommendation is to make the prompt much more detailed than a short style instruction. Include positive examples showing what to do, negative examples showing what sounds wrong, unwanted words, and substantial samples of the intended writer’s work—even a full newsletter. In Bouchard’s experience, more of that relevant material helped. The mechanism matches the benchmark’s task design: give the model observable examples of the desired choices rather than expecting it to infer a personal voice from a label.
Next, use another strong reasoning model for a review, with the same instructions and directions, and send its feedback to the writer. Bouchard generally recommends one pass; he mentions one to three reviews while warning that continued review can “loop into slop.” This is a practice-based observation rather than a measured gain—he explicitly says he lacks quantitative metrics for it. More revision is not assumed to mean better writing.
For an API-based workflow at scale, he proposes spending cheaply on variation and more selectively on judgment: generate about ten drafts of the same assignment with a cheaper model, then ask a stronger reasoning model to select and merge the best parts. This uses the variability exposed earlier in the benchmark as raw material. A model need not produce the best complete script every time for some of its drafts to contain useful passages. Bouchard believes this can cost much less than one expensive generation and produce better results, but presents it as an observation from use rather than a controlled comparison.
A human review still finishes the work. ToneBench measures one person’s voice and supplies one data point; a leading model on this benchmark is not therefore the best writer for everyone. The closing recommendation is to build an evaluation around your own needs, using the published ToneBench leaderboard and methodology as a starting signal rather than a verdict. The goal is a draft you can turn into your writing with less friction—not permission to stop reading it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The writing benchmark introduced in the talk, with model comparisons, graphs, and methodology for investigating writing quality alongside cost.
Further reading
- Anti-Slop AI Writing GuideReference
A practical companion in Bouchard’s cheatsheet collection, offering before-and-after examples, a writing prompt template, separate review and rewrite prompts, and a final checking routine.
Related talks
- The Benchmarks Game: Why It's Rigged and How You Can (Really) Win
Extends the warning against treating a leaderboard as a universal answer, examining benchmark incentives and evaluations tied to actual deployment tasks.
- The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI
Explores the parallel problem of evaluating automated reviewers and codifying human quality standards when generation grows faster than human review capacity.
Read the complete timestamped transcript
- 0:13
So I'm here to talk about, uh, all these words that we call slop, and, uh, more importantly, I want to, uh, try to help fixing what Kirpatim and, uh, termed as the, the slopocalypse that he thought would start in, uh, 2026. And I think we've seen that on social media. So, uh, to try to fix this and to stop slop, we first need to define it.
- 0:43
So first, what is slop? There are tons of different definitions from the various experts in the field, but what I think we can all agree is that it really sucks to read something that feels generated. And it usually looks like this with, uh, the, the words that we know of, the delve and, uh, the generalities and other things like these. And the main problem is that even as models get better with Fable, uh, GPT-6.1 and all the, the better models,
- 1:14
slop doesn't seem to go, uh, go away. It just keeps on scaling. So we need something different than just training the models more it seems. And, uh, I really want to try to at least first define what slop is so that we can understand why it's there. And so to me, slop is made of six main thing. First, there are tons of overused words by language models that we don't really use, like delve, that we really know well. Second, it's usually
- 1:44
too general. It's always talking about most themes, most projects. It doesn't share nothing specific. And I'm myself guilty of using that on LinkedIn as well, which I guess it's the LinkedIn meta so it's, uh, it's normal there. And worse slop, uh, sloppy text usually says nothing. It's just like nice words put together, uh, that we don't even usually use. And even worse than that, it says nothing, but it also repeats itself a lot.
- 2:14
So it repeats nothing and it's too general. And, uh, even worse, it hallucinates. As we know, models hallucinate, so the sloppy text also, uh, we cannot really fully trust it. And lastly, the worst to me is that it usually has the wrong tone or at least it has no tone and like everyone's tone at the same time. So it just doesn't really talk like you. It doesn't share stuff that you would share. So my goal here today and with this
- 2:44
project that I've been working on is to first measure this sloppy writing. Uh, not just slop words. I don't want to rebuild EQ-Bench Slop Score. It's just, it's really about measuring how sloppy is a writing. So in order to do that we are, well, today, right now, releasing a new benchmark that we called ToneBench which, uh, the whole goal is, of the benchmark is to measure if the new models or models are good at creative writing,
- 3:14
which I will try to define as well. So here's the plan for today. Uh, the plan is to justify do we even need another benchmark? Then how it works, what we measure, what, how we built it. Uh, and I want to share like the cost of building it, uh, our decisions, what we learned already from, f- for writing with AI from the benchmark, and, uh, share some more stuff about this. So with that, uh, let's delve in. So the
- 3:43
first, first thing, do we even need another benchmark again? And just looking at LinkedIn and X, X replies, my replies on X, I think we do. It's like people are spamming more and more generated content, so at least have them be decently good. And, uh, more precisely, I also really need it. We are a small team, uh, at Towards AI, my company, and we have one side of the company that is focused on
- 4:13
education. We build tons of educational content mostly for AI engineering. And as a small team, obviously we use AI as well for that, and it's been extremely annoying to review the slop, uh, constantly and try to fix it. We, uh, I published hundreds of videos. We published thousands of articles. We released a book. We are working on a second one, uh, for AI engineering. And so we really need to have the best models possible that will help us write better, uh, and
- 4:43
not be full of slop. And another nice, uh, addition to that is that I've been sharing about this benchmark on, on X for quite some time as I've been building it and it got really nice engagement and people seem to confirm that we did need some sort of more creative, uh, writing benchmark like this one. And lastly, just to prove that I think we are the right team to work on
- 5:13
this benchmark, we-- Well, I've been build- evaluating AI models since my master's and then PhD and now at Towards AI we do build evaluation pipelines for our clients. We have 15 AI engineers with us and we have creative writers, editors, uh, on board. So, uh, we already have that for our content. So I think, uh, well, I wanted to leverage that to build a benchmark and help the community even more. So what did we build
- 5:43
exactly? Hopefully I will try to share as much as possible and I'm happy to chat, uh, if we can help, uh, even build your own or if I can just share some, some of our learnings. So what we built is super subjective. Uh, it's a way to measure if a model can write like me. And how I define that is the, is just that if I can read it out loud like right now without cringing because of the- The, the, the words that I wouldn't use myself. So
- 6:14
the process is very simple. The, the benchma-- the process benchmark... The benchmark process is very simple. We have a, a prompt that is the brief and the research that we send to a model, and hopefully it will generate entirely a video script that I could have written. Uh, obviously, that doesn't happen. It really sucks usually. But that's the whole goal, and we measure that, that final script against our rubrics and, um, just our whole evaluation
- 6:44
pipeline. And the prompt that we send to every model in the benchmark is much more, uh, in-depth than that. It, it looks more like this, where we have a lot of details on how I talk, how I write, all, uh, tons of examples of my previous content, videos. We have a lot of information on storytelling in general, active sentences, and just how to write well. And we have directions on what to generate and what to do for this current
- 7:14
task, which in the end, the model will generate one script. And this is just one of them. In the current benchmark, we have ten different scripts, uh, that we worked really hard to make as good as possible, and we built rubrics for them to help, um, as we will see, LLM judges, uh, rate these, these generations. So we have ten different scripts. Some of them are published, some are not. We don't say which ones are the benchmarks, so that we check for contamination as well.
- 7:44
And, uh, then what do we do with these ten scripts? We take all the models out there, and we try to measure their generations. Uh, what that means is that for every single model, we will do ten times this process, generate the article from the prompt. In fact, we generate five of them. I will very soon share why. Then we have four different LLM judges analyze them. I recently added Jeff as a test. Uh,
- 8:14
not sure if I will keep that, but, yeah, I, I, I added it just to see if it worked well. And then we calculate in the end a final score for the leaderboard. I, I will go through all of these steps, uh, from left to right. So first, why do we generate even five scripts? It's f- to check for intra-model variability, which I highly suggest if you are building your own evaluations because especially I guess on a creative task, sometimes, as we see here with Grok 4.7, some of the
- 8:44
generations are outliers, where it's, as we see for- from the benchmark, it's just as good as Opus 5.5, but on just one generation out of five. So we want to measure for that. And the worse the model is, the widest the variance. So, uh, we have some learnings about that that I'll share at, at the end as well. But yeah, it's, uh, we, we found that five drafts seems to be the good middle ground in terms of, of cost for us to execute and, uh, to measure this, this variability.
- 9:14
So what do we measure exactly, and how do you even quantify writing quality? That's been the challenge, and we worked on it for, for quite some time. Right now, uh, it's all open on the benchmark, and there are way more details on the benchmark page as well. But what we measure is nine different metrics that we detailed, uh, that, that we really detailed. But I can condense them in five main aspects. First, we measure the tone. So does it sound
- 9:44
like me? Which means that, like, can the model adapt to a specific tone? Second, we measure just pure is it good writing? Uh, does it have, like, good hooks, good emotion, uh, emotional transition, and all the, the aspects that we're looking in a, in a good story? Then we measure if it follows instructions. Did it follow the, the length, the, the expected length, the format expected? Uh, did it put the right images and right, right things in the right place?
- 10:15
Fourth, we measure for research accuracy. So, uh, we have a research brief along with each script, and we just measure if it hallucinated or used the research correctly. And lastly, we measure for slop using EQ-Bench Slop Score, but also our own measurements. And what, uh, this looks in terms of what the judges see is more like that. We have around a thousand words of examples and,
- 10:46
a- and instructions and directions around each of the metric that we have, along with the script to analyze, my own, uh, script a- as some sort of, of guidance, and the brief and the research. So all in all, it's a fourteen thousand prompt that we send to, to every judge to then get, uh, the feedback that we'll dive in next. And, uh, so, so the judge just analyzes the writing quality. It doesn't focus on length, uh, and doesn't know
- 11:16
which model is writing. Uh, we just do that after with, with code itself. On one metric, like tone and voice, which is our main metric, it looks a bit like this, where we have a lot of detailed descriptions of what is my tone and what to expect and how to analyze this. Then the judge will categorize between six categories first, and after having found a category, it will give a rating. Uh, we
- 11:46
found that this is the best way to measure for this more qualitative task from, uh, just binary that we usually suggest to use or, uh, directly categories or directly percentage. We found that this actually works really well. And, uh, it has tons of examples of what each score means in terms of, of writing for each of the tasks. And what the judge send back is something like this. It has the nine
- 12:16
different metrics scored with a justification for a- for the score and an example of, like, uh, I, I wrote this score because of this example in particular. But the main thing is that all of this is automated, uh, obviously because we, with 120 models, we, we generate five drafts each for 10 tasks. It quickly goes to the thousands of scripts to review, so we cannot really review this ourselves. And the problem with LLM
- 12:46
judges is that they have preferences. Uh, some models prefer themselves. Most models prefer themselves, and, uh, particularly OpenAI. And what I found quite funny when studying this is that it seems like Anthropic and OpenAI are really in a, in a war here where they seem to have post-training on hating each other which is quite funny. But all this to say that you really want to check for-- When you ha- when you use LLM judges, like
- 13:16
using GPT for instance, if you want to know which model is best for your task, uh, be careful and use different model family just to ensure because we see here that OpenAI really has a strong preference on its own family of models. Not just one model, but all OpenAI models.
- 13:34
And so to fix also for that, we listened to a few comments that I got on X suggesting to use human for a creative task, which would be ideal, but quite impractical. We still tried to, to do that in a scalable way where right now we compared manually, me and three of my teammates, one thousand AB pairs of, uh, the generated scripts. Not the full scripts, but like a good chunk that is comparable between
- 14:04
two scripts, two generations. And hopefully we'll do more and I want to publicize this also for, uh, to get help from the community in the future, but that's not, uh, right now. So what did it cost us and what's the workload behind that? For me, for us, it was a kickoff cost in terms of hours of around five hundred hours to build the 10 scripts, uh, define all the rubrics, find the examples, uh, try to quantify quality which
- 14:33
is very difficult. And then I just iterated myself with, uh, four different 20X subscription for four months to build a benchmark and all the other things and experiment and change the, the met- the, the metrics, uh, ratio and all these things. And so we just spent, uh, thousands of dollars there. And, uh, if you want to do this, a, a similar type of benchmark, uh, you need to know that it costs quite a lot. Uh, right now if I just
- 15:03
change one thing in the rubric, so how the judge rate, um, generations, it will cost us two th- more than two thousand dollars and if I just change one line in an input prompt for a g- uh, a model when it generates, it costs us more than four thousand dollars. So it's very expe- uh, very expensive to experiment and change these things in the prompt, so you want to do this, uh, carefully.
- 15:29
Now what can we get out of this benchmark, out of ToneBench? Uh, you can get tons of things on the, the website itself. There's like a, a lot of details. But importantly, uh, I want to share that what I've learnt from the past four months is that you don't want to pick for, for creative writing, is that you don't want to pick the model that will generate a one-shot amazing post. You just want to find the model that will be the least annoying to you to get to where you want to be. So, uh,
- 15:58
like there won't be any perfection, uh, anytime soon I think. And what that means right now on the benchmark, it means at scale, uh, GLM 5.3 Flash is extremely good, extremely cheap. It's 50 times cheaper than the best one which is Opus 5.5 Extra High which is, uh, 2.5 times cheaper than Max Thinking, but with same results. So you don't want to use Max for writing usually. Um, and otherwise for the best, uh,
- 16:29
how to write with models that we got out of this is to first have an extremely long and detailed prompt. It cannot be long enough and the more the better. You want to add every example you can think of, of what to do and how to do it. Same thing for negative examples of things you don't like and that sou- doesn't sound right, the words you don't want, everything. You, you can share every-- If you write newsletters or whatever you can share it in full and it will just be better as well.
- 17:01
And, uh, what we've seen that really, really helps is to use a high-thinking judge just one time, one to three times otherwise it will loop into slop. So just one review of another model with the same instructions, directions that will give feedback to the writer. Uh, really like I don't have the quantitative metrics on that but it, it helps a lot. And for a company if you work with APIs and want to do this at s- have the best results at scale possible, what I
- 17:31
suggest is to generate several drafts like 10 drafts of one specific thing with a cheaper model like GLM 5.3 Flash and then have a smarter high-thinking judge like Opus or, or whatever you want pick and merge the best parts of each of these scri- these drafts which I think, uh, will still land way, way less expensive than just one generation by Opus 5.5 and, uh, I think will lead to better
- 18:01
results from what I've seen.
- 18:04
And lastly you want a human review. You cannot really go, uh, around that. But all of this benchmark measures obviously my voice so it's super subjective but I-- and, and it's just one data point. You don't want to fully trust it blindly. It doesn't mean Opus 5.5 is the best model. So but ideally it's a strong signal to find which model is good at writing and which is less sloppy. Um, y- I suggest you to build your own evaluation pipelines. If you want we can help. Uh, you can
- 18:34
chat or reach out. And regarding the benchmark it's live. We just like right now published it and released it. So, uh, there are tons of, uh, graphs, tons of insights, tons of comparisons, uh, and details on the implementation, full methodology. So just, uh, check it out and hopefully it's useful. Thank you.