What AI Agents Get Wrong When Writing Docs

Every item is a real repository event

Each item starts from a pull request that changed the product, or from a documentation gap that someone reported. The projects include Helm, PostHog, Mautic, and Doc Detective. We built the benchmark with maintainers and technical writers from those communities, who reviewed items and wrote criteria.

The agent decides first, then writes

Each agent receives the repository as it existed before the change, plus a triggering signal, such as a code pull request or a reported documentation gap. It doesn’t see the docs that maintainers later merged. It then either writes a patch or abstains, leaving the docs alone. A patch can create new docs, update existing guidance, or delete stale or redundant content.

Agents get one attempt with no internet access. Each agent runs in a fresh, sealed container that can reach only its model provider, so it can’t find the merged answer online. In an earlier version of the benchmark, agents found the source pull request online in 31% of their attempts, even though we’d masked the repositories’ identities.

Sometimes the right answer is to leave the docs alone. Of the 292 items, 205 need a patch, and the other 87, the no-change items, need none. A routine refactor, a test change, or a dependency upgrade usually doesn’t affect readers. A good maintainer writes nothing in those cases, and neither should an agent. Earlier documentation benchmarks assume someone already decided an update is needed. We split the items at random into a held-out split for the main results and a public development split. Each split keeps the same mix of answer type, source, task type, and repository.

Every item that needs a patch has its own rubric

Across the 205 items that need a patch, the rubrics hold 3,273 criteria. An AI agent researches the triggering change and the repository, drafts each rubric, and then compares anonymous candidate patches to find what the draft misses. A second model, from a different model family, audits the result. Each criterion tests one decision a reviewer would make, such as whether the patch states a default correctly. Each also has a priority from P0, the most critical, to P3. The 798 P0 criteria would each block a patch in review on its own, and the P1 to P3 criteria are secondary criteria.

The rubrics check what the reader needs. They don’t reward matching the docs the maintainers merged, because many different edits can meet the same need.

The rubrics held up against human judgment. Maintainers reviewed 42 items. On 41 of them, they agreed with the benchmark about whether the docs needed to change. On 23 items that need a patch, maintainers validated 242 criteria of their own, and the generated rubrics covered 90.9% of them.

A separate scoring model, GPT-5.6 Terra, checks each patch against each criterion. Its verdicts matched human reviewers’ verdicts on 310 of 330 criteria (93.9%).

Strengths can’t fully make up for a weakness

Failing one P0 criterion caps a patch score at 60, so many small wins can’t average away a critical defect.

The composite score is the harmonic mean of delivered quality and abstention recall. Delivered quality is the average patch score on the 82 held-out items that need a patch. A missed patch, an item that gets no patch at all, scores zero. Abstention recall is the share of the 35 no-change items on which the agent leaves the docs alone. A harmonic mean stays low when either side is low.

The seven agents finished within 13.3 points of each other

The top composite score was 47.3 out of 100. Qwen3.8 Max with OpenCode led the seven agents in the paper, and GPT-5.5 with Codex placed last at 34.0. Each paired a frontier model with a frontier coding harness.

Figure 2.2 splits each composite score into delivered quality and abstention recall, and adds each agent’s P0-clean delivery.

Critical defects are common

A critical defect is a failed P0 criterion: an error or omission that would mislead readers or block the task. A patch is P0-clean when it has no critical defects. P0-clean delivery is the share of items that need a patch on which an agent delivered a P0-clean patch, so a missed patch counts against it.

We found that humans are better at judging what matters. Merged human patches were P0-clean 85.7% of the time, more than twice the best agent’s rate (Figure 2.3). We applied the same measure to the patches that maintainers merged. The best agent, GPT-5.6 Sol with Codex, delivered a P0-clean patch on 39.0% of the items, or 32 of 82.

Human patches aren’t perfect either. The paper’s appendix lists four merged human patches that fail a P0 criterion. One is a deploy command for Pants, the build tool in chapter 5, whose stray slash keeps it from resolving its target. Another is an OpenCost procedure that creates a Kubernetes Secret in one namespace and mounts it from another, so the procedure fails.

Agents make mistakes in both directions

No agent was strong at both deciding and writing. GPT-5.5 with Codex wrote a patch for 96.3% of the items that needed one, a rate called patch recall. But it left the docs alone on only 28.6% of no-change items. Claude Opus 4.8 with Claude Code made the opposite trade: it left the docs alone on 85.7% of no-change items, but wrote a patch for only 79.3% of the items that needed one.

Treat small gaps between agents as ties

On conditional quality, GPT-5.6 Sol with Codex scored highest, at 46.3. Conditional quality averages only the patches an agent actually produced, so unlike delivered quality, it ignores missed patches. The leaderboard labels it Patch quality.

Each mean has a 95% confidence interval, a range that shows how precisely the data pins the mean down. The interval for GPT-5.6 Sol with Codex overlaps the intervals of the next four agents. Each agent also ran each item once, so these results don’t show how much a rerun would vary.

Cloud agents, including Promptless, show the same weaknesses

We also ran three cloud agents on the same held-out split. A cloud agent is a commercial product that runs its own agent as a hosted service, with a model and harness its vendor chooses. They aren’t part of the paper’s ranking, so we report them as supplemental results. Two of the paper’s authors work at Promptless, which builds documentation agents.

Promptless delivered a P0-clean patch on 40.2% of the items that needed one, so most of those items still didn’t get a patch free of critical defects. It also wrote an unneeded patch on 37.1% of no-change items, 13 of 35. Expert review remains necessary.

Accurate text still leaves the reader stuck

The most common failure mode is the task-completion gap, and a patch can have one even when everything it says is accurate. In 45.5% of submissions, the patch left out a piece the reader needed, such as a decision point, a step, a check, or a recovery path. Fluent prose makes these easy to miss, because the patch reads well and most of it is true. Technical inaccuracy came second, at 36.6%. One patch can have several failure modes, so the rates overlap.

Behind these failure modes, agents have no model of how the product is actually used. That shows up in three patterns of behavior.

1. Agents don’t know when not to write

Agents are eager to help, but sometimes that creates more work. On average, the seven agents wrote an unneeded patch (over-editing) for 37.1% of the 35 held-out no-change items. The rate ran from 14.3% for the most cautious agent to 71.4% for the most eager.

An unneeded patch usually starts with a misleading cue, like an internal name that sounds user-facing, or an existing page that mentions the component. Once an agent decides to edit, it looks for a place to put text and rarely asks again whether docs are needed.

Agents document three kinds of change that readers don’t need:

• Internal refactors, renames, and regenerated types that don’t change the public contract, the behavior users rely on
• Performance, test, continuous integration (CI), and dependency changes with no effect a user can see
• Changes to generated files and reference artifacts that people shouldn’t edit by hand

Unneeded edits aren’t harmless. Each one still has to be read and rejected. If your team merges one, it publishes implementation details that users can’t act on, as if they were part of the product. Then the page has to be versioned, translated, and kept true, and the next agent run reads it as a pattern to follow.

2. Agents don’t question the status quo

Agents take what the docs already say, and what they leave out, as the rule. If an old section breaks the style guide, the agent treats it as precedent and keeps breaking it. If a feature went undocumented by mistake, the agent assumes the omission was deliberate. When a related code change arrives, it skips the update instead of filling the gap.

The same habit causes missed patches (under-editing). A common trap is treating missing coverage as proof that a topic is out of scope, when the gap is the problem to fix. Agents also read a change’s location in the code, or weak keyword overlap with the docs, as a sign that readers aren’t affected. They dismiss real public interfaces in parsers and configuration modules as implementation details.

From the agent’s perspective, deliberate exclusion and an unfilled gap produce the same search result. Documentation owners bring context that a code change alone may not reveal.

3. Agents jump to conclusions and focus too narrowly

Agents switch from investigating to drafting too soon, because they have no reliable rule for when they know enough. This is premature closure, and it runs through many of the root causes we found. In 33.1% of submissions, the agent found related material, stopped short of the decisive evidence, and filled in with a plausible guess. Decisive evidence is the code change or contract that settles the question. In 30.1%, the agent stopped at the first plausible page and missed the other pages the same change affected. The most common root cause, at 36.0%, was explaining an interface without checking how readers use it.

It isn’t about effort. The root causes showed no clear link to trajectory length, whether measured by turns or by token use. A token is a unit of text that a model reads and writes. Effort didn’t rise with task difficulty either, which we defined by how the other agents fared on an item. The correlations were 0.034 for tokens processed, 0.034 for trace events (the logged steps of a trajectory), and 0.030 for tool actions. Every confidence interval included zero. An agent can stop investigating early and still spend a lot of tokens drafting.

A stopping rule would tell an agent when it understands the request, the evidence, and the documentation impact well enough to start writing. SeekBench, a benchmark of search agents, finds the same problem. Agents trained with reinforcement learning answered before they had enough evidence in 76.5% of the trajectories that SeekBench evaluated.

Three likely causes didn’t explain the failures

We tested invention, training data, and project size.

Less invention, more misstated facts

Frontier agents don’t invent features as much. More often, they take something true in one case and present it as true in every case. Fabricated content, such as a flag or an endpoint that doesn’t exist, showed up in only 6.1% of submissions, against 36.6% for technical inaccuracy. In those inaccuracies, the feature is real, but the patch overstates its scope, default, or compatibility. Claims like this are as unfounded as hallucinations, and harder to spot.

Training data and project size showed no clear effect

Four of the agents use a model with a published knowledge cutoff, the date its training data ends. Each of those agents scored lower on items dated after its model’s cutoff, but every confidence interval included zero. Popular repositories gave agents no consistent advantage over less popular ones. Doubling the size of a project’s docs changed scores by −0.98 points, and its confidence interval also included zero.

DoGBench didn’t test these lessons. We tried prompts, skills, and explicit verification informally, without a controlled comparison, so measure each lesson on your own work. The rates in this chapter come from our audit of 1,267 submissions.

Lesson 1: Help the agent understand your company, your docs, and your workflows

Lesson 2: Ask the agent to do research, and ask for a report with predefined fields

Lesson 3: Check that the patch includes the why and when, not just the what and how

Check that every patch answers three questions:

1. Who is this for?
2. When would they choose it, and why?
3. What must they get right?

Lesson 4: Walk through the reader’s whole task

Lesson 5: Fact-check claims, especially their scope and applicability

The request spans several CI jobs

Pants is a build tool that can split a project’s Python tests across several continuous integration (CI) jobs. Each job, called a shard, runs only some of the tests.

Test coverage is the share of the code that the tests exercise, and a shard’s coverage report covers only that shard’s tests. The team wants to know whether all the tests together meet the project’s coverage threshold, its pass mark. The request asked for docs that tell readers to save the raw coverage file from every shard and combine the files after the shard jobs finish. Readers then check the threshold on the combined report.

The request sounds like “add a paragraph about coverage.” Before drafting, an agent has to settle one fact: Pants combines coverage within a single run, but CI shards are separate runs. The procedure also has to reach readers who start on either of two pages.

The highest-scoring patch covers both pages

The highest-scoring patch adds the full procedure to the Python test page, and a caveat on the sharding guide links to it. A reader who starts on either page reaches the procedure.

A warning without a procedure scored 25.0

Kimi K2.7 Code with OpenCode correctly warned that a shard’s coverage is partial, and then stopped. Its one note leaves readers with no working procedure. Its patch failed four P0 criteria and one P1 criterion.

Each failure traces back to a step in the trajectory

The agent never mapped how a CI user gets from partial shard results to one checked coverage total. Figures 5.5 and 5.6 trace each failure to the steps of the trajectory where it began, and name the lesson that could have caught it. Each step groups related actions, so the step numbers don’t count turns.

A better agent run understands the request first

GLM 5.2 with OpenCode scored 92.3 on the same item. It worked out what Pants does within one run, what separate CI jobs must do afterward, which facts come from the code, and which page a reader reaches first. The two agents received differently worded versions of the request, so this isn’t a controlled comparison.

A warning tells the reader that something can go wrong. A procedure tells them how to finish the job.

The reviewer’s checklist

Work through the stages in order, because a failed early check can make the later stages unnecessary. For example, when the product change affects nothing a user can invoke, see, or act on, the patch is unneeded, and you can stop there.

Questions for documentation and content operations leads

Who decides whether a change needs documentation?

Agents misjudge this in both directions. Keep the decision with a person, or review the decision before you review any prose.

Where do your conventions live?

Terminology, audience, and information architecture have to exist in writing before an agent can load them.

How will you evaluate an agent before you adopt it?

Run it on your own past changes, including changes that needed no docs, and review the results against your team’s standards.

What will you measure after adoption?

Track how often reviewers reject unneeded patches and how many critical defects they catch. Both numbers show whether your context and review steps are working.

Who reviews, and is their time in the plan?

Budget for expert review from the start, because it catches the critical defects that agents miss.

Run DoGBench on your own agent

The code for running and scoring DoGBench is on GitHub, and the 175-item public development split is on Hugging Face. Develop and debug on the development split. The leaderboard on dogbench.ai compares agents on the held-out split. For reproducibility, we’re publishing 735 complete agent trajectories from the seven agents in the paper, so you can study each agent’s steps.

Have results to share? Submit your agent’s DoGBench results for review. We review each entry by hand and will follow up about verification and evaluation requirements.

The paper, data, example items, and readout slides are at dogbench.ai. The slides are a short deck for presenting the findings to your team or leadership.