When a completed AI scan
isn’t a complete answer.

A monitoring report can tell you that a scan finished without telling you whether the AI finished its answer. Those are different facts. In our own history, the distinction was material enough to affect which observations we should use as research evidence.

We reviewed the records behind 107 saved AIBL scans. Twelve question records were mocks and were excluded from the empirical counts. Of the remaining 1,613 records, 1,593 were marked completed and 20 failed.

Within the completed group, 253 records—15.9%—were explicitly marked truncated. They contained returned text, but generation had reached a configured token limit.

01What “completed” actually established

In these records, completion established that our processing flow had accepted an answer. It did not establish that the provider ended generation naturally. Our collector could retain returned text, evaluate it and mark the question completed while also recording a truncation flag.

Provider metadata explains the distinction. Anthropic documents max_tokens as reaching the request’s generation limit. Gemini documents MAX_TOKENS similarly. A returned response can therefore be a successful API exchange and still stop before a natural endpoint. These limits are part of the collection conditions we chose. Anthropic documentation; Gemini documentation.

02The counts, with the denominator attached

The 253 flagged records comprise 183 Claude and 70 Gemini executions. Those counts describe this historical workload under its recorded settings. They are not a fair provider comparison: model families, token ceilings, prompts, dates and collection behavior differ.

We also cannot call the other 1,340 completed records “verified complete.” A false flag can reflect a collector default when finish information is unavailable, and 203 completed records have no truncation flag at all. Absence of a recorded problem is not independent confirmation of completeness.

03Why this changes how we read visibility

Imagine an answer was going to list several businesses but stopped after the first two. A business missing from the retained fragment might have appeared in the part that was never generated. The fragment does not establish a whole-answer omission.

The reverse also needs care. A visible business name establishes that the name appeared in the retained text. It does not automatically establish a final recommendation: a qualification or warning might have followed.

These are illustrations of possible measurement errors, not observed outcomes we counted in this audit. We have not established how many brand scores or recommendations would change after a full human re-review.

A truncated answer can still support a narrower observation—for example, that a particular sentence appeared in its visible text. The admissible claim needs to match the evidence that actually exists.

04Four questions a completion label cannot answer

1. Did collection finish? Record success, failure and any retries.
2. Did generation finish naturally? Preserve stop reasons, limits and explicit uncertainty.
3. Is the retained answer usable for this metric? Whole-answer omission, rank and individual factual claims need different eligibility rules.
4. Can a reviewer inspect its sources? Native citation or grounding metadata may be separate from plain text. Missing saved links do not prove that the provider supplied none.

Report these states separately. Keep partial answers and failures in the collection record, then disclose which records qualify for a particular outcome metric. Do not silently delete inconvenient rows or treat all incomplete evidence as a negative brand result.

05What we can change without buying another scan

The saved data already lets us identify explicitly truncated records, separate mocks and failures, and flag records whose completeness is unresolved. It also lets us qualify historical reports that rely on a completed-status count alone.

Those are review and reporting steps, not a reason to regenerate everything. Retrospective gaps should remain visible. A fresh answer would be a new observation; it would not recover the missing portion of an old one.

Our guide revisions now make those evidence boundaries explicit. This audit does not claim that every historical score has been recalculated, that collector behavior has been repaired, or that a new collection method has been validated.

06How we conducted this audit

We used a read-only export of existing scan-question records, joined each row’s stored status with its discovery metadata, excluded mock records, and counted explicit truncation flags. No new paid scans or AI-provider calls were made. The test and demo schedules remained disabled.

A retained row is not necessarily one HTTP attempt or one generation. AIBL collection includes retries and separate evaluation work. The counts here concern retained question records, not provider request failure rates or independent statistical trials.

The history is an operational convenience sample, not a randomized study of buyer questions. Model and method changes prevent a clean universal trend. We did not assess every answer’s factual accuracy or measure business harm.

Native provider payloads were retained selectively: all 253 retained native payloads in this export belong to the flagged-truncated group. That subset cannot support an unbiased corpus-wide citation-quality estimate. Missing historical sources cannot be recreated by searching for them today.

Our arithmetic is reproducible from the accompanying metadata manifest. NIST’s distinction between fixed-benchmark results and generalized performance is relevant here: this audit describes the records we inspected, not all possible AI answers. NIST AI 800-3.