The short answer
Do not automatically count an incomplete AI answer as a normal positive or negative visibility observation. First preserve the collection result, provider stop metadata when available, and returned text. Then label the record as verified natural stop, explicitly partial, completeness unknown, failed, or excluded for another declared reason. A partial answer can support a narrow claim about text that actually appears, but it usually cannot support a whole-answer omission, final rank, complete recommendation set, or citation-coverage judgment. Report collection coverage and metric eligibility separately, and treat any retry or continuation as a new linked observation—not as a silent repair of the original record.
Editorial method · evidence boundaries
This guide proposes a review method. Worked examples and pilot sizes are illustrations unless a dated study is explicitly identified. Provider documentation supports only the behavior and product surface it describes; it does not prove that an AIBL workflow improves recommendations or business outcomes.
The empirical counts in this guide describe AIBL's retained API collection history from June 23 through October 22, 2026. They are an operational convenience sample under changing models, prompts, token limits, and collection methods—not a provider failure benchmark or an estimate of consumer AI behavior.
Historical AIBL records have limitations: completed jobs can contain truncated answers, and native citation metadata was not retained uniformly. Verify answer usability and source availability before scoring or comparing results. Missing saved metadata is not proof that a provider supplied no sources.
What businesses notice
Common signs of the problem
- A provider request returned text and the pipeline marked it completed, but the final sentence stops abruptly or the recorded stop reason shows a token limit.
- A missing brand is scored as an omission even though the retained answer may have ended before the provider finished listing options.
- A dashboard removes partial or failed rows without showing how the performance denominator changed.
- A retry replaces the original result, making the cleaner second answer look like the only observation that ever occurred.
01
Separate collection success from answer completeness
A successful API exchange, a completed processing job, and a naturally finished answer are three different states. Keep them separate before calculating any brand metric.
Record whether collection returned
Start with the attempt: planned, started, returned, failed, blocked, timed out, or canceled. A transport or processing failure is an operational outcome. It is not evidence that the brand was absent, poorly ranked, or unsupported in an answer that never became usable.
Preserve the provider stop signal
Anthropic documents end_turn as a natural completion and max_tokens as reaching the configured output limit; its guidance says to check stop_reason rather than equate an HTTP success with a finished response. Preserve the raw value, provider, model, request settings, timestamp, and collector interpretation instead of reducing them to one completed flag.
Review Anthropic stop reasons ↗Do not normalize unlike reasons
A token limit, safety stop, refusal, custom stop sequence, tool handoff, malformed function call, and natural stop do not mean the same thing. Keep the provider-native value and add a documented cross-provider category only when the mapping is defensible. Unknown must remain a valid state.
Inspect the retained answer too
Metadata can be absent, transformed, or insufficient for the metric. Check whether the answer ends mid-sentence, announces a list that never finishes, leaves a tool workflow unresolved, omits expected source objects, or contains only a fragment. Human review does not prove that an answer is complete, but it can identify additional reasons it is unusable.
02
Learn from the retained AIBL audit without overgeneralizing
The October 22 retrospective gives this problem a concrete denominator. It also shows why one operational dataset cannot become a universal provider-quality claim.
Attach every count to its population
AIBL reviewed 1,625 saved question records behind 107 scans. Twelve mock records were excluded, leaving 1,613 non-mock records: 1,593 marked completed and 20 marked failed. Those are retained question records, not necessarily independent provider requests or statistical trials.
Read the complete retrospective →Name the explicit truncation finding
Among the 1,593 non-mock records marked completed, 253—15.9%—carried an explicit truncation flag. Returned text existed, but generation had reached a configured limit. The result establishes a historical collection-quality issue in this dataset; it does not establish a current consumer-product truncation rate.
Do not call false or missing flags verified
The audit found 1,137 completed records with a false truncation flag and 203 completed records with no stored flag. A false collector value can be a default rather than independent proof of natural completion, and missing metadata cannot be reconstructed from confidence. Use “not explicitly flagged” and “unknown” where those are the facts.
Do not compare providers from a confounded workload
The 253 flagged records comprised 183 Claude and 70 Gemini executions, but the audit spans different model families, token ceilings, prompts, dates, and collection behavior. Those raw counts do not support a fair provider ranking. The public manifest exposes anonymous collection metadata so readers can examine the record-level basis without publishing prompts, answers, brands, or internal identifiers.
Inspect the public metadata manifest ↗03
Use a five-state completeness ledger
A single completed boolean hides the decision the evaluator actually needs to make. Add a separate completeness field with evidence and ownership.
Verified natural stop
Use this only when retained provider metadata supports a natural endpoint and the response passes any additional declared usability checks. Record which native value supports the label. Natural stopping does not make the answer accurate, well sourced, relevant, or complete for every buyer need.
Explicitly partial
Use this when metadata records a token or context limit, an unfinished tool or continuation state, or another documented condition that leaves the response incomplete for the intended observation. Preserve the fragment. Do not erase useful visible evidence merely because the whole answer is ineligible.
Completeness unknown
Use this when the system returned text but the evidence needed to determine a natural endpoint was not retained or is ambiguous. Unknown is not a softer version of complete. It should have its own count, review rule, and eligibility treatment.
Failed or excluded
Failed means the planned collection did not produce a reviewable answer. Excluded means a returned record violates a predeclared rule for another reason, such as a wrong prompt version, duplicated attempt, mock record, or corrupted payload. Keep the reason and do not merge these categories merely to simplify a chart.
04
Match metric eligibility to the evidence that exists
Completeness is not always all-or-nothing. The right question is whether this retained material can support the specific claim the report wants to make.
A visible text claim may survive
If a brand name or sentence appears in an explicitly partial response, the narrow statement “this text appeared in the retained fragment” can remain observable. Qualify the record as partial and avoid upgrading a passing mention into the answer’s final recommendation or position.
Whole-answer omission usually does not
A truncated list cannot establish that an unseen brand would never have appeared later. Exclude an explicitly partial answer from omission, complete candidate-set, final order, and share-of-answer metrics unless the rubric has a specific, defensible rule for the visible scope. Do not score ungenerated text as a negative result.
Citation and claim audits need their own gate
A visible source can be reviewed against a visible claim, but a partial answer cannot establish complete citation coverage for the answer that was never generated. Similarly, a factual statement in the fragment can be verified while the overall answer remains ineligible for completeness-dependent metrics.
Use the claim-to-source audit →Define the rule before viewing results
Write an eligibility matrix for mention, recommendation, order, omission, factual accuracy, source support, and candidate-set metrics. Predeclare how each completeness state is treated. A rule chosen after seeing whether the fragment favors the tracked brand invites selective inclusion.
05
Report attempts, usable evidence, and performance separately
A defensible report needs more than one denominator. Collection reliability and brand performance answer different questions and should not be allowed to repair each other.
Show the attempt ledger
For each question × provider × market × repeat cell, report the planned attempt and its returned, failed, partial, unknown, excluded, and metric-eligible states. If one API workflow produces multiple calls or continuations, retain the relationship rather than pretending each row is an independent buyer observation.
Use metric-specific denominators
A fictional 12-cell pilot might yield eight verified natural stops, one explicitly partial answer, two completeness-unknown answers, and one failure. Mention-in-fragment reporting could include the visible partial record with a qualifier, while whole-answer omission might use only the eight verified cells. The illustration is a reporting method, not an industry threshold.
Publish exclusions beside the score
Say “recommended in 5 of 8 eligible complete answers; 12 planned, one partial, two unknown, one failed,” not merely “62.5% recommendation visibility.” The longer statement exposes both the observed performance and the evidence loss behind it.
Bound the conclusion to tested items
NIST distinguishes performance on a fixed benchmark from claims about a broader item population and discusses the assumptions required when data are missing or imbalanced. This guide does not prescribe a statistical model; it adopts the narrower lesson that the target quantity, observed data, exclusions, and assumptions must remain connected.
Read NIST AI 800-3 ↗06
Treat retries and continuations as new evidence
Fixing the collection path may be useful, but it does not make the original incomplete observation disappear. Preserve both the history and the method change.
Link rather than overwrite
Store the original attempt, returned fragment, stop state, and settings. Give the continuation or retry a new observation ID linked to its predecessor, with the changed token ceiling, prompt, model, tool state, or collector version. This makes operational improvement auditable.
Do not splice a buyer answer silently
A continuation may depend on conversation context, while a fresh retry may produce different candidates or reasoning. If the measurement contract permits a continued answer, label the composite and define how it is scored. If it requires a single fresh answer, keep the continuation outside that metric.
Separate recovery rate from visibility
Track whether explicitly partial attempts can be recovered under a documented procedure, but do not blend recovery success into the original provider or brand performance rate. A collector that retries until it gets a favorable answer would bias the retained evidence.
Annotate method breaks in trends
Raising output limits, changing provider surfaces, retaining new metadata, or adopting a different retry policy can alter completion eligibility. Mark the date and compare like-for-like cells on each side instead of calling the resulting score movement a brand change.
Build a matched-period comparison →07
Put the completeness gate into routine review
The workflow should make the cautious decision easy: preserve the record, classify evidence, decide eligibility, and only then score the brand result.
Assign a quality owner
Name who reviews stop metadata, ambiguous fragments, exclusions, and retry relationships. Calibrate reviewers on a shared codebook and periodically reopen examples. Completeness review is part of measurement quality, not a cleanup task after the dashboard ships.
Make vendors demonstrate the gate
Ask an AI visibility tool to export provider stop metadata or its closest retained equivalent, raw returned text, retries, exclusion reasons, and metric denominators. If the vendor uses “completed” or “successful,” ask what that label proves and which answer-quality conditions it does not test.
Evaluate an AI visibility tool →Connect the gate to the score definition
Document whether the score uses returned answers, verified natural stops, a broader usability rule, or another explicit population. Update the score guide, methodology, client report, and change log together so stakeholders do not compare percentages built from different eligibility rules.
Interpret AI visibility scores →Start with one inspectable snapshot
Run one representative question through the free AI Visibility Snapshot and read the full ChatGPT and Gemini outputs before grading a mention, omission, recommendation, or source. Note any visibly unfinished text and preserve what the product actually shows. A snapshot is a useful two-provider baseline, not proof that every provider response is complete or that one question represents the market.
Run a free AI visibility snapshot →Continue the research
Related AI visibility guides
Primary sources
Primary documentation used for this guide
AI products, search behavior, and platform policies change. Check these maintained first-party sources before making technical decisions.
- When a completed AI scan isn’t a complete answer ↗
AI Brand Lens Research
The October 22, 2026 retrospective provides the 1,625-record audit, scope, methodology, limitations, and interpretation used here. The guide converts that evidence into a practical eligibility workflow without treating the historical workload as a provider benchmark.
- Completed-scans question metadata manifest ↗
AI Brand Lens Research
Anonymous record-level collection metadata behind the retrospective. It excludes business names, prompts, answer text, internal identifiers, and native payloads; a null flag means no flag was stored, and false does not independently verify completeness.
- Stop reasons and fallback ↗
Anthropic Claude Platform Documentation
First-party documentation distinguishing natural end_turn completion from max_tokens truncation, tool-use pauses, refusals, and other stop states, with guidance to inspect stop_reason and preserve incomplete-response handling.
- GenerateContent API: FinishReason ↗
Google AI for Developers
First-party Gemini API reference defining candidate finish reasons, including STOP as a natural endpoint and MAX_TOKENS when the request’s configured maximum is reached, alongside safety, recitation, language, and other states.
- Expanding the AI Evaluation Toolbox with Statistical Models ↗
National Institute of Standards and Technology
NIST AI 800-3 distinguishes results on tested benchmark items from broader generalized claims and discusses missing and imbalanced evaluation data. The completeness ledger and eligibility matrix are AI Brand Lens editorial methods, not a NIST prescription.
Measure your visibility
Turn the questions in this guide into an evidence-backed baseline.
Get AI Visibility ScoreTalk to us
Have a visibility question?
Tell us what your team is trying to measure or improve.
Contact AI Brand Lens →Keep learning
Explore every guide.
Browse practical answers about AI visibility, competitors, and measurement.
View all guides →