The short answer
Evaluate an AI visibility tool by giving every candidate the same small, frozen measurement contract and tracing each headline result back to the original question, completed provider answer, timestamp, citations, evaluation rule, and denominator. Recalculate a few results yourself, inspect failures and exports, and test whether the report supports a real decision. A polished dashboard or a higher proprietary score is not proof that a tool measured the same thing—or measured it reliably.
Editorial method · evidence boundaries
This guide proposes a review method. Worked examples and pilot sizes are illustrations unless a dated study is explicitly identified. Provider documentation supports only the behavior and product surface it describes; it does not prove that an AIBL workflow improves recommendations or business outcomes.
Apply this checklist to AIBL too: legacy API records do not uniformly retain native source metadata, and a completed status does not establish answer usability. The checklist is a proposed evaluation method, not a claim that AIBL has passed every item.
Historical AIBL records have limitations: completed jobs can contain truncated answers, and native citation metadata was not retained uniformly. Verify answer usability and source availability before scoring or comparing results. Missing saved metadata is not proof that a provider supplied no sources.
What businesses notice
Common signs of the problem
- Two vendors report very different visibility scores, but neither exposes enough evidence to reconcile them.
- A demo shows attractive charts while the exact questions, provider answers, failures, and scoring rules remain hidden.
- The tool counts a brand mention as a recommendation or turns a failed provider run into an omission.
- A team learns after purchase that it cannot export the answer-level history needed for client, audit, or renewal review.
01
Define the buying decision before the feature list
Tool evaluation should begin with the decision the measurement must support. A long feature checklist can reward surface area while missing whether the evidence is valid, inspectable, and useful to the people who must act on it.
Name the operating decision
Write the recurring decision in one sentence: identify high-value recommendation gaps, verify brand facts, compare providers, report to clients, monitor selected markets, or test change after an intervention. Different decisions require different question sets, evidence fields, review depth, cadence, and collaboration features.
Define the measured construct
Ask each vendor what “visibility” means in its product. Any mention, shortlist inclusion, first recommendation, rank-weighted presence, citation coverage, sentiment, accuracy, and estimated traffic are different constructs. NIST emphasizes construct validity: a procedure should measure the concept it claims to measure, within a stated context.
Read NIST on AI measurement validity ↗Separate must-have evidence from convenience
Treat the exact prompt, provider response, provider and model context when available, timestamp, citations, completion state, evaluation output, and denominator as evidence requirements. Treat alerts, summaries, presentation polish, suggested actions, and integrations as workflow features. Convenience cannot repair a missing evidence chain.
Set a bounded claim standard
Require every demo claim to match its support. “Visible in 8 of 12 completed test cells” is reviewable; “dominates AI search” usually reaches beyond that sample. The FTC says advertising claims need a reasonable basis and that the advertiser must possess at least the level of evidence it says it has. Apply that same discipline to vendor performance and coverage claims.
Review FTC advertising substantiation guidance ↗02
Freeze a proof-of-method test every vendor must run
A comparable trial uses the same eligible brand, question wording, provider set, market, time window, repeat count, and outcome definitions. Do not let each vendor choose the examples most favorable to its method.
Choose six materially different questions
Use a small but deliberate mix: two unaided category or problem questions, two use-case or comparison questions, and two branded fact checks. Each row should state the buyer, market, eligible brands, why the question matters, and the outcome to review. Six questions are an illustrative trial shape, not a universal minimum.
Build a coverage-led question set →Use the same providers and context
Specify the consumer or product surface, provider, market, language, search or grounding state when observable, fresh-conversation rule, and collection window. A ChatGPT result collected with web search is not automatically comparable to an API call, a cached answer, a different model surface, or a result localized to another market.
Repeat without hiding variability
Run each question-provider cell twice in a fresh context. Six questions across two providers and two repeats create 24 planned answer attempts: 6 × 2 × 2. Report completed, failed, blocked, excluded, and duplicate attempts separately. The trial tests the vendor’s collection and evidence discipline; it does not estimate the whole market.
Predeclare the grading rules
Before results arrive, define mention, eligible recommendation, first choice, ordered rank, competitor, owned citation, material factual claim, and unusable response. OpenAI’s evaluation guidance recommends task-specific objectives, representative cases, explicit metrics, complete logging, and human judgment; use those principles without implying that an API eval predicts every consumer product answer.
Review OpenAI evaluation best practices ↗03
Verify the collection layer before trusting the analysis
If the stored answer is wrong, incomplete, stale, or detached from the tested context, every downstream score can be internally consistent and still describe the wrong observation.
Open every record in the trial
For all 24 planned attempts, confirm that the record contains the frozen question, full returned answer, provider and product surface, timestamp, market and context fields, citations or source links when returned, collection status, and any retry or exclusion reason. A screenshot alone is not a durable evidence record.
Spot-check provider output independently
Manually run a few cells close to the vendor collection window using the closest reproducible public surface and the same visible settings. Expect wording variation; the purpose is to catch wrong prompts, mislabeled providers, truncated answers, stale demonstrations, missing citations, or a collection method materially different from what the sales claim implied.
Interrogate failures and retries
Ask what happens after a timeout, refusal, blocked response, empty answer, rate limit, or parser failure. Verify that the original attempt and retry are distinguishable, that silent retries do not manufacture a preferred result, and that failed or unusable responses are not scored as brand omissions.
Confirm the unit of observation
Determine whether one row represents an answer, a prompt-provider pair, a daily aggregate, a selected “best” result, or an average across unseen attempts. The evidence view, export, and dashboard denominator should use compatible units. If they do not, require a written reconciliation.
04
Audit the evaluation layer with a blinded recode
The same raw answer can yield different scores because tools classify mentions, recommendations, competitors, rank, citations, and facts differently. Recode a sample without looking at the vendor label first.
Distinguish mention from recommendation
A brand can appear in a caveat, source title, historical example, excluded-options list, or neutral comparison without being recommended. Require the rubric to preserve absent, passing mention, considered option, qualified recommendation, primary recommendation, and caution or rejection when those distinctions affect the buying decision.
Test entity resolution
Include an alias, product name, similarly named organization, or parent-subsidiary relationship only when it is realistic and safe. Check whether the tool maps the observed text to the intended entity and exposes ambiguity. False matching can inflate visibility while making the raw answer look superficially correct.
Inspect rank and list assumptions
Record rank only when the answer returns an evaluable order. Bullets can be unordered, categories can create separate lists, and prose can recommend one option while naming another first. The tool should retain “unordered” or “not applicable” instead of forcing every multi-brand answer into a numeric rank.
Recode citations and factual claims separately
A returned link is not automatically an owned citation, support for the nearby claim, or evidence that the brand was recommended. Verify source identity, claim alignment, and material facts independently. Keep citation presence, recommendation role, and factual accuracy as separate observations.
05
Reconcile the score instead of comparing labels
Two tools can both calculate their published formulas correctly and still produce incomparable totals. Convert the trial into a shared worksheet before deciding that one score is more accurate.
Build one row per planned cell
Use columns for question ID and version, buyer intent, provider, market, repeat, attempt state, completion state, brand mention, recommendation role, order, competitors, citations, material fact status, raw-evidence link, and reviewer notes. Add each vendor’s output as separate columns rather than merging disagreements away.
Recalculate simple observable rates
From completed eligible answers, calculate mention coverage, recommendation coverage, and first-choice coverage with raw counts beside percentages. Report collection coverage from all planned attempts separately. Do not try to reproduce a proprietary composite until the primitive observations agree.
Explain every disagreement
Classify each mismatch as collection difference, response variability, entity mapping, rubric interpretation, formula or weight, denominator, excluded cell, provider or market mismatch, or unresolved. The disagreement log is more useful than asking which headline number “feels right.”
Refuse false equivalence
NIST AI 800-3 distinguishes results conditioned on a fixed benchmark from estimates intended to generalize beyond it and notes that evaluation metrics depend on assumptions and uncertainty. A 70 from one question-provider sample is not directly better than a 55 from another. Compare like-for-like primitive observations first.
Read NIST AI 800-3 ↗06
Inspect evidence portability and reporting behavior
A tool becomes part of the organization’s measurement record. Test whether the evidence can survive a personnel change, client handoff, contract end, methodology update, or challenge to a published conclusion.
Export the answer-level evidence
Request an export during the trial, not after signature. Verify stable question IDs, prompt versions, provider and market fields, timestamps, full answers, citations, completion and exclusion states, structured evaluations, and links or identifiers that reconnect summary rows to evidence. Note documented format or retention limits.
Test history and methodology versioning
Change one sandbox prompt or rubric only after the frozen trial. Confirm that the tool preserves the prior version, marks the break, and does not present unlike samples as one uninterrupted trend. A historical chart without a method-change ledger can turn configuration drift into apparent brand movement.
Trace a chart back to its denominator
Choose one dashboard percentage, one competitor comparison, and one alert. Reconstruct each from exported rows. The report should reveal which completed observations contributed, which failed or were excluded, and how weights or grouping operated. If the vendor cannot reproduce its own chart, treat the result as unverified.
Document ownership and review controls
Confirm who can edit prompts, rubrics, exclusions, brand aliases, competitors, and reports; whether changes are logged; and how reviewer disagreements are handled. Also review the vendor’s current contractual statements about data use, retention, deletion, security, subprocessors, and support with the appropriate internal owners rather than inferring them from marketing copy.
07
Use pass, caution, and stop rules
Predeclared decision rules keep the trial from becoming a negotiation over every surprising result. Weight the failures according to the decisions and risks the tool will support.
Pass the method, not the demo
A candidate passes when the agreed cells can be traced to complete evidence, the observable fields survive blinded review, failures remain visible, simple rates can be reproduced, exports are usable, methodology changes are versioned, and the resulting findings change or support a real operating decision.
Use caution for explainable limitations
A tool may still be useful when provider coverage is narrower, some fields need human review, citation capture varies by surface, or exports require cleanup—if those limits are disclosed, consistent with the use case, and priced accordingly. Record the workaround, owner, and residual risk.
Stop for evidence-breaking behavior
Stop or narrow the purchase when raw answers are unavailable, provider or method claims cannot be substantiated, failed runs become omissions, prompts are silently rewritten, unlike samples are compared as trends, exports omit essential evidence, or material scoring disagreements cannot be traced to visible rules.
Run the same test at renewal
Repeat a stable subset before renewal and after material provider, collection, rubric, or product changes. Compare evidence quality, completion, analyst effort, disagreement rate, and decision value—not just feature count. NIST’s AI RMF Playbook recommends documenting test sets, metrics, tools, and measurement details to support repeatability and consistency.
Review the NIST AI RMF Measure playbook ↗08
Start with one independent evidence cell
The free AI visibility snapshot can produce one same-question comparison across ChatGPT and Gemini. It is not a complete vendor evaluation, but it gives the buying team a concrete answer record to use when testing what a platform stores, classifies, and explains.
Choose a real unaided buyer question
Use one category, problem, use-case, or local question where the tested brand is legitimately eligible without naming it in the prompt. Record the market and expected decision. Avoid a vanity query designed to force the brand into the answer.
Run a free AI visibility snapshot →Read both complete answers first
Before checking a score, inspect which brands appeared, how each was positioned, whether the lists were ordered, what rationale was offered, which sources appeared, and whether material claims were accurate. This establishes the human interpretation the vendor evaluation must preserve or explain.
Ask the candidate tool to explain the same fields
Compare the vendor’s evidence record and classifications with the snapshot and your manual notes while allowing for dated answer variation. The useful question is not whether the words match exactly; it is whether the tool makes its collection context, evaluation rules, and differences inspectable.
Expand only when the pilot informs the purchase
Add the remaining frozen questions and repeats when the extra cells will test a required provider, market, workflow, or failure mode. The conclusion should remain bounded: the trial evaluates the tool on the declared sample and decision, not every future answer or every vendor capability.
Continue the research
Related AI visibility guides
Primary sources
Primary documentation used for this guide
AI products, search behavior, and platform policies change. Check these maintained first-party sources before making technical decisions.
- Expanding the AI Evaluation Toolbox with Statistical Models ↗
National Institute of Standards and Technology
NIST AI 800-3 distinguishes fixed-benchmark performance from broader generalization, formalizes evaluation assumptions and measurement targets, and examines statistical validity and uncertainty. This guide applies those principles conservatively to tool due diligence; it does not claim a six-question trial estimates the wider market.
- Accelerating AI Innovation Through Measurement Science ↗
National Institute of Standards and Technology
NIST discussion of construct validity, generalization, uncertainty, model and benchmark comparison, and reporting detail. The page frames open measurement-science questions rather than certifying any AI visibility vendor or metric.
- AI RMF Playbook: Measure ↗
NIST AI Resource Center
Voluntary suggested practices for documenting test sets, metrics, tools, limitations, and measurement details to support repeatability and consistency. The proof-of-method workflow here is an editorial application, not a NIST procurement checklist.
- Evaluation best practices ↗
OpenAI Platform Documentation
First-party guidance on task-specific objectives, representative datasets, explicit metrics, logging, continuous evaluation, comparison, and human calibration. It addresses evaluation design and does not validate any public AI visibility score or vendor.
- Advertising FAQs: A Guide for Small Business ↗
Federal Trade Commission
FTC business guidance explaining that advertising claims require a reasonable basis and at least the level of evidence claimed. The guide uses that general substantiation principle for vendor due diligence and does not provide legal advice.
Measure your visibility
Turn the questions in this guide into an evidence-backed baseline.
Get AI Visibility ScoreTalk to us
Have a visibility question?
Tell us what your team is trying to measure or improve.
Contact AI Brand Lens →Keep learning
Explore every guide.
Browse practical answers about AI visibility, competitors, and measurement.
View all guides →