Most AI visibility software will happily charge you to watch yourself lose. The demo shows your brand missing from a ChatGPT answer, a competitor cited instead, and a chart that will look just as clean next month.

We ranked the market once in AI Visibility Tools Ranked by Whether They Do the Work. We are taking a second look because nearly every vendor now says it “optimizes.” The label no longer tells you whether the tool monitors, recommends, or executes. A trial can.

Run one fixed set of buyer prompts against treated pages and untouched controls for 30 days. Then ask the question the demo cannot answer: did your citations in AI Overviews and ChatGPT move because the tool did work, or did it only describe the problem?

Write the pass condition before day zero

Set the pass condition in writing before a vendor sees the account. I used to tell clients to judge SEO tools by report quality. I was wrong. A polished report with flat citations is a failed trial; a clumsy interface that helps ship cited pages deserves a closer look.

My bar is this: the treated set gains citations on at least 20% more prompts than the control set, with team hours logged per fix. Use comparable prompt groups for each set and count a citation only when the answer links to one of the pages being tested. Pick a different bar if you want, but pick it now. Otherwise, any uptick becomes a success story after the fact.

AI answers give a vendor plenty of room to tell that story. Brands show up in 90% of AI Mode responses versus 43% of AI Overviews, and answers change between checks. Your written threshold is the control on the sales conversation, not a promise that every rerun will look identical.

Day 0: freeze the prompts, pages, and baseline

Write 30 to 50 unbranded buyer questions before the vendor sees them. Cover the funnel rather than stacking the list with easy wins. A 40-prompt set, for example, could contain:

  1. 10 problem questions: “How to reduce chargebacks for subscriptions?”
  2. 15 comparison questions: “X vs Y for …?”
  3. 15 purchase questions: “Best … for … under $…?”

The leading tools in this tracker comparison run buyer prompts and report mentions. If the vendor writes the entire prompt list, it can steer the test toward questions it already handles well. Keep your list fixed for the outcome score. Put prompts the tool discovers later in a separate discovery log.

Record the baseline by hand. For each prompt, fix the country, device, and language; note the date and whether an AI Overview appeared; and distinguish a brand mention from a citation linking to your URL. Save a screenshot. Follow the same repeatable citation-testing method on later checks, with 5 to 10 live spot checks each week. If a surface produces no answer for a prompt, record that too rather than quietly dropping the question.

Now select 6 to 8 URLs the tool may change and 6 to 8 matched control URLs nobody touches. Record which prompts relate to each set, who may edit the treated pages, and how much time every person spends on a fix. Keep unrelated site changes out of both sets where you can; when you cannot, log them. If day zero is not written down, day 30 is a story, not a score.

Week 1: make prompt discovery prove itself

Hand the vendor your fixed prompt list, but not the baseline results. Ask it to return its own prompt discoveries before you reveal where your brand appears. You are testing whether it finds buyer language you missed, not whether it can restate your spreadsheet.

Different approaches are worth testing. Otterly’s Query Fan Out expands a seed prompt into related queries an engine might generate; HubSpot suggests prompts from CRM data (HubSpot vs Otterly breakdown). Count net-new prompts that trigger a brand mention for you or a competitor. Do not count 200 minor variations of a question nobody asks as 200 discoveries.

Then test precision yourself:

  1. Take 10 prompts the tool says trigger a mention.
  2. Run each in a clean session using your fixed country and language.
  3. Record whether the claimed mention appears.

Answers are non-deterministic. A SparkToro test of around 3,000 prompts found identical brand lists less than 1 in 100 times; Growthner consequently advises tracking visibility-percentage trends across 25 to 50 unbranded prompts for 30 or more days (Growthner comparison). My week-one rule is still blunt: if fewer than 7 of 10 claimed triggers reproduce, treat the discovery list as noise. Keep those discoveries separate from the fixed outcome set, or the vendor gets to move the goalposts mid-trial.

Weeks 1–2: demand a page-level diagnosis

A page can rank 1 to 3 in Google and still be skipped in an AI answer. A concise comparison table, fresher data, or third-party list may be easier to quote; key facts may also sit behind JavaScript, forms, or blocked bots (the ranking–citation gap). That gives you a concrete standard for the gap report.

“Add more authority” is a fortune cookie. “This pricing page lacks the comparison table the cited competitor has, and its specs render client-side” gives someone work to do. Score every treated URL on four yes-or-no checks:

  • Does the report name an exact missing block, such as a comparison table, use-case by role, pricing context, or limitation? Compare the proposed fix with this page-level checklist of answer-first summaries, tables, schema, and consistent details.
  • Does it name the source cited instead for the relevant prompt, with a URL and date?
  • Does it check retrievability, including bot access, rendered HTML, and Organization, Product, or FAQ schema?
  • Does it assign the fix to a specific page and owner rather than to “the site”?

A Content Audit with a predictive score and a Crawlability Checker can help identify work. But if the tool stops at recommendations, as the HubSpot vs Otterly teardown describes Otterly, execution still belongs to your writers and developers. Count that as a diagnosis pass, then move their hours onto your side of the ledger. If the tool cannot name the gap at page level, do not credit it with a page-level fix.

Weeks 2–4: count shipped changes, not recommendations

From week two, stop counting suggestions and start counting changes someone actually ships. Give each treated URL two columns:

  • Column A — tool execution: what the tool published or fixed itself.
  • Column B — team work: what it handed back as a ticket for your people to complete.

A crawler fix, comparison block, published page, or third-party listing update needs a timestamp and a URL diff. “Optimized” is not an entry. Log your team’s hours beside every item in column B, including the time spent translating a vague recommendation into a usable task.

Lab notebook with columns for tool changes and team tickets beside an untouched control group

Require an action log so you can compare a change made on Tuesday with a citation check the following Monday. The timing will not prove the change caused the citation, but without the log you cannot even examine the claim. I have watched account charts move while nobody could say whether the tool, an intern, or a competitor dropping out deserved credit.

If column A is empty and column B consumed six writer hours per page, you bought a monitor priced as a solution. Keep the invoices and the hour log together. The renewal deck will not do that for you.

Day 30: report three numbers, per surface

Rerun the full fixed prompt set twice in the final week. Use the same settings and clean sessions as the baseline. For each surface, report these three numbers:

  1. Citation rate: the percentage of checks where the answer links to a tested URL, separated into treated and control sets.
  2. New cited prompts: prompts that gained a citation to a tested URL relative to baseline, again separated by set.
  3. Team hours: the time your people spent getting those changes live.

Score Google AI Overviews, ChatGPT, and Perplexity separately. BrightEdge data in this Search Engine Land breakdown shows why surface-level results matter: AI Overviews carry 20 or more inline citations and swing week to week, while AI Mode shows 5 to 7 source cards with less churn. Do not blend unlike surfaces into one comforting number. If treated pages do not pull ahead of control pages on at least one surface, the trial fails, whatever the dashboard says.

Then price what moved. Add the subscription to team hours at a loaded rate you wrote down on day zero, and divide by new cited prompts. For an illustrative comparison, say you spend $20k a month on acquisition and test a $29 Otterly tier for 15 prompts against a $95 Peec tier for 50 prompts and a $99 Profound tier that starts ChatGPT-only (entry pricing comparison). If one tool brings four newly cited prompts with two hours of your time and another brings none with nine hours of tickets, the second tool cannot win on a prettier report. When the denominator is zero, write “no new citations,” not a made-up cost per citation.

For the tool-by-tool companion to this protocol, see Tools That Help You Get Featured in Google AI Overviews, Not Just Monitored. For this trial, keep the decision tied to citations moved and hours spent.

Read the result without letting the sales call rewrite it

There are three useful endings:

  • Treated beats control with low team hours: keep the tool, expand the treated set, and rerun the same prompt list for another 30 days before raising the budget.
  • Reports improve but citations stay flat while hours climb: you bought a monitor priced as a solution. A $29 tracker can still earn its place if your team ships fast. It should not get credit for your writers’ work.
  • Treated and control rise together: investigate what changed outside the tool before claiming a win. A competitor may have published a better table, answers may have shifted, or seasonality may have moved purchase prompts.

Single snapshots lie. The control group, repeated checks, and action log give you a way to challenge the story on either side. If treated pages alone rise and the claimed prompts reproduce in 7 of 10 spot checks, take that result seriously. If the evidence is mixed, say so. End the trial on the pre-written number, not on how the sales call felt.

Put groas through the same trial

Run this protocol on groas without changing a rule: same fixed prompts, same treated-versus-control split, same hour log, same citation checks. In the trial, look for shipped work: crawler access checked, answer-first blocks and comparison tables published, missing pages added, third-party coverage built, and each change timestamped against a URL. That is the mechanism to inspect. A citation lift is an outcome to measure, not a label we get to award ourselves.

Fail us if the treated pages do not beat your control on your pre-written number. Fail us if column A is thin while your team spends six hours per page clearing tickets. Fail us if there is no timestamped log connecting changes to URLs. A trial that cannot fail is a demo with catering.

Copy this scorecard into your trial sheet

  • Prompt discovery: record net-new triggering prompts and spot-check 10 claimed triggers. Pass at 7 of 10 reproducing.
  • Gap diagnosis: for each treated URL, score the missing block, cited alternative URL, retrievability check, and assigned owner. Pass at 3 of 4 on most pages.
  • Execution: count timestamped fixes the tool shipped versus tickets it handed back; log your team’s hours.
  • Outcome: compare treated and control citation rates and new cited prompts on each surface, using two final-week reruns.
  • Cost: add subscription cost to loaded team hours and calculate cost per new cited prompt where there is one.

Keep the vendor that moves citations with the fewest of your hours. Cancel the one that taught you new words for losing.