Observed vs. estimated AI search queries

Observed queries

An observed query must appear inside a supported search-specific UI or transport boundary. Generic conversation text and arbitrary fields named query are not eligible.

If the provider changes its interface and that boundary becomes uncertain, the adapter fails closed rather than broadening collection.

The observation contract should name what makes an event eligible. A search-specific UI component or structured transport event can provide an explicit query field. Generic text inside the conversation, source titles and arbitrary fields named query do not qualify merely because they contain search-like words.

An observed label supports a narrow claim: this provider surface exposed this string at this time under this adapter contract. It does not prove the complete retrieval plan, the user's original wording, the source selected because of the query or a stable provider preference.

Versioning turns failure into evidence. When the interface changes, fixture and live checks determine whether the semantic boundary still holds. If not, the adapter stops collecting and marks the period unsupported instead of broadening its selector until data reappears.

Estimated queries

An estimated query is created only after privacy is accepted and a user asks for adjacent fan-out ideas. The named provider model generates the candidates, and provider-native evidence ranks or summarizes them.

The score supports ordering within one bounded experiment. It is not a claim that the production assistant issued the query.

A useful estimator declares the target and method. Open Queries asks the named provider model for a bounded set of plausible adjacent searches under a versioned minimal prompt. When native token log probabilities are available, they provide within-run likelihood evidence. Otherwise repeated provider-native samples provide inclusion frequency and uncertainty.

Neither method recreates a hidden production trace. The prompt is an experimental condition, model versions drift and product retrieval can use undisclosed context or systems. Estimates are best used to plan research, diversify a test set and identify candidate qualifiers.

The estimate should be generated only on explicit request and kept in a separate storage and reporting contract. A user can compare it with observations, but aggregation must not silently count it as provider activity.

The evidence contract

The label should answer where the string came from, which collection contract recognized it and what claim is safe after the record leaves the original interface.

Provenance also defines the denominator. An observed count is the number of eligible captured events in a window. An estimated frequency is the number of model samples containing a candidate. Google Ads volume is a monthly market estimate. Search Console impressions are recorded site appearances. Identical numerals do not make those units comparable.

Safe dashboards keep class, provider, method version and time window visible. If a summary number cannot be explained from those fields, it is too compressed to drive an editorial decision.

One string can support different claims

The words alone do not determine the evidence class. The same query text could be observed in a provider interface, generated in an experiment or measured in Google Ads.

Suppose the string “AI search optimization workflow” appears three times: once in ChatGPT Search, in nine of sixteen Claude estimation samples and with monthly Google Ads volume in a verified export. The correct record has three provenance rows. It does not have a synthetic score that adds 1 + 9 + volume.

An editor can still use the convergence. The human-demand estimate supports prioritization, the observation supports retrieval relevance and the estimate suggests that the phrasing is plausible under another bounded context. Research then verifies what a genuinely useful workflow must contain before the canonical is changed.

A hard storage and reporting boundary

Estimated candidates are never inserted into observed-query events. Reports must also keep Store installations, GitHub downloads, D1 product events and Search Console traffic separate.

This discipline makes the data less dramatic and far more useful: every metric keeps a defensible meaning.

Data boundaries should be enforceable in code. Event schemas use different discriminants for observed and estimated rows; queries cannot be reclassified by a reporting view; and migrations preserve old provenance. Tests should reject estimated rows inserted into the observed-event path.

The same discipline applies to growth reporting. Store installations require a verified install event. GitHub downloads, D1 activity, Search Console clicks and assistant referrals remain separately named signals. Precision at the query layer is undermined if the business layer uses convenient euphemisms.

Use a provenance ledger that survives aggregation

The ledger records enough context for a future reviewer to reproduce the interpretation. A normalized-query column can group matching strings without deleting their original evidence rows.

Apply five rules when evidence moves into content

A provenance-safe database is not enough if an editorial brief strips all of the qualifiers. These rules keep the published claim no stronger than the underlying record.

The reviewer should be able to walk backward from a sentence on the public page to the primary source that establishes the fact and separately to the query evidence that motivated the research. The source supports truth; the query record explains prioritization. Combining those roles makes a private trace appear authoritative and hides whether the factual claim was ever verified.

The forward path matters too. When a paragraph changes, the decision log records the owning canonical, changed claim, release date, expected observable signal and earliest review date. A later movement can then be evaluated against the exact intervention without pretending that production search offers controlled experimental conditions.

• Quote or paraphrase a provider behavior only as broadly as its official documentation or dated interface evidence permits.
• Use an observed query as a clue to investigate, not as the public authority for a factual claim.
• Use estimates to diversify research and tests, never to claim production demand or provider frequency.
• Keep missing evidence unknown; explain coverage gaps instead of backfilling them with assumptions.
• Attach the final claim to the strongest primary source, while retaining the query evidence in the private decision log.