Estimating fan-out queries with log probabilities

The estimand

Suppose a closed retrieval system has emitted one observed web-search query x as part of a latent query fan-out. Its full internal query distribution is not observable through the public product API. The defensible target is therefore narrower: the most likely other queries from that same fan-out under a fixed, minimal generation experiment with the named provider model.

Open Queries records the provider model m, prompt p and prompt version v. The structured output Y contains exactly 12 strings. The prompt specifies no domain, search operator, language, query category or ranking rule. This does not identify the production assistant's hidden policy; it defines a repeatable proxy whose assumptions can be inspected.

Narrowing the estimand is the most important methodological choice. “What did the production assistant secretly search?” is not identifiable from a public closed-model endpoint. “Which other queries are plausible under this documented generation experiment?” is answerable and useful, provided the output remains labeled as a proxy.

The fixed output size makes candidates comparable within the experiment, while the minimal prompt reduces researcher-imposed categories. It does not remove prompt dependence. Provider, model, temperature or sampling controls, prompt version and output schema therefore travel with every result.

Why the generator and evidence source must match

A score from a second model estimates compatibility under the second model, not under the generator. Even a capable shared ranker changes the target distribution and adds prompt-sensitive judgment. Provider-native evidence removes that avoidable cross-model mismatch.

For OpenAI—and for any explicitly enabled provider endpoint that returns logprobs—the candidate strings and token evidence come from the same completion. There is no second ranking prompt, candidate rewriting step or ordinal fallback.

Cross-model judging introduces an extra latent variable. If Model A generates a query and Model B scores it, the score describes Model B's conditional preferences. A provider-native method keeps generation and evidence under the same model family and removes that particular mismatch, though it does not eliminate model drift or calibration differences.

A provider-native score is still not comparable across providers by magnitude. Tokenizers split strings differently, endpoint implementations expose different probability information and probability calibration varies. Use the scores to order candidates inside the declared run, not to claim that 0.82 from one model is stronger than 0.74 from another.

Token-level inverse perplexity

Let T(q) contain every output token whose UTF-8 byte interval overlaps the JSON string content of candidate q. A token is counted once even if it contains several characters. Character weighting would make the statistic depend twice on token length and would no longer be the standard token-average log likelihood.

The arithmetic mean of conditional token log probabilities is converted into perplexity and inverse perplexity. Since log probabilities are non-positive, inverse perplexity lies in (0, 1]. Higher values mean the realized query required less surprisal per token under this particular model and context.

Token averaging controls for the simple fact that longer strings contain more token log probabilities. Summing would systematically penalize length. Character averaging would impose a second weighting unrelated to the model's tokenization. The chosen statistic is therefore the arithmetic mean over eligible output tokens that overlap the serialized query content.

Inverse perplexity is a monotonic transformation of the mean log likelihood. It is easier to read because larger is better and the result lies at or below one, but the interpretation remains conditional: lower surprisal for this realized candidate under this provider response and prompt context.

Do not present it as a calibrated probability that the production system would issue the exact query. The candidate appeared in a structured multi-item output whose items interact through decoding; the endpoint may not mirror the search product; and only one bounded response is being scored.

• Scores are ordinal evidence inside one provider run.
• They are not calibrated probabilities of a hidden production search.
• Cross-model comparisons inherit different tokenizers and probability calibration.

UTF-8 alignment and failure conditions

JSON escaping and multi-byte characters make JavaScript character offsets insufficient. Open Queries locates each serialized query, converts its boundaries to UTF-8 byte offsets and intersects those boundaries with cumulative provider-token byte spans. OpenAI byte arrays are used when supplied; otherwise the token text is UTF-8 encoded.

A candidate without finite overlapping token evidence is omitted. If fewer than six valid native candidates remain, the request fails closed. The service does not invent scores from output order and does not call a different provider.

The serialized JSON matters because providers return token evidence for output bytes, not for semantic string objects. Escaped quotes, backslashes, emoji and non-Latin scripts can make JavaScript code-unit offsets diverge from UTF-8 byte offsets. Alignment must therefore happen against the exact serialized response.

Property names, commas and quotation marks are excluded from candidate scoring. Tokens whose byte spans overlap the string content are included once. The implementation should retain enough diagnostic metadata to reproduce why a token was included without storing unrelated prompt or response content.

Failure is preferable to a hidden fallback. If alignment cannot produce finite native evidence for enough candidates, the method returns an explicit error. Output order, a second model judge or hand-written weights would change the estimand and must not be substituted silently.

Gemini and Claude as binomial inclusion experiments

When a configured provider endpoint does not return usable output-token log probabilities, the closest working provider-native proxy is repeated sampling from that model under the same prompt. Open Queries makes 16 independent structured calls and requires at least 12 valid samples.

For a normalized query q, K(q) is the number of samples containing q and n is the valid sample count. Inclusion frequency estimates the probability that q appears somewhere in the bounded output under this experiment. A Wilson interval is reported because the naive normal interval performs poorly for small n and proportions near zero or one.

Repeated sampling turns candidate inclusion into a Bernoulli event per valid structured output. Normalization should merge trivial case or whitespace differences while avoiding semantic stemming that could combine distinct queries. A candidate counts at most once per sample even if the model repeats it inside that output.

Sixteen requested samples balance latency and an initial uncertainty estimate, while the minimum-valid threshold exposes provider or parsing failures. With a small n, intervals remain wide; that is information, not a defect to hide. A candidate seen in 8 of 16 runs is less certain than a large-scale estimate, and the Wilson interval makes that limitation visible.

Sampling outputs can be ranked by inclusion count, with deterministic tie-breaking defined in the method. They remain provider-experiment frequencies. Increasing n narrows sampling uncertainty but does not solve prompt dependence or production mismatch.

What the proxy can support

The output is useful for discovering plausible retrieval vocabulary and comparing candidates generated under one controlled provider context. It cannot reveal chain of thought, prove which search a production system issued or substitute for population demand measurement.

Observed UI queries remain the stronger evidence class. Estimated candidates stay in a separate contract and never enter observed-query aggregates. Model, method, prompt version, token counts or sample counts travel with every response so future drift is auditable.

A useful application is research triage. Candidates with strong within-run evidence can seed primary-source searches, reveal a missing comparison axis or diversify a manual prompt panel. The editor then validates the underlying question and publishes only sourced answers.

A dangerous application is synthetic market sizing. Model likelihood and inclusion frequency are not monthly searches, user frequency or expected traffic. They should never be multiplied by conversion rates or blended with Google Ads volume as if the units matched.

Reproducibility is bounded by provider access and model versions. Store the response method, model identifier, prompt hash or version, timestamp, valid-candidate count and token/sample diagnostics. A rerun after a model update is a new experiment, not a continuation that can be compared without qualification.

Worked example: score one candidate without overstating it

Suppose the serialized candidate “AI search visibility workflow” overlaps four output tokens with log probabilities −0.20, −0.45, −0.35 and −0.60. The mean log likelihood is −0.40, so inverse perplexity is exp(−0.40), approximately 0.6703.

The correct interpretation is that this realized candidate had an average per-token likelihood corresponding to 0.6703 under the model, prompt and surrounding output in that run. A second candidate with 0.74 ranks higher within the same run. Neither value is a 67% or 74% probability of a production search.

Publish enough metadata to reproduce the proxy

A mathematical formula without an execution contract is not reproducible. The implementation and response should expose the choices that can change candidate generation, alignment or ranking.

• Provider and exact model identifier returned or configured for the run.
• Prompt/template version and structured output schema version.
• Sampling controls, requested candidate count and normalization rules.
• Native log-probability availability and the byte-alignment method used.
• Valid and omitted candidate counts with explicit failure thresholds.
• For repeated sampling: requested calls, valid calls, inclusion count and Wilson interval.
• Timestamp and a warning that cross-model and cross-version score magnitudes are not calibrated.