The estimand

Suppose a closed retrieval system receives an information need x and may emit latent search queries. Its true internal query distribution is not observable through the public product API. The defensible target is therefore narrower: candidate queries that are compatible with the distribution of a named provider model under a fixed, minimal generation experiment.

Open Queries records the provider model m, prompt p and prompt version v. The structured output Y contains exactly 12 strings. This does not identify the production assistant’s hidden policy; it defines a repeatable proxy whose assumptions can be inspected.

Y = (q₁, …, q₁₂) ~ Pₘ(· | x, p, v)

Why the generator and evidence source must match

A score from a second model estimates compatibility under the second model, not under the generator. Even a capable shared ranker changes the target distribution and adds prompt-sensitive judgment. Provider-native evidence removes that avoidable cross-model mismatch.

For OpenAI—and for any explicitly enabled provider endpoint that returns logprobs—the candidate strings and token evidence come from the same completion. There is no second ranking prompt, candidate rewriting step or ordinal fallback.

Token-level inverse perplexity

Let T(q) contain every output token whose UTF-8 byte interval overlaps the JSON string content of candidate q. A token is counted once even if it contains several characters. This matters because character weighting would make the statistic depend twice on token length and would no longer be the standard token-average log likelihood.

The arithmetic mean of conditional token log probabilities is converted into perplexity and inverse perplexity. Since log probabilities are non-positive, inverse perplexity lies in (0, 1]. Higher values mean the realized query required less surprisal per token under this particular model and context.

ℓ̄(q) = (1 / |T(q)|) Σₜ∈T(q) log Pₘ(t | t<t, x, p, v)PP(q) = exp(−ℓ̄(q))s(q) = PP(q)⁻¹ = exp(ℓ̄(q))
  • Scores are ordinal evidence inside one provider run.
  • They are not calibrated probabilities of a hidden production search.
  • Cross-model comparisons inherit different tokenizers and probability calibration.

UTF-8 alignment and failure conditions

JSON escaping and multi-byte characters make JavaScript character offsets insufficient. Open Queries locates each serialized query, converts its boundaries to UTF-8 byte offsets and intersects those boundaries with cumulative provider-token byte spans. OpenAI byte arrays are used when supplied; otherwise the token text is UTF-8 encoded. The same mapping is available for Gemini chosen-candidate tokens whenever a Google endpoint enables them.

A candidate without finite overlapping token evidence is omitted. If fewer than six valid native candidates remain, the request fails closed. The service does not invent scores from output order and does not call a different provider.

Gemini and Claude as binomial inclusion experiments

Anthropic does not return output token log probabilities for Claude through its public Messages API. Google documents the responseLogprobs field, but the configured Gemini 3.1 Flash-Lite Developer API endpoint currently rejects it. The closest working provider-native proxy is repeated sampling from each model itself under the same prompt. Open Queries makes 16 independent structured calls and requires at least 12 valid samples.

For a normalized query q, K(q) is the number of samples containing q and n is the valid sample count. Inclusion frequency estimates the probability that q appears somewhere in the bounded output under this experiment. A Wilson interval is reported because the naive normal interval performs poorly for small n and proportions near zero or one.

K(q) = Σᵢ₌₁ⁿ 𝟙[q ∈ Yᵢ]p̂(q) = K(q) / nCI₉₅ = (p̂ + z²/2n ± z√(p̂(1−p̂)/n + z²/4n²)) / (1 + z²/n)

What the proxy can support

The output is useful for discovering plausible retrieval vocabulary and comparing candidates generated under one controlled provider context. It cannot reveal chain of thought, prove which search a production system issued or substitute for population demand measurement.

Observed UI queries remain the stronger evidence class. Estimated candidates stay in a separate contract and never enter observed-query aggregates. Model, method, prompt version, token counts or sample counts travel with every response so future model drift is auditable.