Apify Actor Discoverability & SEO Audit

Design notes for an Actor that grades Apify Actors against an evidence-grounded rubric for AI-agent discoverability.

Design notes for apify-discoverability-audit, an Apify Actor that scores another Apify Actor against an evidence-grounded rubric for AI-agent discoverability. The audit fetches the target Actor’s Store metadata, latest build (full README, input schema, dataset schema), and Store row (pricing model, agentic-payment whitelist flag), then runs an eleven-dimension rubric — nine deterministic checks over that metadata plus two live probes that ask whether an agent searching for the capability actually finds the Actor — and returns a punch list of failed checks plus LLM-written suggestion text for everything that didn’t pass.

The Apify Store page covers the schema, current pricing, and how to try it. This page is for the design questions — why the rubric looks the way it does, why every status verdict is deterministic, and what the Actor deliberately refuses to grade.

What this is

Submit an Actor identifier (username/name or a 17-character Actor ID). Optionally pass a canonical-page URL (for off-platform cross-link checks). The audit returns one dataset record with:

• An overall verdict — PASS / WARN / FAIL — derived from the worst-category status across eleven rubric dimensions.
• A per-dimension category summary — one status per dimension (agentic-payment eligibility, description quality, schema completeness, disambiguation, tool name, README structure, store hygiene, maintenance (build recency and reliability), off-platform canonical signals, live agent-search retrieval, and Store presence — the Actor’s rank and runnable status on the Store’s own listing).
• The full punch list — every individual check with its status, a short evidence line, and the dimension it belongs to.
• Suggestions — one copy-paste-ready improvement text per failed or warned check, written to the rubric’s concepts (six-component description framework, the five-hundred-character MCP truncation, the thirteen-section README template, the disambiguation block, PAY_PER_EVENT pricing).
• Warnings — any data-collection issues (private Actor, build endpoint outage, unreachable canonical page) surfaced separately from rubric verdicts.

The same record also lands as a readable Markdown punch list in the run’s Key-Value Store, served as text/plain so the bytes return verbatim (the KVS serve layer would otherwise inject a cookie banner script into HTML responses).

Why this exists

AI agents pick tools by reading their descriptions and schemas, not by browsing to a documentation site. The Apify MCP server truncates input-schema property descriptions at five hundred characters before appending enum values and examples — tools whose first five hundred characters don’t pre-load purpose, parameters, and constraints fail at the routing layer. Tools that lack explicit disambiguation guidance (“use this other tool instead for that other need”) lose to similar-sounding alternatives by default. Tools that aren’t on the PAY_PER_EVENT pricing model with isWhiteListedForAgenticPayments=true are invisible to agents using x402 or Skyfire payment rails.

There’s a body of recent research on this. Wang et al. 2026 (“From Docs to Descriptions”, arxiv 2602.18914) measured a two-hundred-sixty-percent lift in tool selection from standards-compliant descriptions. “Tool Preferences in Agentic LLMs” (arxiv 2505.18135) found greater-than-ten-times usage swings from description edits alone, across multiple model families. BiasBusters (arxiv 2510.00307) identified semantic alignment as the dominant tool-selection driver. Anthropic’s engineering guidance points at description quality and explicit disambiguation as the highest-impact surfaces for tool authors.

The rubric this Actor scores is the executable form of that evidence. Eleven dimensions, weighted by what the research says matters. Nine are deterministic checks over the Actor’s own metadata; two are live, extrinsic probes that ask whether an agent searching for the capability actually surfaces the Actor. The LLM writes suggestion prose and generates the retrieval probe’s search phrases — it never sets a verdict.

Why every status verdict is deterministic

Tool-selection auditing is easy to do badly. The naive shape is “hand the Actor’s metadata to an LLM and ask if it’s good.” That works for a single pass but it’s unreproducible (the LLM picks different verdicts each run), unauditable (you can’t trace why a given dimension got a given status), and trivially fooled (a slick top-level description can mask a half-empty input schema).

This Actor runs every check as a pure function against the fetched metadata. The agentic-payment eligibility check reads pricingInfos[-1].pricingModel. The MCP-truncation check measures len(property.description) > 500. The disambiguation check applies a regex against the “What this doesn’t do” section heading and its body. The build-validator-gotcha check looks for items.enum on array properties (which Apify’s build validator hard-rejects).

The LLM is invoked at most twice per audit, and never to set a verdict: once to synthesize copy-paste-ready suggestion prose for already-failed checks, and once — a small call — to generate the intent phrases for the live retrieval probe described below (both are independently switchable, and the probe’s phrase generation falls back to a deterministic extractor when no model key is set). The suggestion call receives the list of failed check IDs, the target Actor’s description and README excerpt (wrapped in NO_EXECUTE injection-protection tags), and the suggestion-seed hints from the checks. It returns a Pydantic-validated SuggestionsResponse structured-output object, which the pipeline then filters to drop any suggestion whose check_id isn’t in the actually-failed set. The model can’t invent dimensions, change verdicts, or score subjective categories — and the audit’s output is reproducible run to run.

How the rubric is weighted

Not all eleven dimensions matter equally. The research is clear on which ones move the needle:

The “low weight, hygiene only” framing on canonical signals is deliberate. JSON-LD and llms.txt are no-cost to add, but Search Atlas’s December 2025 three-hundred-thousand-domain LLM-citation study found a null effect on citations, and Limy’s 2025 ninety-day bot-traffic log study found llms.txt fetched less than one percent of bot visits. The audit emits informational checks for those signals without scoring them — the work-to-payoff ratio doesn’t justify treating them as anything more.

Why the agentic-payment check has tri-state semantics

When an audit can’t fetch the target’s pricing information — most commonly because the Actor is unpublished or its Store row hasn’t indexed yet — the agentic-payment dimension reports WARN, not FAIL, and the agentic_payment_eligible field comes back as null rather than false. “Unconfirmed” is genuinely different from “confirmed non-PAY_PER_EVENT”, and propagating an unknown as a hard fail is the worst kind of misleading audit output.

Two paths feed the pricing lookup. The first is the authenticated Actor-object endpoint (/v2/acts/<slug>?token=...), which returns the full pricingInfos[] history whenever you own the target Actor — works for private Actors pre-publish, works for newly-published Actors whose Store row hasn’t indexed yet. The fallback is the public Store search (/v2/store?search=<name>), which works for any other published Actor but is fuzzy and popularity-ranked enough that recently-published Actors sometimes return zero hits. The audit tries both before reporting WARN; the warning makes the cause explicit.

The two live dimensions: retrieval, not readability

Nine of the eleven dimensions grade the Actor’s own copy — its description, schema, README, pricing. They answer “is this well-written for an agent that’s already looking at it?” That’s selection readability. But an agent has to find the Actor before it can read it, and findability is a different question that the Actor’s own metadata can’t answer. So two dimensions step off-platform and measure it directly — the only two that make a live network call, and the reason they’re diagnostic rather than hard-weighted.

Live agent-search retrieval queries the agent-facing Actor search — the surface an MCP client actually routes through — for the Actor’s own name and for the intent phrases an agent would type, and reports where it ranks. Not resolving for your own name is a real bug and fails. A capability the Actor has but never names in the words agents search is a fixable under-naming gap and warns. Everything else is informational, on purpose: rankings shift run to run, and losing an intent phrase to a specialist Actor literally named for it is a positioning fact, not a copy defect — surfacing it is useful, letting it drag down a verdict about fixable copy quality is not.

Store presence measures a different surface: the Store’s own default listing, which website browsers and third-party scanners read. It reports the Actor’s rank there and — more usefully — whether the Actor is silently dropped from that listing because it’s flagged unrunnable, the symptom of incomplete payout/KYC onboarding. An unrunnable Actor stays reachable by direct link and through the agent search, so nothing else in the audit notices; the default Store list is the one place it goes invisible, and this is the check that catches it. The probe is deliberately anonymous: querying the Store while authenticated returns your own rows regardless of listing status, which would mask exactly the hiding it’s meant to detect.

How it compares to alternatives

The differentiating axis is deterministic-first. Every status verdict comes from a pure function against the fetched metadata, so the audit reads what an MCP-mediated agent would actually read, and grades it against the same rubric a writing tool uses to author it.

For one-shot improvement of an existing Actor’s customer-facing copy, a writing skill that writes against the same standards is the better tool. For ongoing tracking of a portfolio’s discoverability over time, run this audit on a cron and diff the dataset records. For competitive landscape analysis against other published Actors in the same niche, use a dedicated Store-search workflow. For deep canonical-page rewrites, the off-platform canonical-signals dimension here only checks cross-link hygiene — actually rewriting a canonical page is a different job.

Pricing model

Pay-per-event, billed only on success. A single flat charge per successful audit, after the dataset record is pushed. Setting include_suggestions=false skips the LLM call but the audit charge is the same — the deterministic punch list is the costed work.

The platform-managed apify-actor-start event fires once per run at the platform minimum, so per-run overhead is effectively zero. Failed runs (invalid input, target Actor unreachable) don’t trigger the audit charge.

For the current per-event rate and any active subscriber discounts, see the Pricing tab on the Apify Store page.

Open questions / future work

A few things worth watching:

• Batch input. Right now the Actor takes one target per run. A batch mode (actors: [...]) would make portfolio sweeps cheaper, but introduces a different cost model and the need for per-target failure isolation — non-trivial enough to wait until there’s real demand.
• Auto-fixer mode. The audit already writes suggestion prose; a future variant could optionally push the suggested rewrites back into the target Actor’s source files via apify push. Tempting but risky — agents auto-rewriting their own descriptions is one bad merge away from a degenerate fixed point.
• Per-dimension weighting in the verdict. The current verdict is a worst-category-wins rollup. Some users may want a weighted-sum score where a single weak dimension doesn’t drag the whole verdict to FAIL. The Wang et al. 2026 paper has weights I could lift directly; worth doing once the rubric stabilizes against real usage.
• Heuristic improvements for the disambiguation regex. “use [X] instead for [Y]” is detected by three regex variants today. False positives are easy to imagine (use the input field…); a small classifier trained on real examples would be more robust without losing transparency.
• Audit the audit. A self-recursive run on the audit Actor itself is already part of the dogfood path, but a periodic CI job that runs the audit against the latest published build and fails on regressions would catch description-quality drift across edits — exactly the use case the Actor was built for, applied to itself.