Convergence Radar Convergence Engine

← Feed

F

Skill Telemetry: Usage Analytics for Agent-Facing Skill Files

23/100

A drop-in telemetry convention plus dashboard that shows skill publishers whether AI agents actually load, follow, or violate their published skills β€” web analytics for the new agent-docs substrate.

Kill. Β· created 2026-07-10 03:48 UTC

aisaasapiagentrevisit later

Scorecard

newness 8/10
convergence 3/10
demand evidence 0/10
existing spend 1/10
solo feasibility 5/10
speed to mvp 6/10
speed to revenue 3/10
distribution 3/10
competitive gap 6/10
expansion 6/10
founder fit 3/10

Penalty flags
large integrations long trust cycle no clear buyer no urgent pain platform policy risk (βˆ’18 from raw 41)

Opportunity brief

What changed
HYPOTHESIS (from convergence description, not from any provided source): developer documentation is shifting from human-read web docs to agent-loaded skill files (e.g. vendor-published skills, community skill repos), and publishers of those skills have no equivalent of web analytics to see usage or compliance.
Why now
If the shift is real, the substrate is brand new and has zero incumbent observability tooling β€” a genuine greenfield moment. However, NONE of the provided signals substantiate the shift: the two attached signals (a Chrome speculation-rules experiment and a one-off robot mobility build) are unrelated to agent skills. The 'why now' rests entirely on the unshown signals 488/177 referenced in the causal chain. Treat the entire premise as unverified inference.
Converging signals
Claimed convergence: docs-to-skills migration + publisher blindness => paid instrumentation layer. The graph cluster metadata (c3, growth +4, cohesion 0.917) suggests semantic coherence, but the two concrete signals actually supplied are off-topic, so the convergence cannot be verified from the evidence in front of me. Convergence scored down accordingly.
Customer pain
HYPOTHESIS: DevRel/platform teams spend headcount writing agent skills and cannot measure whether agents load them, follow them, or fail on them, so they cannot justify or tune the spend. Plausible, but the demand_evidence array is EMPTY β€” there is not one complaint, job posting, or mandate in the input proving anyone feels this pain today.
Who pays
HYPOTHESIS: DevRel and platform-engineering teams at AI-tooling vendors that publish skills (the convergence names Supabase-type vendors). This is a budget-holder that exists, but reaching them is developer-tools B2B sales into companies β€” not the founder's proven forced-buyer channel, and no evidence of willingness to pay was provided.
Solved today
HYPOTHESIS: not solved. Publishers ship skill files into repos/registries and get, at best, GitHub stars and clone counts as a proxy. No provided source confirms this, but no counter-evidence of an incumbent exists in the input either.
Why current solutions are bad
Repo metrics measure distribution, not behavior: they cannot show whether an agent loaded the skill mid-session, followed its instructions, or deviated. That gap is structural β€” but note it is also structurally hard for a THIRD PARTY to fill, because the telemetry must be emitted from inside the agent harness, which the publisher and founder do not control.
Proposed product
A lightweight telemetry convention (a snippet/hook skill publishers embed) plus a hosted dashboard: skill load counts, instruction-compliance signals, common failure/deviation patterns, version comparison. Sold per-publisher as micro-SaaS.
MVP version
A hosted endpoint + embeddable instrumentation line for skill files, a compliance-checking harness that replays a publisher's skill against test prompts across agent versions, and a dashboard showing load/follow/violate rates. The replay-harness variant ('skill CI/testing') is more buildable solo than passive telemetry because it does not require field adoption inside third-party agent sessions.
30-day build
Validate the premise before building: interview 10-15 people who maintain published skill repos; scrape skill registries/repos to size the publisher population; verify whether agent harnesses even permit outbound telemetry from a loaded skill (this is the load-bearing technical question and a potential hard kill).
60-day build
If validated: ship the replay/testing harness MVP for 3-5 design-partner publishers free, publish a 'state of skill compliance' benchmark report as the demand-gen asset (fits founder's demonstrated-value sales style).
90-day revenue plan
Convert design partners to $99-$299/mo subscriptions; target 5-10 paying publishers. Realistic first revenue is in the 120-180 day window given the validation phase β€” acceptable under the founder's current runway per the capital-and-runway lesson (confidence 0.9), but only if 30-day validation passes.
Distribution path
Direct outreach to skill-repo maintainers on GitHub, the benchmark report, and presence in agent-tooling communities. No paid-ad dependence. Weakness: the buyer population is currently small and undefined, and no channel exists yet where they congregate.
Pricing hypothesis
$99-$299/mo per publisher (micro-SaaS), or per-skill-tested pricing for the CI variant. HYPOTHESIS β€” zero pricing evidence in input.
Technical difficulty
Moderate for the replay/CI variant (solo-feasible with AI-assisted builds). HIGH for true field telemetry: a skill file cannot reliably phone home from inside someone else's agent session without harness cooperation β€” meaning the honest version of the headline product depends on platform vendors (Anthropic et al.) adopting the convention, which is out of a solo founder's control.
Legal / regulatory risk
Low-moderate: telemetry emitted from inside user agent sessions raises privacy/consent questions, and instructing an agent to make outbound calls for tracking could be treated as abusive by platform vendors.
Platform dependency
SEVERE. The skill format, the harness, and any native analytics all belong to the platform vendors. Anthropic shipping first-party skill analytics β€” an obvious roadmap item β€” kills the third-party product overnight. This is the single strongest kill argument.
Founder fit
Weak-to-moderate. This is developer-tools B2B to DevRel teams β€” no forced buyer, no government portal, no filing mandate, none of the founder's proven ELDT-shaped edge (government-mandate lesson, confidence 0.8). His AI-workflow and fast-prototyping strengths apply to building it, but the selling motion (evangelizing a new convention to platform teams) is a trust/adoption play he explicitly avoids.
Breakout potential
If the convention were adopted broadly it becomes the analytics layer of a new substrate β€” genuinely large. But convention-setting is a network-effect, ecosystem-politics game that favors platform vendors, not solo outsiders.
Final recommendation
KILL for now / REVISIT LATER. The idea is conceptually coherent and early, but it fails on the right kill criteria: no evidence any buyer exists or pays, the honest version of the product requires platform cooperation a solo founder cannot compel, and the incumbent (platform vendor) can ship it natively. The only survivable wedge β€” a skill replay/CI testing harness that runs on the publisher's side β€” is worth a 2-week validation sprint, not a build. Revisit if/when demand signals (job postings mentioning skill authoring/maintenance, complaints from skill publishers) actually appear in ingestion.
Next action
Do NOT build. Add a demand-side watch: monitor GitHub issues/discussions on major skill repos and job postings for 'agent skills'/'skill authoring' pain; if 5+ genuine publisher complaints about usage blindness surface within 60 days, run the 10-interview validation sprint on the replay/CI variant.

Kill arguments (adversarial)

Competitors

β€’ Anthropic (first-party skill/agent analytics) (link) β€” HYPOTHESIS: not shipped today, but the platform vendor is the natural owner of skill usage analytics and can make any third-party convention obsolete.
β€’ LLM observability vendors (LangSmith/Langfuse/Helicone class) (link) β€” HYPOTHESIS: adjacent incumbents already instrumenting agent traces; adding per-skill attribution is an incremental feature for them.

Source citations (facts)

β€’ [WEB API] Speculation Rules - moderate viewport heuristics controls β€” Provided as a converging signal but is FACTUALLY unrelated to agent skills (it concerns Chrome prefetch heuristics marked experimentation-only); it provides no support for the skill-telemetry premise.
β€’ Son Modifies Industrial Robot to Take His Father Places Wheelchairs Can't β€” Provided as a converging signal but is FACTUALLY unrelated to agent skills (a one-off six-figure hardware build); it provides no support for the skill-telemetry premise.

Actions