What changed
HYPOTHESIS (per convergence description; supporting sources NOT present in this input's signals array): platform vendors like Supabase have started shipping agent-consumable skill files as a documentation substrate, curated skill repos are emerging, and agent task-benchmarking methodology (IBM ScarfBench) has matured. Skills are now code-like artifacts that can silently break when models update.
Why now
If the premise holds, every frontier-model release is a potential silent breakage event for every published skill file, and no QA layer exists for this substrate. The window is early: the artifact class is new enough that no incumbent sells 'skill regression testing', but eval tooling to build it is commodity.
Converging signals
The convergence description asserts three signals (vendor skill files, curated skill repos, matured agent benchmarking). CAUTION — DATA INTEGRITY: the three signal URLs actually provided in this input (IRS estate-tax closing-letter fee, Chrome speculation-rules experiment, quadruped-robot mobility chair) are unrelated to this thesis and do not support it. The entire converging-signal basis is therefore unverified hypothesis within this input.
Customer pain
HYPOTHESIS: DevRel/platform teams see support tickets and churn rise when coding agents scaffold their product wrong, and currently have no way to know a model update broke their skill file until users complain. No demand_evidence was provided — zero complaints, zero job postings, zero mandate — so this pain is asserted, not evidenced.
Who pays
DevRel / developer-experience / platform teams at API-first dev-tools companies that publish agent skill or context files (Supabase-tier and below). Identifiable and reachable (their skill files are public on GitHub), but willingness to pay is unproven.
Solved today
HYPOTHESIS: vendors hand-test skills ad hoc when authoring them, or run internal one-off evals; most likely do nothing after publishing and find out from support tickets. General eval platforms (promptfoo, Braintrust, LangSmith) exist but are frameworks the vendor must operate, not a per-skill regression service.
Why current solutions are bad
Ad-hoc testing doesn't recur on model releases, which is exactly when breakage happens. Generic eval frameworks require the vendor to design task batteries and maintain harness infra — the specific artifact (a skill file steering a coding agent through real scaffolding tasks) needs opinionated, maintained task suites per product domain.
Proposed product
Skill CI: vendor points the service at their skill file; the harness runs a battery of realistic agent tasks (scaffold an app, wire auth, run migrations) with and without the skill, on each new model/agent-CLI release, and emails a diff report: pass/fail per task, regressions since last release, and suggested skill edits. Public free tier: a 'skill health' leaderboard of popular open skill files as the demand-gen demo.
MVP version
A harness around headless agent runs (the founder already operates exactly this pattern in production with claude -p) + 10–15 canonical tasks for 3–5 popular skill files + a scoring rubric + a markdown/HTML report. 2–4 weeks of solo AI-assisted work; compute cost is modest and fundable.
30-day build
Build harness; run it free against 5–10 public vendor skill files on the next model release; publish the regression report publicly and send it directly to each vendor's DevRel team. This is a demand probe disguised as marketing — replies and 'can you run this on ours' requests are the validation gate.
60-day build
If ≥2 vendors engage: productize recurring runs, add their private task suites, close 1–3 paid pilots at $500–1000/mo. If zero engage after direct delivery of a report showing their skill broke, kill the idea — that is a clean falsification.
90-day revenue plan
2–4 vendor subscriptions at $500–2000/mo ($1k–6k MRR) plus possible one-off 'skill authoring + baseline eval' projects at $2–5k. Realistic first revenue day 60–120, within the 180-day window but NOT 30-day cash.
Distribution path
Public skill-health leaderboard + direct outreach to a finite, enumerable list of vendors who publish skill files (discoverable via GitHub). Sells through demonstrated value (here is your broken skill, with evidence) — matches the founder's stated selling style. List is small today, which caps near-term TAM.
Pricing hypothesis
$500–2000/mo per vendor subscription depending on task-suite depth and release cadence; one-off baseline evals $2–5k. Priced against the cost of one support-engineer week per quarter, not against eval-tool licenses.
Technical difficulty
Low-moderate for this founder: headless agent orchestration, sandboxed task runs, diffing/scoring, report generation — all squarely inside his proven automation/AI-workflow stack. Hard part is task-suite design per product domain, which is grind, not risk.
Legal / regulatory risk
Minimal. Running public skill files against agents in sandboxes; no PII, no regulated data. Only mild ToS considerations around automated use of agent CLIs at volume.
Platform dependency
Real but diversifiable: depends on agent CLIs/models remaining accessible for headless runs, and on the 'skills' substrate itself persisting. If skill files turn out to be a 6-month fad, the product dies with them — this is the core macro bet.
Founder fit
Moderate (5/10). Strengths align on the build side: headless-agent automation is literally his production pattern, fast prototyping, demonstrated-value selling. But this is NOT his proven wedge — no government mandate, no forced buyer, no per-filing transaction; buyers are discretionary-budget DevRel teams. The high-confidence lesson favoring government-portal mandate shapes applies and this idea scores structurally below that bar.
Breakout potential
If agent-skill files become the standard doc substrate, Skill CI becomes 'CI for the agent layer' — expansion into skill authoring, MCP-server regression testing, and agent-integration certification. Real upside, but entirely contingent on the substrate winning.
Final recommendation
CONDITIONAL GO — as a cheap 30-day demand probe only, not a committed build. The idea survives the right-reasons kill test (reachable buyer, plausible wedge, solo-operable, no VC needed) but has zero evidenced demand and sits below the founder's proven forced-buyer pattern. Spend ≤4 weeks and modest compute to publish one free regression report on the next model release and let vendor response decide. Do not build the subscription product before that signal.
Next action
Enumerate 10 public vendor-published skill files on GitHub; on the next major model/agent release, run a with/without-skill task battery on 5 of them and email each vendor's DevRel contact the regression report. Gate all further investment on replies.