A real report, written by 00 · Product Improvement Discovery, run by Claude Code against agentic-dev-os — the maintainer's agent-governance repo, a codebase this catalog was not tuned on — dogfood output, committed unedited. This is the artifact every brief ends in: findings that cite their evidence, ranked by severity, with a fix sketch each.
Date: 2026-07-12 (First run — no previous IMPROVEMENTS.md existed.)
Agentic Development OS is a repository-native, zero-dependency (Python 3.11+ stdlib) governance system for teams shipping software with AI agents: it chains product intent (outcome → opportunity → bet → PRD → spec → ticket) to bounded agent execution loops, and proves the chain with append-only JSONL events, 15 delivery metrics, threshold alerts, and a static HTML dashboard. Its users are AI-native product/platform teams (PRD-001 docs/prd/PRD-001-agentic-development-os.md:21) — and, critically, their agents, which are expected to operate the CLI (scripts/ados.py, 16 subcommands) inside "Ralph loops" with file-scope and risk-tier enforcement. The intended journey is: clone → make verify + make demo (README quick start) → adapt one ticket in shadow mode → graduate to autonomous R0/R1 execution once metrics stabilize (docs/specs/SPEC-002-adoption.md) → publish case studies per the growth playbook (docs/growth/). The repo is one day old (git log: 2026-07-11..12, 5 commits), fully green on its own 100-obligation compliance gate, and all 30 tests pass (verified this run). Its core promise is "evidence over memory" (README:104) — which makes the verified findings below sting: verification evidence can be forged with a flag, the demo command deletes the real event history, and CI's metrics step emits false alerts on every fresh checkout. There is also notable half-built infrastructure: a 10-tool agent catalog with no executable surface (tools/catalog.json), a dashboard definition file the renderer never reads (observability/dashboards/default.json), five catalogued event types nothing can emit, and timestamps on every event that no report ever aggregates into a trend.
| Effort S (≤1 day) | Effort M (days) | Effort L (week+) | |
|---|---|---|---|
| Impact H | #1 dry-run forgery, #2 demo wipes history, #3 false/duplicate alerts, #8 loop status + help | #10 real trace impact analysis, #12 metric history & trends | #13 upstream scaffolding, #14 tool catalog → agent server |
| Impact M | #4 input hardening, #5 stale audit dead in CI, #9 show verification output | #6 product-gate filter, #7 corrupt-event recovery, #11 dashboard uses its definition, #15 emit-able lifecycle events | — |
Generated ~30 raw ideas; curated to 15. Cut in curation: wiring the configured-but-never-run ruff/pyright (pyproject.toml:8-12 vs Makefile:18-20; the unused import shutil at scripts/ados.py:8 shows the gap), README badges from metrics.json, Windows make support, glob support in scope rules — real but low-impact or generic.
Quick wins (~a day each): #1 close the --dry-run verification hole · #2 make demo non-destructive · #3 no-data alert guard + alert dedupe · #4 CLI input hardening · #8 loop status + named-ticket errors + subcommand help.
Big bets: #13 scaffolding for the entire upstream chain (new bet|outcome|… + trace add) · #14 turn tools/catalog.json into a working agent tool server · #12 metric history and trends (unlocks the product's own graduation gates).
--dry-run — FIX · Fixes & felt debtscripts/ados.py:89 passes not args.dry_run into LoopManager.verify; src/agentic_os/governance.py:154-160 then records returncode: 0 for every command without executing anything and sets run["verified"]=True; governance.py:173-174 lets stop --outcome success through on that flag. Reproduced in a sandbox copy: loop start → loop verify --dry-run → loop stop --outcome success succeeds, and loops.jsonl contains verification.completed outcome: passed, checks: [{make verify, rc 0}, {make demo, rc 0}] — indistinguishable from a real run.executed: false in the event and loop state; make stop --outcome success reject non-executed verification (tests that use execute=False should assert the rejection instead).tests/test_end_to_end.py:19 and tests/test_governance.py:27 which stop successfully after execute=False — update them deliberately.make demo destroys real event history — FIX · Retentionscripts/ados.py:161 — event_path.unlink(missing_ok=True) deletes observability/events/loops.jsonl before emitting demo events. README:24-26 tells every new user to run make demo. Reproduced: 6 real events → demo → 5 synthetic events. (make clean, Makefile:37, also deletes all event streams.)observability/events/demo.jsonl or a temp dir), render to preview paths, and never unlink the canonical stream.src/agentic_os/metrics.py:44-53 breaches thresholds on zero-run rates: with no events, first_pass_success_rate 0.0 < 0.80 and event_completeness_rate 0.0 < 0.95 (ado.config.json:10-11). Reproduced: fresh tree → ados.py metrics → 2 alerts. Since loops.jsonl is gitignored (.gitignore:8), CI's make metrics step (.github/workflows/governance.yml:28-29) uploads false alerts as "governance evidence" on every PR. Additionally scripts/ados.py:103 re-appends alert.triggered on every run — reproduced: 3 runs → 6 duplicate alert events in the canonical stream.runs_started == 0; emit alert.triggered only on state transition by diffing against observability/alerts/latest.json.approve --expires-at 2030-01-01 is accepted, then loop start dies with a raw TypeError: can't compare offset-naive and offset-aware datetimes at governance.py:74; (b) new ticket --title "Fix A/B flow" dies with FileNotFoundError from the unsanitized slug at scripts/ados.py:34; (c) status ticket --status Bananas is accepted (scripts/ados.py:57-65) even though gates only recognize Ready/In Progress (governance.py:102) and Complete/Cancelled (ado.config.json:5); (d) loop durations diff a persisted time.monotonic_ns() across processes (governance.py:118,179) — after a reboot the negative value makes telemetry.py:58-60 raise, so the loop can never stop.[a-z0-9-]; validate status against the real vocabulary; use wall-clock epoch (or clamp ≥0) for durations.src/agentic_os/validation.py:107-112 uses st_mtime; the scheduled audit job does a fresh checkout (.github/workflows/governance.yml:42-49), where every file's mtime is checkout time, so stale_artifacts is always empty. The feature only "works" in long-lived local clones.git log -1 --format=%ct -- <path> (subprocess git is already a pattern: governance.py:82-84).docs/prd/PRD-001-agentic-development-os.md:58).git log --name-only pass if needed.scripts/ados.py:152 — relevant=[issue for issue in issues if args.prd in issue or "bet" in issue or "outcome" in issue or "opportunity" in issue]. Any repo-wide validation issue mentioning "bet"/"outcome" blocks an unrelated PRD; a PRD-specific issue phrased without those words passes the gate.validate_product return structured issues (artifact ref + code), and gate on the named PRD's actual ancestry.validate_product's return type — keep a string view for existing callers/tests.src/agentic_os/telemetry.py:76-85 raises on the first bad line. Reproduced: appending one truncated line makes both ados.py metrics and ados.py validate fail (ERROR EVENT_INVALID: invalid event at line 7…). A killed process mid-append produces exactly this, and the only fix is hand-editing the evidence file the docs say never to hand-edit.ados.py events repair that quarantines bad lines to a sidecar and records the repair as an event; or a --skip-invalid read mode that reports a completeness warning.loop status action exists (scripts/ados.py:172 choices are plan/start/record-action/retry/verify/stop). The single-loop guard error says only "only one loop may be active in this checkout" (governance.py:115-116, reproduced) — not which ticket, run, or how to resolve it. ados.py --help lists 16 subcommands with zero descriptions (no help= anywhere in parser(), scripts/ados.py:167-185), and make help (Makefile:5-6) surfaces that same bare list as the primary onboarding aid.loop status (active ticket, run id, retries, verified flag, elapsed); include the active ticket in the guard error; add one-line help= per subcommand and a workflow epilog mirroring README's daily workflow.governance.py:158-160 captures stdout/stderr but stores only returncode; a failing loop verify prints just "failed" (scripts/ados.py:89), forcing a manual rerun to learn why.verification.completed metadata (through _sanitize).docs/metrics/catalog.md METRIC-004); giving the agent the failure reason is the cheapest way to reduce it.trace-query undersells the trace graph it owns — IMPROVE · Helpfulness / Synergiesscripts/ados.py:51-54 does exact value matching only. Reproduced: trace-query --id PRD-001 → [], exit 1, no hint — despite 18 related records in docs/trace/traceability.json. README:66 sells it as "forward/backward impact analysis". The 17-field records plus ci/impact-map.json already encode the full chain.--json exact-match behavior for scripts; add the rich view alongside.render_dashboard (src/agentic_os/metrics.py:79-86) hardcodes cards and formats every value as a bare 2-decimal float — the committed observability/reports/dashboard.html shows "Task Completion Rate 1.00" and "Lead Time Ms 1200.00". observability/dashboards/default.json (titles + thresholds per metric) is only existence-checked (validation.py:38) and never consumed; ado.config.json thresholds aren't used for display either; alerts render as metric: value without the breached rule; dark-only palette; no generated-at timestamp.default.json + config thresholds (green/amber/red status per card), format rates as %, cost as $, durations as s, show the rule in alerts, stamp generation time, add a prefers-color-scheme: light variant.docs/growth/operating-system.md:13); right now it reads as a debug page.telemetry.py:36-44), yet summarize (metrics.py:10-41) collapses all time into one snapshot; docs/metrics/latest.md is overwritten each run; scripts/metrics/aggregate.py just re-runs the snapshot. The adoption spec's own graduation gate — "First-pass success is stable for four weeks" (docs/specs/SPEC-002-adoption.md:21) — is unmeasurable with current reporting.observability/reports/history.json; add a trend column (and dashboard sparklines) for the key rates; alert on deterioration, not just absolute thresholds.ados.py new supports only prd/spec/ticket (scripts/ados.py:33), but the enforced chain demands outcome/opportunity/experiment/bet/milestone/CR/review files with exact section contracts validated in src/agentic_os/product.py:59-131 — and no TEMPLATE.md exists anywhere under .ai/ (full file inventory: templates exist only under docs/). The daily workflow also requires hand-editing a 17-field JSON record (README:47, docs/trace/traceability.json).ados.py new outcome|opportunity|experiment|bet|milestone|change-request|review generating section-complete stubs from the same contracts dict the validator uses, plus ados.py trace add that composes a record interactively with defaults from sibling records and validates before writing.product validate stops erroring." This converts the validator from a wall into a guide.tools/catalog.json defines 10 governed tools (validate_repository, create_ticket, start_loop, record_event, update_trace, archive_ticket, validate_product_chain, export_product_views, evaluate_product_gate, record_product_review) with descriptions, parameters, and server-side validation notes; src/agentic_os/tooling.py lint-enforces the naming standard — but nothing executes any of them, and record_product_review has no CLI equivalent at all. The declared primary users are AI-native teams (docs/prd/PRD-001-agentic-development-os.md:21) whose agents consume tools, not READMEs.scripts/ados_server.py) mapping catalog entries 1:1 onto the existing agentic_os functions, with a parity test (every catalog tool has a handler and vice versa). Risk-tier gating and digest approvals already exist to protect the write paths.check_scope/approval_valid, keep R2/R3 tools human-gated.approval.revoked, finding.recorded, human.intervened, product_review.completed, reversal.completed are all in EVENT_NAMES (telemetry.py:15-22) and the JSON schema, but no command produces them (only the generic low-level emit). reversal.completed is even consumed by the regression metric (metrics.py:23). Approvals are written with status: "active" (governance.py:60) and there is no revoke path; docs/findings/README.md asks for FINDING-NNN markdown files but no command creates or instruments them.ados.py approve --revoke, ados.py finding record (scaffolds the markdown + emits finding.recorded), ados.py review record (writes the review log + emits product_review.completed / milestone.reviewed).verification.completed meaning verification happened. A one-flag forgery, provable in three commands, is an existential credibility bug for a governance tool.loop status + named errors + subcommand help. The cheapest first-touch UX repair: the funnel in docs/growth/strategy.md is README → make demo → first ticket, and today the first stumble ("only one loop may be active…" — which one?) has no in-tool answer. Ship the trust fixes, then make the daily driver explain itself; the big bets (#13, #14) then have a foundation worth building on.Report only — no code was modified. Which of these items should I build?