diff --git a/BETA_LAUNCH_PLAN.md b/BETA_LAUNCH_PLAN.md index 8f927b2..b7d05d0 100644 --- a/BETA_LAUNCH_PLAN.md +++ b/BETA_LAUNCH_PLAN.md @@ -96,6 +96,7 @@ server: { | 2.6 | Bake `HEALTHCHECK 127.0.0.1` into `vibn-frontend/Dockerfile` so future apps inherit | AI | ✓ done 2026-05-01 | Already in `vibn-frontend/Dockerfile:67-68`; comment explains the IPv6 trap | | 2.7 | Audit other Dockerfile-based apps for the same `localhost`/IPv6 trap | AI | ✓ done 2026-05-01 | Audited `vibn-dev/Dockerfile` and `vibn-agent-runner/Dockerfile` — neither defines a HEALTHCHECK, so neither can hit the localhost/IPv6 trap. No action needed today; revisit when either gets a healthcheck added. | | 2.8 | **Tool-error recovery middleware** (AI_HARNESS_GAPS.md §1) — pattern-match known-recoverable tool errors and inject synthetic instructions before the model's next round | AI | ✓ done 2026-05-01 | `vibn-frontend/lib/ai/error-recovery.ts`. Initial rules: orphan container conflict, image pull denied, port allocated. Wired into `app/api/chat/route.ts` tool-result loop. | +| 2.9 | **Sentry-as-product loop** (SENTRY_AS_PRODUCT.md) — auto-provision per-project Sentry, bake into scaffolds, expose error feed to AI as MCP tools, auto-surface unresolved errors at chat-turn start | AI | 8 hr | Highest-leverage item still ahead of beta. Turns AI from "codes for you" into "owns the product." Reuses today's Sentry org + tokens. See proposal doc for staged rollout (4 sub-stages, each independently shippable). | **Definition of done:** force-fail a route in staging → Sentry alert lands in < 1 min. Force-fail a Coolify deploy → notification fires. Reproduce an diff --git a/SENTRY_AS_PRODUCT.md b/SENTRY_AS_PRODUCT.md new file mode 100644 index 0000000..dcc8588 --- /dev/null +++ b/SENTRY_AS_PRODUCT.md @@ -0,0 +1,258 @@ +# Sentry-as-Product — Proposal + +> Today's Sentry wiring catches errors in **the Vibn platform**. +> The bigger opportunity is wiring Sentry into **every project Vibn +> ships**, then feeding those errors back into the user's AI chat. +> Difference between "an AI that codes" and "an AI that owns the +> product." + +## TL;DR + +Today, when a Vibn user's deployed app crashes for real users: + +``` +real user → site 500s → user closes tab, never tells founder + → founder finds out hours/days later (or never) + → AI in Vibn chat has zero idea anything is wrong +``` + +The fix is to make every Vibn project ship with Sentry pre-wired, +then expose the error feed to the AI as a tool. Total effort: +**~8 hours**, in 4 stages, each independently shippable. + +| Stage | Capability | Effort | Unlocks | +|---|---|---|---| +| 1 | Auto-provision a Sentry project per Vibn project on first deploy | ~3 hr | Real-user errors captured at all | +| 2 | Bake Sentry into every scaffold template | ~2 hr | Capture works without user setup | +| 3 | Add `project_recent_errors` MCP tool for the AI | ~2 hr | AI can answer "is anything broken?" | +| 4 | Auto-surface unresolved errors at chat-turn start | ~1 hr | AI proactively offers fixes | + +Total: **~8 hr**, no new infra (we already have Sentry org access, +Coolify env API, scaffold templates, MCP tool registry). + +--- + +## Why this is the right next investment + +### The current loop is broken at the seam between user and platform + +Vibn's value proposition is "the AI is your technical co-founder." +That promise breaks the moment the AI's last commit causes a real +user error and the AI doesn't know about it. The current loop: + +``` +1. User describes feature in chat +2. AI ships code +3. AI says "deployed, give it a try" +4. (silence) +5. Real users hit edge cases → 500s → bounce +6. Founder eventually notices via support ticket / analytics dip +7. Founder pastes error back to AI +8. AI fixes +``` + +Steps 4–6 are dead air for the founder, **and the AI cannot help +during them.** This is the gap that separates Vibn from "any IDE +with an LLM." + +### What it looks like with this proposal shipped + +``` +1. User describes feature in chat +2. AI ships code +3. AI says "deployed, give it a try" +4. Real users hit edge cases → 500s → Sentry captures +5. (Founder opens Vibn chat 3 hrs later for unrelated reason) +6. AI: "Hey — checkout has 500'd for 3 users in the last hour + because `customer.email` is undefined on + app/checkout/route.ts:47. Want me to fix it?" +7. AI fixes, deploys, marks issue resolved in Sentry +``` + +The AI becomes the on-call engineer. This is what "technical +co-founder" actually means and we are 8 hours away from it. + +### Why now (not Phase 4) + +- The Sentry wiring we just shipped for vibn-frontend gave us: + - A working Sentry org (`vibnai`) + - An auth token with project-management scope + - Verified knowledge that the build args / source maps flow works + - A working `withSentryConfig` recipe in `vibn-frontend/next.config.ts` +- All of those are reusable for stage 1 and 2 of this proposal. +- Doing this **before** the beta means user projects start emitting + error data on day one, so by the time we're debugging real beta + user pain, we have a month of history to reason about. +- Doing it after the beta means we'd have to retroactively + instrument projects that have already been deployed for weeks. + +--- + +## Stage 1 — Auto-provision a Sentry project per Vibn project (~3 hr) + +**Goal:** when a user creates a Vibn project, the platform creates a +matching Sentry project under the `vibnai` org and stashes the DSN ++ auth token in Coolify env vars on the user's app. + +**What gets built:** + +1. **A `provisionSentryProject(projectId, name)` helper** in + `vibn-frontend/lib/integrations/sentry.ts`. Calls Sentry's + `POST /api/0/teams/vibnai/{team}/projects/` with the project + slug, returns the DSN. +2. **Hook into project-create flow** — on first successful deploy, + call the helper and write the resulting DSN + auth token into + Coolify env vars (`NEXT_PUBLIC_SENTRY_DSN`, + `SENTRY_AUTH_TOKEN`) for that app via the same Coolify API we + used today. +3. **Idempotency** — if the Sentry project already exists, fetch + its DSN instead of creating a duplicate. Same project name + convention every time: `vibn-{workspace}-{projectSlug}`. +4. **Storage** — store `sentryProjectSlug` and `sentryAuthTokenId` + on the Postgres `projects` row so we can look them up later + without re-walking the Sentry org. + +**Risk:** Sentry's API rate-limits team-project creation. We bypass +this by reading-before-writing, so the only API cost on subsequent +deploys is one GET. + +**Definition of done:** create a fresh Vibn project → check Sentry +org → see a project named `vibn-{ws}-{slug}` → check Coolify env on +that app → see DSN populated. + +--- + +## Stage 2 — Bake Sentry into every scaffold template (~2 hr) + +**Goal:** every Next.js / Vite / etc. starter template Vibn ships +already has Sentry wired up. User does nothing. + +**What gets built:** + +1. **For each scaffold template in `vibn-frontend/lib/scaffold/`**, + add the same files we shipped today: + - `instrumentation.ts` + - `instrumentation-client.ts` + - `app/global-error.tsx` (Next.js) / equivalent boundary (Vite) + - `next.config.ts` wrapped with `withSentryConfig` (Next.js) + - `vite.config.ts` with `sentryVitePlugin` (Vite) + - `Dockerfile` ARG declarations for `NEXT_PUBLIC_SENTRY_DSN` + + `SENTRY_AUTH_TOKEN` +2. **Add `@sentry/nextjs` (or `@sentry/react` + `@sentry/vite-plugin`) + to each template's `package.json` `dependencies`.** +3. **Document in template README** that Sentry is pre-wired and the + user doesn't need to do anything. + +**Risk:** Sentry's wrapper sometimes interacts badly with custom +build configs (e.g. monorepos, custom webpack rules). Mitigation: +the `errorHandler` we set today (`console.warn` instead of throw) +ensures source map upload failures don't break builds. + +**Definition of done:** scaffold a fresh Next.js project from Vibn +templates → deploy → throw a test error → see it in Sentry, +de-minified. + +--- + +## Stage 3 — Expose error feed to the AI as MCP tools (~2 hr) + +**Goal:** the AI can ask Sentry "what's broken in project X?" and +get a real answer. + +**What gets built:** + +Three new MCP tools in `vibn-frontend/lib/ai/vibn-tools.ts`: + +1. **`project_recent_errors { projectId, since?, limit? }`** + - Returns: `[{ id, title, count, lastSeen, culprit, level }]` + - Default `since`: 24h. Default `limit`: 10. + - Filters to unresolved issues only. + - Implementation: read `sentryProjectSlug` off the project row, + call Sentry's `GET /api/0/projects/{org}/{slug}/issues/`. + +2. **`project_error_detail { projectId, issueId }`** + - Returns: `{ stacktrace, breadcrumbs, request, user, replay_url }` + - Implementation: Sentry's `GET /api/0/issues/{id}/events/latest/`. + +3. **`project_error_resolve { projectId, issueId }`** + - Side-effect: marks the issue resolved in Sentry. + - Used by the AI after it ships a fix and confirms via tests. + - Implementation: Sentry's `PUT /api/0/issues/{id}/` with + `status: "resolved"`. + +**Auth:** token storage is per-project (from Stage 1's `projects` +row). Each project's AI sees only its own project's errors. No +cross-project leakage. + +**Definition of done:** in a Vibn chat for a project with known +errors, ask the AI "any errors lately?" → AI calls +`project_recent_errors` → shows real list. + +--- + +## Stage 4 — Auto-surface unresolved errors at chat-turn start (~1 hr) + +**Goal:** the AI doesn't wait to be asked. When the user opens a +chat and there are unresolved errors, the AI mentions them on the +first turn. + +**What gets built:** + +In `vibn-frontend/app/api/chat/route.ts`, at the start of each chat +turn (before calling the model): + +1. Call the same `project_recent_errors` logic Stage 3 exposed. +2. If `count > 0`, prepend a synthetic system message: + +``` +[PROJECT HEALTH] +{N} unresolved Sentry issues in the last 24 hours: +- {title} (×{count}, last seen {time}) — {culprit} +- ... + +If the user's first message is unrelated to these, you may still +proactively mention them: "Quick FYI before we get into that — +{X} has been failing for users." + +If their message IS about a broken thing, prefer the matching +Sentry issue's stack trace over guessing. +``` + +3. Only fire this once per N chat turns (configurable, default 1 + per session opening) — we don't want to spam every turn. + +**Risk:** false alarms (Sentry issue from yesterday's deploy that +no one cares about anymore) make the AI annoying. Mitigation: +tighten the `since` window to the last 6h, and only surface issues +with `count >= 2` (one-off errors don't count). + +**Definition of done:** intentionally break a deployed user +project, open chat, type "what's up?" → AI's first response +mentions the issue, with file path. + +--- + +## Out of scope for this proposal + +- **User-owned Sentry orgs.** Some users will eventually want their + own Sentry account, not the shared `vibnai` org. Ship-later; + doesn't block the loop. Easy retrofit because storage is already + per-project. +- **Performance / Tracing data.** Sentry also captures spans / + traces. Useful for "this endpoint is slow" but not the urgent + product loop. Ship-later. +- **Front-end UI for errors in Vibn.** A "Health" tab showing the + Sentry feed in the Vibn UI is nice but not required for the AI + loop to work. Ship-later. + +--- + +## Recommendation + +Add a **Phase 2.9 (Sentry-as-product loop)** to `BETA_LAUNCH_PLAN.md` +covering Stages 1–4 as a single bundle. Estimate: **8 hr engineering**. + +This is the second-highest-leverage item still ahead of beta, +behind only the deploy-failed webhook (which is 30 min). Every +hour spent here directly upgrades the value of every other beta +test session that follows it.