docs: SENTRY_AS_PRODUCT.md — wire Sentry into every Vibn project
The Sentry wiring shipped today catches errors in the platform itself. The bigger product opportunity is wiring Sentry into every project Vibn ships, then feeding those errors back into the user's AI chat — turning the AI from "codes for you" into "owns the product." Proposal lays out 4 staged shippable chunks (~8 hr total): 1. Auto-provision per-project Sentry on first deploy 2. Bake Sentry into every scaffold template 3. Expose project_recent_errors / _detail / _resolve as MCP tools 4. Auto-surface unresolved errors at chat-turn start Folded into BETA_LAUNCH_PLAN.md as Phase 2.9. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -96,6 +96,7 @@ server: {
|
||||
| 2.6 | Bake `HEALTHCHECK 127.0.0.1` into `vibn-frontend/Dockerfile` so future apps inherit | AI | ✓ done 2026-05-01 | Already in `vibn-frontend/Dockerfile:67-68`; comment explains the IPv6 trap |
|
||||
| 2.7 | Audit other Dockerfile-based apps for the same `localhost`/IPv6 trap | AI | ✓ done 2026-05-01 | Audited `vibn-dev/Dockerfile` and `vibn-agent-runner/Dockerfile` — neither defines a HEALTHCHECK, so neither can hit the localhost/IPv6 trap. No action needed today; revisit when either gets a healthcheck added. |
|
||||
| 2.8 | **Tool-error recovery middleware** (AI_HARNESS_GAPS.md §1) — pattern-match known-recoverable tool errors and inject synthetic instructions before the model's next round | AI | ✓ done 2026-05-01 | `vibn-frontend/lib/ai/error-recovery.ts`. Initial rules: orphan container conflict, image pull denied, port allocated. Wired into `app/api/chat/route.ts` tool-result loop. |
|
||||
| 2.9 | **Sentry-as-product loop** (SENTRY_AS_PRODUCT.md) — auto-provision per-project Sentry, bake into scaffolds, expose error feed to AI as MCP tools, auto-surface unresolved errors at chat-turn start | AI | 8 hr | Highest-leverage item still ahead of beta. Turns AI from "codes for you" into "owns the product." Reuses today's Sentry org + tokens. See proposal doc for staged rollout (4 sub-stages, each independently shippable). |
|
||||
|
||||
**Definition of done:** force-fail a route in staging → Sentry alert lands in
|
||||
< 1 min. Force-fail a Coolify deploy → notification fires. Reproduce an
|
||||
|
||||
258
SENTRY_AS_PRODUCT.md
Normal file
258
SENTRY_AS_PRODUCT.md
Normal file
@@ -0,0 +1,258 @@
|
||||
# Sentry-as-Product — Proposal
|
||||
|
||||
> Today's Sentry wiring catches errors in **the Vibn platform**.
|
||||
> The bigger opportunity is wiring Sentry into **every project Vibn
|
||||
> ships**, then feeding those errors back into the user's AI chat.
|
||||
> Difference between "an AI that codes" and "an AI that owns the
|
||||
> product."
|
||||
|
||||
## TL;DR
|
||||
|
||||
Today, when a Vibn user's deployed app crashes for real users:
|
||||
|
||||
```
|
||||
real user → site 500s → user closes tab, never tells founder
|
||||
→ founder finds out hours/days later (or never)
|
||||
→ AI in Vibn chat has zero idea anything is wrong
|
||||
```
|
||||
|
||||
The fix is to make every Vibn project ship with Sentry pre-wired,
|
||||
then expose the error feed to the AI as a tool. Total effort:
|
||||
**~8 hours**, in 4 stages, each independently shippable.
|
||||
|
||||
| Stage | Capability | Effort | Unlocks |
|
||||
|---|---|---|---|
|
||||
| 1 | Auto-provision a Sentry project per Vibn project on first deploy | ~3 hr | Real-user errors captured at all |
|
||||
| 2 | Bake Sentry into every scaffold template | ~2 hr | Capture works without user setup |
|
||||
| 3 | Add `project_recent_errors` MCP tool for the AI | ~2 hr | AI can answer "is anything broken?" |
|
||||
| 4 | Auto-surface unresolved errors at chat-turn start | ~1 hr | AI proactively offers fixes |
|
||||
|
||||
Total: **~8 hr**, no new infra (we already have Sentry org access,
|
||||
Coolify env API, scaffold templates, MCP tool registry).
|
||||
|
||||
---
|
||||
|
||||
## Why this is the right next investment
|
||||
|
||||
### The current loop is broken at the seam between user and platform
|
||||
|
||||
Vibn's value proposition is "the AI is your technical co-founder."
|
||||
That promise breaks the moment the AI's last commit causes a real
|
||||
user error and the AI doesn't know about it. The current loop:
|
||||
|
||||
```
|
||||
1. User describes feature in chat
|
||||
2. AI ships code
|
||||
3. AI says "deployed, give it a try"
|
||||
4. (silence)
|
||||
5. Real users hit edge cases → 500s → bounce
|
||||
6. Founder eventually notices via support ticket / analytics dip
|
||||
7. Founder pastes error back to AI
|
||||
8. AI fixes
|
||||
```
|
||||
|
||||
Steps 4–6 are dead air for the founder, **and the AI cannot help
|
||||
during them.** This is the gap that separates Vibn from "any IDE
|
||||
with an LLM."
|
||||
|
||||
### What it looks like with this proposal shipped
|
||||
|
||||
```
|
||||
1. User describes feature in chat
|
||||
2. AI ships code
|
||||
3. AI says "deployed, give it a try"
|
||||
4. Real users hit edge cases → 500s → Sentry captures
|
||||
5. (Founder opens Vibn chat 3 hrs later for unrelated reason)
|
||||
6. AI: "Hey — checkout has 500'd for 3 users in the last hour
|
||||
because `customer.email` is undefined on
|
||||
app/checkout/route.ts:47. Want me to fix it?"
|
||||
7. AI fixes, deploys, marks issue resolved in Sentry
|
||||
```
|
||||
|
||||
The AI becomes the on-call engineer. This is what "technical
|
||||
co-founder" actually means and we are 8 hours away from it.
|
||||
|
||||
### Why now (not Phase 4)
|
||||
|
||||
- The Sentry wiring we just shipped for vibn-frontend gave us:
|
||||
- A working Sentry org (`vibnai`)
|
||||
- An auth token with project-management scope
|
||||
- Verified knowledge that the build args / source maps flow works
|
||||
- A working `withSentryConfig` recipe in `vibn-frontend/next.config.ts`
|
||||
- All of those are reusable for stage 1 and 2 of this proposal.
|
||||
- Doing this **before** the beta means user projects start emitting
|
||||
error data on day one, so by the time we're debugging real beta
|
||||
user pain, we have a month of history to reason about.
|
||||
- Doing it after the beta means we'd have to retroactively
|
||||
instrument projects that have already been deployed for weeks.
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 — Auto-provision a Sentry project per Vibn project (~3 hr)
|
||||
|
||||
**Goal:** when a user creates a Vibn project, the platform creates a
|
||||
matching Sentry project under the `vibnai` org and stashes the DSN
|
||||
+ auth token in Coolify env vars on the user's app.
|
||||
|
||||
**What gets built:**
|
||||
|
||||
1. **A `provisionSentryProject(projectId, name)` helper** in
|
||||
`vibn-frontend/lib/integrations/sentry.ts`. Calls Sentry's
|
||||
`POST /api/0/teams/vibnai/{team}/projects/` with the project
|
||||
slug, returns the DSN.
|
||||
2. **Hook into project-create flow** — on first successful deploy,
|
||||
call the helper and write the resulting DSN + auth token into
|
||||
Coolify env vars (`NEXT_PUBLIC_SENTRY_DSN`,
|
||||
`SENTRY_AUTH_TOKEN`) for that app via the same Coolify API we
|
||||
used today.
|
||||
3. **Idempotency** — if the Sentry project already exists, fetch
|
||||
its DSN instead of creating a duplicate. Same project name
|
||||
convention every time: `vibn-{workspace}-{projectSlug}`.
|
||||
4. **Storage** — store `sentryProjectSlug` and `sentryAuthTokenId`
|
||||
on the Postgres `projects` row so we can look them up later
|
||||
without re-walking the Sentry org.
|
||||
|
||||
**Risk:** Sentry's API rate-limits team-project creation. We bypass
|
||||
this by reading-before-writing, so the only API cost on subsequent
|
||||
deploys is one GET.
|
||||
|
||||
**Definition of done:** create a fresh Vibn project → check Sentry
|
||||
org → see a project named `vibn-{ws}-{slug}` → check Coolify env on
|
||||
that app → see DSN populated.
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 — Bake Sentry into every scaffold template (~2 hr)
|
||||
|
||||
**Goal:** every Next.js / Vite / etc. starter template Vibn ships
|
||||
already has Sentry wired up. User does nothing.
|
||||
|
||||
**What gets built:**
|
||||
|
||||
1. **For each scaffold template in `vibn-frontend/lib/scaffold/`**,
|
||||
add the same files we shipped today:
|
||||
- `instrumentation.ts`
|
||||
- `instrumentation-client.ts`
|
||||
- `app/global-error.tsx` (Next.js) / equivalent boundary (Vite)
|
||||
- `next.config.ts` wrapped with `withSentryConfig` (Next.js)
|
||||
- `vite.config.ts` with `sentryVitePlugin` (Vite)
|
||||
- `Dockerfile` ARG declarations for `NEXT_PUBLIC_SENTRY_DSN` +
|
||||
`SENTRY_AUTH_TOKEN`
|
||||
2. **Add `@sentry/nextjs` (or `@sentry/react` + `@sentry/vite-plugin`)
|
||||
to each template's `package.json` `dependencies`.**
|
||||
3. **Document in template README** that Sentry is pre-wired and the
|
||||
user doesn't need to do anything.
|
||||
|
||||
**Risk:** Sentry's wrapper sometimes interacts badly with custom
|
||||
build configs (e.g. monorepos, custom webpack rules). Mitigation:
|
||||
the `errorHandler` we set today (`console.warn` instead of throw)
|
||||
ensures source map upload failures don't break builds.
|
||||
|
||||
**Definition of done:** scaffold a fresh Next.js project from Vibn
|
||||
templates → deploy → throw a test error → see it in Sentry,
|
||||
de-minified.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3 — Expose error feed to the AI as MCP tools (~2 hr)
|
||||
|
||||
**Goal:** the AI can ask Sentry "what's broken in project X?" and
|
||||
get a real answer.
|
||||
|
||||
**What gets built:**
|
||||
|
||||
Three new MCP tools in `vibn-frontend/lib/ai/vibn-tools.ts`:
|
||||
|
||||
1. **`project_recent_errors { projectId, since?, limit? }`**
|
||||
- Returns: `[{ id, title, count, lastSeen, culprit, level }]`
|
||||
- Default `since`: 24h. Default `limit`: 10.
|
||||
- Filters to unresolved issues only.
|
||||
- Implementation: read `sentryProjectSlug` off the project row,
|
||||
call Sentry's `GET /api/0/projects/{org}/{slug}/issues/`.
|
||||
|
||||
2. **`project_error_detail { projectId, issueId }`**
|
||||
- Returns: `{ stacktrace, breadcrumbs, request, user, replay_url }`
|
||||
- Implementation: Sentry's `GET /api/0/issues/{id}/events/latest/`.
|
||||
|
||||
3. **`project_error_resolve { projectId, issueId }`**
|
||||
- Side-effect: marks the issue resolved in Sentry.
|
||||
- Used by the AI after it ships a fix and confirms via tests.
|
||||
- Implementation: Sentry's `PUT /api/0/issues/{id}/` with
|
||||
`status: "resolved"`.
|
||||
|
||||
**Auth:** token storage is per-project (from Stage 1's `projects`
|
||||
row). Each project's AI sees only its own project's errors. No
|
||||
cross-project leakage.
|
||||
|
||||
**Definition of done:** in a Vibn chat for a project with known
|
||||
errors, ask the AI "any errors lately?" → AI calls
|
||||
`project_recent_errors` → shows real list.
|
||||
|
||||
---
|
||||
|
||||
## Stage 4 — Auto-surface unresolved errors at chat-turn start (~1 hr)
|
||||
|
||||
**Goal:** the AI doesn't wait to be asked. When the user opens a
|
||||
chat and there are unresolved errors, the AI mentions them on the
|
||||
first turn.
|
||||
|
||||
**What gets built:**
|
||||
|
||||
In `vibn-frontend/app/api/chat/route.ts`, at the start of each chat
|
||||
turn (before calling the model):
|
||||
|
||||
1. Call the same `project_recent_errors` logic Stage 3 exposed.
|
||||
2. If `count > 0`, prepend a synthetic system message:
|
||||
|
||||
```
|
||||
[PROJECT HEALTH]
|
||||
{N} unresolved Sentry issues in the last 24 hours:
|
||||
- {title} (×{count}, last seen {time}) — {culprit}
|
||||
- ...
|
||||
|
||||
If the user's first message is unrelated to these, you may still
|
||||
proactively mention them: "Quick FYI before we get into that —
|
||||
{X} has been failing for users."
|
||||
|
||||
If their message IS about a broken thing, prefer the matching
|
||||
Sentry issue's stack trace over guessing.
|
||||
```
|
||||
|
||||
3. Only fire this once per N chat turns (configurable, default 1
|
||||
per session opening) — we don't want to spam every turn.
|
||||
|
||||
**Risk:** false alarms (Sentry issue from yesterday's deploy that
|
||||
no one cares about anymore) make the AI annoying. Mitigation:
|
||||
tighten the `since` window to the last 6h, and only surface issues
|
||||
with `count >= 2` (one-off errors don't count).
|
||||
|
||||
**Definition of done:** intentionally break a deployed user
|
||||
project, open chat, type "what's up?" → AI's first response
|
||||
mentions the issue, with file path.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope for this proposal
|
||||
|
||||
- **User-owned Sentry orgs.** Some users will eventually want their
|
||||
own Sentry account, not the shared `vibnai` org. Ship-later;
|
||||
doesn't block the loop. Easy retrofit because storage is already
|
||||
per-project.
|
||||
- **Performance / Tracing data.** Sentry also captures spans /
|
||||
traces. Useful for "this endpoint is slow" but not the urgent
|
||||
product loop. Ship-later.
|
||||
- **Front-end UI for errors in Vibn.** A "Health" tab showing the
|
||||
Sentry feed in the Vibn UI is nice but not required for the AI
|
||||
loop to work. Ship-later.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
Add a **Phase 2.9 (Sentry-as-product loop)** to `BETA_LAUNCH_PLAN.md`
|
||||
covering Stages 1–4 as a single bundle. Estimate: **8 hr engineering**.
|
||||
|
||||
This is the second-highest-leverage item still ahead of beta,
|
||||
behind only the deploy-failed webhook (which is 30 min). Every
|
||||
hour spent here directly upgrades the value of every other beta
|
||||
test session that follows it.
|
||||
Reference in New Issue
Block a user