259 lines
9.6 KiB
Markdown
259 lines
9.6 KiB
Markdown
# Sentry-as-Product — Proposal
|
||
|
||
> Today's Sentry wiring catches errors in **the Vibn platform**.
|
||
> The bigger opportunity is wiring Sentry into **every project Vibn
|
||
> ships**, then feeding those errors back into the user's AI chat.
|
||
> Difference between "an AI that codes" and "an AI that owns the
|
||
> product."
|
||
|
||
## TL;DR
|
||
|
||
Today, when a Vibn user's deployed app crashes for real users:
|
||
|
||
```
|
||
real user → site 500s → user closes tab, never tells founder
|
||
→ founder finds out hours/days later (or never)
|
||
→ AI in Vibn chat has zero idea anything is wrong
|
||
```
|
||
|
||
The fix is to make every Vibn project ship with Sentry pre-wired,
|
||
then expose the error feed to the AI as a tool. Total effort:
|
||
**~8 hours**, in 4 stages, each independently shippable.
|
||
|
||
| Stage | Capability | Effort | Unlocks |
|
||
|---|---|---|---|
|
||
| 1 | Auto-provision a Sentry project per Vibn project on first deploy | ~3 hr | Real-user errors captured at all |
|
||
| 2 | Bake Sentry into every scaffold template | ~2 hr | Capture works without user setup |
|
||
| 3 | Add `project_recent_errors` MCP tool for the AI | ~2 hr | AI can answer "is anything broken?" |
|
||
| 4 | Auto-surface unresolved errors at chat-turn start | ~1 hr | AI proactively offers fixes |
|
||
|
||
Total: **~8 hr**, no new infra (we already have Sentry org access,
|
||
Coolify env API, scaffold templates, MCP tool registry).
|
||
|
||
---
|
||
|
||
## Why this is the right next investment
|
||
|
||
### The current loop is broken at the seam between user and platform
|
||
|
||
Vibn's value proposition is "the AI is your technical co-founder."
|
||
That promise breaks the moment the AI's last commit causes a real
|
||
user error and the AI doesn't know about it. The current loop:
|
||
|
||
```
|
||
1. User describes feature in chat
|
||
2. AI ships code
|
||
3. AI says "deployed, give it a try"
|
||
4. (silence)
|
||
5. Real users hit edge cases → 500s → bounce
|
||
6. Founder eventually notices via support ticket / analytics dip
|
||
7. Founder pastes error back to AI
|
||
8. AI fixes
|
||
```
|
||
|
||
Steps 4–6 are dead air for the founder, **and the AI cannot help
|
||
during them.** This is the gap that separates Vibn from "any IDE
|
||
with an LLM."
|
||
|
||
### What it looks like with this proposal shipped
|
||
|
||
```
|
||
1. User describes feature in chat
|
||
2. AI ships code
|
||
3. AI says "deployed, give it a try"
|
||
4. Real users hit edge cases → 500s → Sentry captures
|
||
5. (Founder opens Vibn chat 3 hrs later for unrelated reason)
|
||
6. AI: "Hey — checkout has 500'd for 3 users in the last hour
|
||
because `customer.email` is undefined on
|
||
app/checkout/route.ts:47. Want me to fix it?"
|
||
7. AI fixes, deploys, marks issue resolved in Sentry
|
||
```
|
||
|
||
The AI becomes the on-call engineer. This is what "technical
|
||
co-founder" actually means and we are 8 hours away from it.
|
||
|
||
### Why now (not Phase 4)
|
||
|
||
- The Sentry wiring we just shipped for vibn-frontend gave us:
|
||
- A working Sentry org (`vibnai`)
|
||
- An auth token with project-management scope
|
||
- Verified knowledge that the build args / source maps flow works
|
||
- A working `withSentryConfig` recipe in `vibn-frontend/next.config.ts`
|
||
- All of those are reusable for stage 1 and 2 of this proposal.
|
||
- Doing this **before** the beta means user projects start emitting
|
||
error data on day one, so by the time we're debugging real beta
|
||
user pain, we have a month of history to reason about.
|
||
- Doing it after the beta means we'd have to retroactively
|
||
instrument projects that have already been deployed for weeks.
|
||
|
||
---
|
||
|
||
## Stage 1 — Auto-provision a Sentry project per Vibn project (~3 hr)
|
||
|
||
**Goal:** when a user creates a Vibn project, the platform creates a
|
||
matching Sentry project under the `vibnai` org and stashes the DSN
|
||
+ auth token in Coolify env vars on the user's app.
|
||
|
||
**What gets built:**
|
||
|
||
1. **A `provisionSentryProject(projectId, name)` helper** in
|
||
`vibn-frontend/lib/integrations/sentry.ts`. Calls Sentry's
|
||
`POST /api/0/teams/vibnai/{team}/projects/` with the project
|
||
slug, returns the DSN.
|
||
2. **Hook into project-create flow** — on first successful deploy,
|
||
call the helper and write the resulting DSN + auth token into
|
||
Coolify env vars (`NEXT_PUBLIC_SENTRY_DSN`,
|
||
`SENTRY_AUTH_TOKEN`) for that app via the same Coolify API we
|
||
used today.
|
||
3. **Idempotency** — if the Sentry project already exists, fetch
|
||
its DSN instead of creating a duplicate. Same project name
|
||
convention every time: `vibn-{workspace}-{projectSlug}`.
|
||
4. **Storage** — store `sentryProjectSlug` and `sentryAuthTokenId`
|
||
on the Postgres `projects` row so we can look them up later
|
||
without re-walking the Sentry org.
|
||
|
||
**Risk:** Sentry's API rate-limits team-project creation. We bypass
|
||
this by reading-before-writing, so the only API cost on subsequent
|
||
deploys is one GET.
|
||
|
||
**Definition of done:** create a fresh Vibn project → check Sentry
|
||
org → see a project named `vibn-{ws}-{slug}` → check Coolify env on
|
||
that app → see DSN populated.
|
||
|
||
---
|
||
|
||
## Stage 2 — Bake Sentry into every scaffold template (~2 hr)
|
||
|
||
**Goal:** every Next.js / Vite / etc. starter template Vibn ships
|
||
already has Sentry wired up. User does nothing.
|
||
|
||
**What gets built:**
|
||
|
||
1. **For each scaffold template in `vibn-frontend/lib/scaffold/`**,
|
||
add the same files we shipped today:
|
||
- `instrumentation.ts`
|
||
- `instrumentation-client.ts`
|
||
- `app/global-error.tsx` (Next.js) / equivalent boundary (Vite)
|
||
- `next.config.ts` wrapped with `withSentryConfig` (Next.js)
|
||
- `vite.config.ts` with `sentryVitePlugin` (Vite)
|
||
- `Dockerfile` ARG declarations for `NEXT_PUBLIC_SENTRY_DSN` +
|
||
`SENTRY_AUTH_TOKEN`
|
||
2. **Add `@sentry/nextjs` (or `@sentry/react` + `@sentry/vite-plugin`)
|
||
to each template's `package.json` `dependencies`.**
|
||
3. **Document in template README** that Sentry is pre-wired and the
|
||
user doesn't need to do anything.
|
||
|
||
**Risk:** Sentry's wrapper sometimes interacts badly with custom
|
||
build configs (e.g. monorepos, custom webpack rules). Mitigation:
|
||
the `errorHandler` we set today (`console.warn` instead of throw)
|
||
ensures source map upload failures don't break builds.
|
||
|
||
**Definition of done:** scaffold a fresh Next.js project from Vibn
|
||
templates → deploy → throw a test error → see it in Sentry,
|
||
de-minified.
|
||
|
||
---
|
||
|
||
## Stage 3 — Expose error feed to the AI as MCP tools (~2 hr)
|
||
|
||
**Goal:** the AI can ask Sentry "what's broken in project X?" and
|
||
get a real answer.
|
||
|
||
**What gets built:**
|
||
|
||
Three new MCP tools in `vibn-frontend/lib/ai/vibn-tools.ts`:
|
||
|
||
1. **`project_recent_errors { projectId, since?, limit? }`**
|
||
- Returns: `[{ id, title, count, lastSeen, culprit, level }]`
|
||
- Default `since`: 24h. Default `limit`: 10.
|
||
- Filters to unresolved issues only.
|
||
- Implementation: read `sentryProjectSlug` off the project row,
|
||
call Sentry's `GET /api/0/projects/{org}/{slug}/issues/`.
|
||
|
||
2. **`project_error_detail { projectId, issueId }`**
|
||
- Returns: `{ stacktrace, breadcrumbs, request, user, replay_url }`
|
||
- Implementation: Sentry's `GET /api/0/issues/{id}/events/latest/`.
|
||
|
||
3. **`project_error_resolve { projectId, issueId }`**
|
||
- Side-effect: marks the issue resolved in Sentry.
|
||
- Used by the AI after it ships a fix and confirms via tests.
|
||
- Implementation: Sentry's `PUT /api/0/issues/{id}/` with
|
||
`status: "resolved"`.
|
||
|
||
**Auth:** token storage is per-project (from Stage 1's `projects`
|
||
row). Each project's AI sees only its own project's errors. No
|
||
cross-project leakage.
|
||
|
||
**Definition of done:** in a Vibn chat for a project with known
|
||
errors, ask the AI "any errors lately?" → AI calls
|
||
`project_recent_errors` → shows real list.
|
||
|
||
---
|
||
|
||
## Stage 4 — Auto-surface unresolved errors at chat-turn start (~1 hr)
|
||
|
||
**Goal:** the AI doesn't wait to be asked. When the user opens a
|
||
chat and there are unresolved errors, the AI mentions them on the
|
||
first turn.
|
||
|
||
**What gets built:**
|
||
|
||
In `vibn-frontend/app/api/chat/route.ts`, at the start of each chat
|
||
turn (before calling the model):
|
||
|
||
1. Call the same `project_recent_errors` logic Stage 3 exposed.
|
||
2. If `count > 0`, prepend a synthetic system message:
|
||
|
||
```
|
||
[PROJECT HEALTH]
|
||
{N} unresolved Sentry issues in the last 24 hours:
|
||
- {title} (×{count}, last seen {time}) — {culprit}
|
||
- ...
|
||
|
||
If the user's first message is unrelated to these, you may still
|
||
proactively mention them: "Quick FYI before we get into that —
|
||
{X} has been failing for users."
|
||
|
||
If their message IS about a broken thing, prefer the matching
|
||
Sentry issue's stack trace over guessing.
|
||
```
|
||
|
||
3. Only fire this once per N chat turns (configurable, default 1
|
||
per session opening) — we don't want to spam every turn.
|
||
|
||
**Risk:** false alarms (Sentry issue from yesterday's deploy that
|
||
no one cares about anymore) make the AI annoying. Mitigation:
|
||
tighten the `since` window to the last 6h, and only surface issues
|
||
with `count >= 2` (one-off errors don't count).
|
||
|
||
**Definition of done:** intentionally break a deployed user
|
||
project, open chat, type "what's up?" → AI's first response
|
||
mentions the issue, with file path.
|
||
|
||
---
|
||
|
||
## Out of scope for this proposal
|
||
|
||
- **User-owned Sentry orgs.** Some users will eventually want their
|
||
own Sentry account, not the shared `vibnai` org. Ship-later;
|
||
doesn't block the loop. Easy retrofit because storage is already
|
||
per-project.
|
||
- **Performance / Tracing data.** Sentry also captures spans /
|
||
traces. Useful for "this endpoint is slow" but not the urgent
|
||
product loop. Ship-later.
|
||
- **Front-end UI for errors in Vibn.** A "Health" tab showing the
|
||
Sentry feed in the Vibn UI is nice but not required for the AI
|
||
loop to work. Ship-later.
|
||
|
||
---
|
||
|
||
## Recommendation
|
||
|
||
Add a **Phase 2.9 (Sentry-as-product loop)** to `BETA_LAUNCH_PLAN.md`
|
||
covering Stages 1–4 as a single bundle. Estimate: **8 hr engineering**.
|
||
|
||
This is the second-highest-leverage item still ahead of beta,
|
||
behind only the deploy-failed webhook (which is 30 min). Every
|
||
hour spent here directly upgrades the value of every other beta
|
||
test session that follows it.
|