docs: heavily compress and simplify remaining reference files to represent current state
This commit is contained in:
@@ -1,258 +1,9 @@
|
||||
# Sentry-as-Product — Proposal
|
||||
# Sentry as a Product (Shipped)
|
||||
|
||||
> Today's Sentry wiring catches errors in **the Vibn platform**.
|
||||
> The bigger opportunity is wiring Sentry into **every project Vibn
|
||||
> ships**, then feeding those errors back into the user's AI chat.
|
||||
> Difference between "an AI that codes" and "an AI that owns the
|
||||
> product."
|
||||
> **Note:** This spec was implemented in May 2026.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Today, when a Vibn user's deployed app crashes for real users:
|
||||
|
||||
```
|
||||
real user → site 500s → user closes tab, never tells founder
|
||||
→ founder finds out hours/days later (or never)
|
||||
→ AI in Vibn chat has zero idea anything is wrong
|
||||
```
|
||||
|
||||
The fix is to make every Vibn project ship with Sentry pre-wired,
|
||||
then expose the error feed to the AI as a tool. Total effort:
|
||||
**~8 hours**, in 4 stages, each independently shippable.
|
||||
|
||||
| Stage | Capability | Effort | Unlocks |
|
||||
|---|---|---|---|
|
||||
| 1 | Auto-provision a Sentry project per Vibn project on first deploy | ~3 hr | Real-user errors captured at all |
|
||||
| 2 | Bake Sentry into every scaffold template | ~2 hr | Capture works without user setup |
|
||||
| 3 | Add `project_recent_errors` MCP tool for the AI | ~2 hr | AI can answer "is anything broken?" |
|
||||
| 4 | Auto-surface unresolved errors at chat-turn start | ~1 hr | AI proactively offers fixes |
|
||||
|
||||
Total: **~8 hr**, no new infra (we already have Sentry org access,
|
||||
Coolify env API, scaffold templates, MCP tool registry).
|
||||
|
||||
---
|
||||
|
||||
## Why this is the right next investment
|
||||
|
||||
### The current loop is broken at the seam between user and platform
|
||||
|
||||
Vibn's value proposition is "the AI is your technical co-founder."
|
||||
That promise breaks the moment the AI's last commit causes a real
|
||||
user error and the AI doesn't know about it. The current loop:
|
||||
|
||||
```
|
||||
1. User describes feature in chat
|
||||
2. AI ships code
|
||||
3. AI says "deployed, give it a try"
|
||||
4. (silence)
|
||||
5. Real users hit edge cases → 500s → bounce
|
||||
6. Founder eventually notices via support ticket / analytics dip
|
||||
7. Founder pastes error back to AI
|
||||
8. AI fixes
|
||||
```
|
||||
|
||||
Steps 4–6 are dead air for the founder, **and the AI cannot help
|
||||
during them.** This is the gap that separates Vibn from "any IDE
|
||||
with an LLM."
|
||||
|
||||
### What it looks like with this proposal shipped
|
||||
|
||||
```
|
||||
1. User describes feature in chat
|
||||
2. AI ships code
|
||||
3. AI says "deployed, give it a try"
|
||||
4. Real users hit edge cases → 500s → Sentry captures
|
||||
5. (Founder opens Vibn chat 3 hrs later for unrelated reason)
|
||||
6. AI: "Hey — checkout has 500'd for 3 users in the last hour
|
||||
because `customer.email` is undefined on
|
||||
app/checkout/route.ts:47. Want me to fix it?"
|
||||
7. AI fixes, deploys, marks issue resolved in Sentry
|
||||
```
|
||||
|
||||
The AI becomes the on-call engineer. This is what "technical
|
||||
co-founder" actually means and we are 8 hours away from it.
|
||||
|
||||
### Why now (not Phase 4)
|
||||
|
||||
- The Sentry wiring we just shipped for vibn-frontend gave us:
|
||||
- A working Sentry org (`vibnai`)
|
||||
- An auth token with project-management scope
|
||||
- Verified knowledge that the build args / source maps flow works
|
||||
- A working `withSentryConfig` recipe in `vibn-frontend/next.config.ts`
|
||||
- All of those are reusable for stage 1 and 2 of this proposal.
|
||||
- Doing this **before** the beta means user projects start emitting
|
||||
error data on day one, so by the time we're debugging real beta
|
||||
user pain, we have a month of history to reason about.
|
||||
- Doing it after the beta means we'd have to retroactively
|
||||
instrument projects that have already been deployed for weeks.
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 — Auto-provision a Sentry project per Vibn project (~3 hr)
|
||||
|
||||
**Goal:** when a user creates a Vibn project, the platform creates a
|
||||
matching Sentry project under the `vibnai` org and stashes the DSN
|
||||
+ auth token in Coolify env vars on the user's app.
|
||||
|
||||
**What gets built:**
|
||||
|
||||
1. **A `provisionSentryProject(projectId, name)` helper** in
|
||||
`vibn-frontend/lib/integrations/sentry.ts`. Calls Sentry's
|
||||
`POST /api/0/teams/vibnai/{team}/projects/` with the project
|
||||
slug, returns the DSN.
|
||||
2. **Hook into project-create flow** — on first successful deploy,
|
||||
call the helper and write the resulting DSN + auth token into
|
||||
Coolify env vars (`NEXT_PUBLIC_SENTRY_DSN`,
|
||||
`SENTRY_AUTH_TOKEN`) for that app via the same Coolify API we
|
||||
used today.
|
||||
3. **Idempotency** — if the Sentry project already exists, fetch
|
||||
its DSN instead of creating a duplicate. Same project name
|
||||
convention every time: `vibn-{workspace}-{projectSlug}`.
|
||||
4. **Storage** — store `sentryProjectSlug` and `sentryAuthTokenId`
|
||||
on the Postgres `projects` row so we can look them up later
|
||||
without re-walking the Sentry org.
|
||||
|
||||
**Risk:** Sentry's API rate-limits team-project creation. We bypass
|
||||
this by reading-before-writing, so the only API cost on subsequent
|
||||
deploys is one GET.
|
||||
|
||||
**Definition of done:** create a fresh Vibn project → check Sentry
|
||||
org → see a project named `vibn-{ws}-{slug}` → check Coolify env on
|
||||
that app → see DSN populated.
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 — Bake Sentry into every scaffold template (~2 hr)
|
||||
|
||||
**Goal:** every Next.js / Vite / etc. starter template Vibn ships
|
||||
already has Sentry wired up. User does nothing.
|
||||
|
||||
**What gets built:**
|
||||
|
||||
1. **For each scaffold template in `vibn-frontend/lib/scaffold/`**,
|
||||
add the same files we shipped today:
|
||||
- `instrumentation.ts`
|
||||
- `instrumentation-client.ts`
|
||||
- `app/global-error.tsx` (Next.js) / equivalent boundary (Vite)
|
||||
- `next.config.ts` wrapped with `withSentryConfig` (Next.js)
|
||||
- `vite.config.ts` with `sentryVitePlugin` (Vite)
|
||||
- `Dockerfile` ARG declarations for `NEXT_PUBLIC_SENTRY_DSN` +
|
||||
`SENTRY_AUTH_TOKEN`
|
||||
2. **Add `@sentry/nextjs` (or `@sentry/react` + `@sentry/vite-plugin`)
|
||||
to each template's `package.json` `dependencies`.**
|
||||
3. **Document in template README** that Sentry is pre-wired and the
|
||||
user doesn't need to do anything.
|
||||
|
||||
**Risk:** Sentry's wrapper sometimes interacts badly with custom
|
||||
build configs (e.g. monorepos, custom webpack rules). Mitigation:
|
||||
the `errorHandler` we set today (`console.warn` instead of throw)
|
||||
ensures source map upload failures don't break builds.
|
||||
|
||||
**Definition of done:** scaffold a fresh Next.js project from Vibn
|
||||
templates → deploy → throw a test error → see it in Sentry,
|
||||
de-minified.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3 — Expose error feed to the AI as MCP tools (~2 hr)
|
||||
|
||||
**Goal:** the AI can ask Sentry "what's broken in project X?" and
|
||||
get a real answer.
|
||||
|
||||
**What gets built:**
|
||||
|
||||
Three new MCP tools in `vibn-frontend/lib/ai/vibn-tools.ts`:
|
||||
|
||||
1. **`project_recent_errors { projectId, since?, limit? }`**
|
||||
- Returns: `[{ id, title, count, lastSeen, culprit, level }]`
|
||||
- Default `since`: 24h. Default `limit`: 10.
|
||||
- Filters to unresolved issues only.
|
||||
- Implementation: read `sentryProjectSlug` off the project row,
|
||||
call Sentry's `GET /api/0/projects/{org}/{slug}/issues/`.
|
||||
|
||||
2. **`project_error_detail { projectId, issueId }`**
|
||||
- Returns: `{ stacktrace, breadcrumbs, request, user, replay_url }`
|
||||
- Implementation: Sentry's `GET /api/0/issues/{id}/events/latest/`.
|
||||
|
||||
3. **`project_error_resolve { projectId, issueId }`**
|
||||
- Side-effect: marks the issue resolved in Sentry.
|
||||
- Used by the AI after it ships a fix and confirms via tests.
|
||||
- Implementation: Sentry's `PUT /api/0/issues/{id}/` with
|
||||
`status: "resolved"`.
|
||||
|
||||
**Auth:** token storage is per-project (from Stage 1's `projects`
|
||||
row). Each project's AI sees only its own project's errors. No
|
||||
cross-project leakage.
|
||||
|
||||
**Definition of done:** in a Vibn chat for a project with known
|
||||
errors, ask the AI "any errors lately?" → AI calls
|
||||
`project_recent_errors` → shows real list.
|
||||
|
||||
---
|
||||
|
||||
## Stage 4 — Auto-surface unresolved errors at chat-turn start (~1 hr)
|
||||
|
||||
**Goal:** the AI doesn't wait to be asked. When the user opens a
|
||||
chat and there are unresolved errors, the AI mentions them on the
|
||||
first turn.
|
||||
|
||||
**What gets built:**
|
||||
|
||||
In `vibn-frontend/app/api/chat/route.ts`, at the start of each chat
|
||||
turn (before calling the model):
|
||||
|
||||
1. Call the same `project_recent_errors` logic Stage 3 exposed.
|
||||
2. If `count > 0`, prepend a synthetic system message:
|
||||
|
||||
```
|
||||
[PROJECT HEALTH]
|
||||
{N} unresolved Sentry issues in the last 24 hours:
|
||||
- {title} (×{count}, last seen {time}) — {culprit}
|
||||
- ...
|
||||
|
||||
If the user's first message is unrelated to these, you may still
|
||||
proactively mention them: "Quick FYI before we get into that —
|
||||
{X} has been failing for users."
|
||||
|
||||
If their message IS about a broken thing, prefer the matching
|
||||
Sentry issue's stack trace over guessing.
|
||||
```
|
||||
|
||||
3. Only fire this once per N chat turns (configurable, default 1
|
||||
per session opening) — we don't want to spam every turn.
|
||||
|
||||
**Risk:** false alarms (Sentry issue from yesterday's deploy that
|
||||
no one cares about anymore) make the AI annoying. Mitigation:
|
||||
tighten the `since` window to the last 6h, and only surface issues
|
||||
with `count >= 2` (one-off errors don't count).
|
||||
|
||||
**Definition of done:** intentionally break a deployed user
|
||||
project, open chat, type "what's up?" → AI's first response
|
||||
mentions the issue, with file path.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope for this proposal
|
||||
|
||||
- **User-owned Sentry orgs.** Some users will eventually want their
|
||||
own Sentry account, not the shared `vibnai` org. Ship-later;
|
||||
doesn't block the loop. Easy retrofit because storage is already
|
||||
per-project.
|
||||
- **Performance / Tracing data.** Sentry also captures spans /
|
||||
traces. Useful for "this endpoint is slow" but not the urgent
|
||||
product loop. Ship-later.
|
||||
- **Front-end UI for errors in Vibn.** A "Health" tab showing the
|
||||
Sentry feed in the Vibn UI is nice but not required for the AI
|
||||
loop to work. Ship-later.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
Add a **Phase 2.9 (Sentry-as-product loop)** to `BETA_LAUNCH_PLAN.md`
|
||||
covering Stages 1–4 as a single bundle. Estimate: **8 hr engineering**.
|
||||
|
||||
This is the second-highest-leverage item still ahead of beta,
|
||||
behind only the deploy-failed webhook (which is 30 min). Every
|
||||
hour spent here directly upgrades the value of every other beta
|
||||
test session that follows it.
|
||||
## Architecture
|
||||
- Sentry is automatically provisioned for every new project (`lib/integrations/sentry.ts`).
|
||||
- Environment variables (`NEXT_PUBLIC_SENTRY_DSN` and `SENTRY_AUTH_TOKEN`) are injected into the Coolify app.
|
||||
- The AI has access to `project_recent_errors`, `project_error_detail`, and `project_error_resolve` MCP tools to automatically read, diagnose, and fix exceptions directly from the Sentry API.
|
||||
- If unhandled exceptions are firing, the AI is prompted at the start of a conversation to address them (`app/api/chat/route.ts`).
|
||||
|
||||
Reference in New Issue
Block a user