mark/vibn-agent-runner
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3563b98de18a072bb77a436df3f1b88228b93bfa
vibn-agent-runner/docs/AGENT_TELEMETRY_STREAMING_PROJECT.md

12 KiB
Raw Blame History

Agent telemetry & live execution stream — project spec

This document captures concrete product and engineering additions discussed for Vibn: moving from poll-based session updates and in-memory jobs to a durable, ordered, push-friendly execution timeline—the web equivalent of a terminal agent’s clarity (step-by-step visibility, tool boundaries, failures, and later multi-agent signals).

1. Why this exists

Current behavior (baseline)

Surface How progress reaches the user Limits
Agent sessions (agent_sessions) Runner PATCHes output, status, changed_files to Next; UI polls GET …/agent/sessions/[id]. Latency, reconnect story, no single ordered stream; rich semantics encoded only in text.
Jobs (/api/agent/run, /api/jobs/:id) In-memory job-store (progress, toolCalls[]); UI polls job endpoint. Lost on restart; not shared across runner replicas; not unified with session UI.
Orchestrator / Atlas chat Request/response to runner; advisor path may be remote URL. No execution timeline for “long COO run” in-product unless you add the same event layer.

Product intent

  • Trust during long runs: users see what happened, when, and whether something was blocked—not only a final status.
  • Differentiation: “Ink-like” clarity in the browser—structured steps, not a blob of logs.
  • Foundation for multi-agent: handoffs, child work, and safety events need a common event pipe, not ad-hoc strings.

2. Goals

  1. Append-only execution events with monotonic ordering (per session or per job), suitable for replay after refresh.
  2. Server-push to the client (recommend SSE first; WebSocket if you need bi-directional on the same channel).
  3. Persistence so reconnect, refresh, and horizontal scaling do not lose history.
  4. Single conceptual model (AgentEvent) usable by:
    • Build → Agent tab (sessions),
    • Job flows (create/analyze-style),
    • optionally orchestrator long runs later.
  5. Backward compatibility during rollout: existing PATCH + output can remain as a fallback or be fed from the same emitter.

Non-goals (for v1)

  • Full OpenTelemetry export (optional later).
  • Real-time collaborative multi-user cursors on the same session.
  • Merging claude-code-fork—this spec is API + UI + persistence only.

3. Concept: AgentEvent

Core shape (suggested)

type AgentEvent = {
  seq: number;           // monotonic per stream (session_id or job_id)
  ts: string;            // ISO-8601
  runId: string;         // session UUID or job id — ties events to a run
  runKind: 'session' | 'job';
  phase: 'queued' | 'running' | 'completed' | 'failed' | 'stopped';

  type: AgentEventType;
  payload: Record<string, unknown>;  // type-specific
};

type AgentEventType =
  | 'run.started'
  | 'run.phase'              // e.g. planning, executing, committing
  | 'llm.turn.start'
  | 'llm.turn.end'
  | 'tool.start'
  | 'tool.end'
  | 'tool.output'            // chunked stdout/stderr if needed
  | 'safety.block'           // policy / protected path / command denied
  | 'file.changed'           // maps to today’s changed_files semantics
  | 'git.commit'
  | 'deploy.triggered'
  | 'deploy.status'
  | 'error'
  | 'run.completed'
  | 'handoff'                // v2: parent → child agent
  | 'child_job.started'      // v2: linked run id
  ;

Mapping from today’s session outputLine

Today (outputLine.type) Suggested event(s)
step / info run.phase or llm.turn.* with summary in payload.message
stdout / stderr tool.output or dedicated stream events
error error + optional safety.block if policy-driven
done run.completed

Keep human-readable message on events for UI defaults; add structured fields (tool, argsSummary, durationMs) for timeline rendering and filters.

4. Architecture (high level)

flowchart LR
  subgraph runner [vibn-agent-runner]
    RA[runSessionAgent / runAgent]
    EMIT[emitAgentEvent]
  end
  subgraph api [vibn-frontend Next.js]
    ING[POST internal ingest or PATCH extend]
    DB[(Postgres agent_events)]
    SSE[SSE GET /api/.../stream]
  end
  subgraph browser [Browser]
    UI[Timeline + live log]
  end
  RA --> EMIT
  EMIT -->|HTTPS + secret or mTLS| ING
  ING --> DB
  UI -->|EventSource| SSE
  SSE --> DB

Principles

  • Runner remains stateless regarding “truth”: it emits events; Next + DB are the source of truth for the UI (matches today’s session model).
  • Alternatively, runner could expose SSE directly—usually worse for auth, CORS, and one domain for the product. Prefer Next as SSE endpoint reading from DB.

5. Backend: vibn-agent-runner

5.1 Emit from execution paths

Location Action
agent-session-runner.ts Replace or supplement patchSession output-only updates with emitAgentEvent each turn / tool / error.
runAgent / tool loop (executeTool) Same emitter for job runs.
server.ts /agent/execute Emit run.started after 202; run.completed / error on exit.
Security / blocked tools (security.ts or equivalent) Emit safety.block with reason code (no secrets in payload).

5.2 Transport runner → Next

Option A (recommended): extend existing PATCH or add POST /api/internal/agent-events (or per-session batch append):

  • Headers: x-agent-runner-secret (same as today’s PATCH).
  • Body: single event or small batch { events: AgentEvent[] } with server-assigned seq to avoid races.

Option B: Runner writes to Redis/Postgres directly—couples runner to DB credentials; only do if you already run runner inside the same trust zone with DB URL.

5.3 Jobs store

  • Short term: continue in-memory for job metadata; persist events to Postgres keyed by jobId.
  • Medium term: optional Redis for job status + pub/sub to Next for low-latency SSE fanout (only if DB polling becomes a bottleneck).

6. Backend: vibn-frontend (Next.js)

6.1 Persistence

New table (example): agent_run_events

Column Notes
id UUID
run_id Session id or job id (text)
run_kind 'session' | 'job'
seq BIGSERIAL or per-run sequence enforced with unique constraint (run_id, seq)
project_id Nullable for jobs if not scoped
event JSONB — full AgentEvent or { type, ts, payload }
created_at default now()

Index: (run_id, seq) for range queries (WHERE run_id = $1 AND seq > $lastSeen).

Optional: migrate legacy agent_sessions.output to be derived (last N lines for email export) or dual-write during transition.

6.2 SSE route (example contract)

  • GET /api/projects/[projectId]/agent/sessions/[sessionId]/events/stream
    • Auth: session cookie / same as GET session (user must own project).
    • Query: ?afterSeq=123 for replay.
    • Response: text/event-stream; each message: data: {JSON}\n\n.
    • Heartbeat comments every ~15–30s to keep proxies alive.

For jobs (if not project-scoped): GET /api/jobs/[jobId]/events/stream with appropriate auth.

6.3 Ingest route (runner-only)

  • POST /api/internal/agent-events (or nested under project/session as you prefer).
  • Validates x-agent-runner-secret.
  • Inserts rows with server-generated seq (transaction per run or advisory lock per run_id).

7. Frontend (product UI)

7.1 Agent tab — timeline

  • EventSource (SSE) subscription when session is running; on load, fetch historical events (GET …/events?afterSeq=0 or SSE from 0).
  • Timeline components:
    • Group by llm.turn / tool.start–tool.end.
    • Expandable tool args (sanitized).
    • Distinct styling for safety.block and error.
  • Reconnect: on EventSource error, reopen with lastSeq from last received event.

7.2 Jobs / analyze flows

  • Same timeline component keyed by jobId if you surface those runs in UI.
  • Unifies mental model: “every run has a stream.”

7.3 Deprecate slow polling

  • Reduce GET …/agent/sessions/[id] poll interval when SSE connected; keep single poll for status / changed_files if those stay on session row only, or also emit file.changed events and drive UI from stream + one final consistency read.

8. Security & privacy

  • Never put tokens, env values, or full file contents in events by default; use truncation and hashes where needed.
  • safety.block: log reason code + user-safe message; align with security.ts behavior.
  • Rate limits on ingest endpoint (per run_id / per IP) to avoid abuse if misconfigured.

9. Environment variables

Variable Where Purpose
AGENT_RUNNER_SECRET Runner + Next Ingest / extended PATCH auth
VIBN_API_URL Runner Base URL for callbacks
AGENT_RUNNER_URL Next Start runs (unchanged)

Add if needed:

Variable Purpose
AGENT_EVENTS_INGEST_PATH Optional override for ingest URL
SSE_MAX_BUFFER Cap replay batch size

10. Phased roadmap (suggested)

Phase 1 — Foundation

  • Define AgentEvent TypeScript types in a shared package or duplicated minimal types in runner + frontend.
  • Create agent_run_events (or equivalent) + migration.
  • Implement ingest endpoint; wire runner session path to emit core events: run.started, tool.start / tool.end, error, run.completed, file.changed.
  • Dual-write: keep existing PATCH outputLine so nothing breaks.

Phase 2 — Push

  • SSE route + EventSource in Agent tab.
  • Backfill UI from DB on mount; then live tail.
  • Lower or gate polling on GET session.

Phase 3 — Jobs + durability

  • Emit same events from job execution path; persist by jobId.
  • Optional: replace in-memory job list with DB for multi-instance runner (later).

Phase 4 — Rich semantics

  • safety.block from policy layer.
  • deploy.* events if Coolify integration is user-visible.
  • Multi-agent: handoff, child_job.* with links in payload.

11. Success metrics

  • Time-to-first-visible-step after Run < 1s p95 (SSE).
  • After hard refresh mid-run, user sees consistent history (no duplicate seq, no gaps if you guarantee at-least-once ingest with idempotency keys later).
  • Support tickets / confusion drops on “what is the agent doing?” (qualitative).

12. Related code (repo anchors)

Use these when implementing:

  • Runner session loop + PATCH bridge: vibn-agent-runner/src/agent-session-runner.ts
  • Runner HTTP: vibn-agent-runner/src/server.ts (/agent/execute, /agent/stop, /agent/approve, /api/agent/run, /api/jobs/:id)
  • In-memory jobs: vibn-agent-runner/src/job-store.ts
  • Next session API + runner callback: vibn-frontend/app/api/projects/[projectId]/agent/sessions/[sessionId]/route.ts
  • Session create + fire-and-forget execute: vibn-frontend/app/api/projects/[projectId]/agent/sessions/route.ts

13. Open decisions

  1. Single table for sessions + jobs vs two tables (simpler queries vs flexibility).
  2. Seq generation: DB sequence per run_id vs global monotonic with (run_id, seq) composite only in app logic.
  3. Idempotency: runner retries may duplicate events—use event_id UUID from runner for dedupe on ingest.
  4. Orchestrator chat: treat as v2 unless you need a COO run timeline immediately.

Document version: 1.0 — aligned with discussion of runner ↔ frontend telemetry, SSE-first delivery, Postgres persistence, and future multi-agent event types.

Reference in New Issue View Git Blame Copy Permalink