# Checklist & Document Upload - All Issues Fixed ✅

## Problems Identified

1. **Checklist showed 0 documents** even after upload
2. **Pasted text content** wasn't counted as documents
3. **Aggressive fallback detection** triggered extraction too early (on "Perfect!" alone)

---

## Root Causes Found

### Issue 1: Firestore Index Missing
The checklist query used:
```typescript
where('projectId', '==', projectId)
where('sourceType', '==', 'imported_document')
```

This requires a composite index that was missing. **FIXED** ✅
- Added index to `firestore.indexes.json`
- Deployed to Firebase
- Index takes 5-15 minutes to build (now complete)

### Issue 2: Pasted Content Not Creating knowledge_items
When users pasted text via "Add Context" → "Text Paste":
- Only created `contextSources` subcollection entry
- Did NOT create `knowledge_item`
- Result: Not counted in checklist, not included in extraction

**FIXED** ✅
- Now calls `/api/projects/[projectId]/knowledge/import-ai-chat`
- Creates `knowledge_item` with `sourceType: 'imported_ai_chat'`
- Pasted content now shows in checklist and gets extracted

### Issue 3: Checklist Only Counted One sourceType
Checklist query filtered for ONLY `'imported_document'`:
```typescript
where('sourceType', '==', 'imported_document')  // ← Too narrow!
```

Missed `'imported_ai_chat'` (pasted content).

**FIXED** ✅
- Changed to query ALL knowledge_items for project
- Filter in memory for both types:
  ```typescript
  sourceType === 'imported_document' || sourceType === 'imported_ai_chat'
  ```

### Issue 4: Aggressive Fallback Detection
Fallback detection triggered on ANY message containing "Perfect!":
```typescript
const confirmPhrases = ['perfect! let me analyze', ...];
replyLower.includes(phrase);  // ← Matches "Perfect, I can see..."
```

This caused premature extraction when AI said "Perfect, I can see your GitHub repo".

**FIXED** ✅
- Now requires BOTH readiness word AND analysis action:
  ```typescript
  // Must contain analysis keywords
  const analysisKeywords = ['analyze', 'analyzing', 'digging', 'extraction', 'processing'];
  
  // AND match specific phrases
  const confirmPhrases = [
    'let me analyze what you',
    'i\'ll start digging into',
    'i\'m starting the analysis',
    //...
  ];
  ```

---

## Files Changed

### 1. `firestore.indexes.json`
**Added:**
```json
{
  "collectionGroup": "knowledge_items",
  "queryScope": "COLLECTION",
  "fields": [
    { "fieldPath": "projectId", "order": "ASCENDING" },
    { "fieldPath": "sourceType", "order": "ASCENDING" }
  ]
}
```

### 2. `firestore.rules`
**Added rules for:**
- `knowledge_items` - users can read their own project's items
- `chat_extractions` - users can read their own project's extractions
- `chat_conversations` - users can read their own project's conversations
- `githubConnections` - users can read their own connections
- `linkedExtensions` - users can read their own extension links

### 3. `components/ai/collector-checklist.tsx`
**Changed:**
- Query loads ALL knowledge_items (no sourceType filter)
- Filters in memory for `'imported_document'` OR `'imported_ai_chat'`
- Listens to project document for GitHub/extension status
- All with real-time `onSnapshot` listeners

### 4. `app/[workspace]/project/[projectId]/context/page.tsx`
**Added to `handleAddChatContent`:**
- Calls `/api/projects/[projectId]/knowledge/import-ai-chat`
- Creates `knowledge_item` in addition to `contextSources` entry
- Pasted content now treated same as uploaded files

### 5. `app/api/ai/chat/route.ts`
**Changed fallback detection:**
- Requires `analysisKeywords` AND specific confirmation phrases
- No longer triggers on "Perfect!" alone
- More precise phrase matching

---

## How It Works Now

### Document Upload Flow
1. User clicks "Add Context" → "File Upload"
2. Selects file(s) → clicks "Upload X Files"
3. Frontend calls `/api/projects/[projectId]/knowledge/upload-document`
4. Backend creates:
   - File in Firebase Storage
   - `knowledge_item` with `sourceType: 'imported_document'`
   - `contextSources` subcollection entry
5. Checklist listener detects new `knowledge_item`
6. Checklist updates: "1 of 3 complete" ✅

### Text Paste Flow
1. User clicks "Add Context" → "Text Paste"
2. Enters title + content → clicks "Add Context"
3. Frontend calls:
   - `/api/context/summarize` (generates AI summary)
   - `/api/projects/[projectId]/knowledge/import-ai-chat` (creates knowledge_item)
4. Backend creates:
   - `knowledge_item` with `sourceType: 'imported_ai_chat'`
   - `contextSources` subcollection entry
5. Checklist listener detects new `knowledge_item`
6. Checklist updates: "1 of 3 complete" ✅

### Checklist Real-Time Updates
```typescript
// Project data (GitHub, extension)
onSnapshot(doc(db, 'projects', projectId), ...)

// Document count (files + pasted content)
onSnapshot(query(
  collection(db, 'knowledge_items'),
  where('projectId', '==', projectId)
), ...)
```

Updates **instantly** when:
- ✅ Documents uploaded
- ✅ Text pasted
- ✅ GitHub connected
- ✅ Extension linked

No chat message needed!

---

## Test Results

### ✅ Upload File
- File uploads successfully
- `knowledge_item` created with `sourceType: 'imported_document'`
- Checklist shows "1 of 3 complete" immediately
- Console log: `[CollectorChecklist] Document count: 1`

### ✅ Paste Text
- Text pasted successfully
- `knowledge_item` created with `sourceType: 'imported_ai_chat'`
- Checklist shows "1 of 3 complete" (or 2 if already had files)
- Console log: `[CollectorChecklist] Document count: 2`

### ✅ Connect GitHub
- GitHub OAuth completes
- Checklist shows "✓ GitHub connected" immediately
- Shows repo name: "MawkOne/dr-dave"

### ✅ No Premature Extraction
- AI says "Perfect, I can see your GitHub repo"
- Fallback does NOT trigger (no "analyze" keyword)
- Phase stays as `'collector'`
- User must explicitly say "that's everything" or similar

---

## Date
November 17, 2025