📚 Zap Platform Documentation

Complete documentation for the Zap platform, including development guides, testing procedures, and infrastructure details

Core Documentation
Platform-wide documentation and guides
Topic Documentation
Specific guides and troubleshooting
External Links
TESTING
Test procedures and verification • Core Documentation

Zap Testing Guide

Comprehensive test procedures for all Zap subsystems. Run these after any significant change, deployment, or when debugging an issue. All commands assume you are on orcus.lan as user jd.

---

0. Windsurf Chat Capture

0.1 Database Schema

psql -U zap_user -d zap -c "SELECT COUNT(*) FROM windsurf_chats;"
psql -U zap_user -d zap -c "SELECT COUNT(*) FROM windsurf_chat_events;"

Should return counts (14+ chats, 20+ events if paris.lan is sending data).

0.2 API Endpoints

# Test windsurf-chats.php
curl -s "https://zap.orcus.lan/projects/api/windsurf-chats.php?limit=5" | jq '.chats | length'
Test windsurf-manual-capture.php

curl -s -X POST "https://zap.orcus.lan/projects/api/windsurf-manual-capture.php" \
  -H "Content-Type: application/json" \
  -d '{"sessionId":"test-'$(date +%s)'","workspace":"test-workspace","messages":[{"role":"user","content":"Test message","timestamp":"'$(date -Iseconds)'"}]}' | jq '.success'

Both should return valid JSON with data.

0.3 Web UI

  • Open https://zap.orcus.lan/projects/windsurf
  • Verify manual capture form appears at top
  • Check workspace filter pills populate
  • Toggle between card and list views
  • Click a chat to open transcript viewer

    • 0.4 Extension (on paris.lan)

  • Install .vsix file in Windsurf/Cursor
  • Run command palette: Windsurf Chat Capture: Capture Chat
  • Copy some text to clipboard first
  • Verify "✓ Chat captured" notification appears
  • Check Zap-Projects hub for new chat entry

    • ---

    1. Infrastructure Health

    1.1 Systemd Services

    sudo systemctl status orcus_tusd --no-pager
sudo systemctl status orcus_centrifugo --no-pager
sudo systemctl status orcus_zap-chat-watcher --no-pager
sudo systemctl status orcus-tunnel --no-pager

    All four should show active (running). If any are failed, check logs with journalctl -u -n 50 --no-pager.

    1.2 Cron Jobs

    crontab -l | grep -E 'zap|deep-analyse|sync-cursor'

    Expected entries:

  • /30 * -- deep analysis safety net
  • /5 * -- cursor title sync from paris.lan

    • 1.3 Database Connectivity

    psql -U zap_user -d zap -c "SELECT COUNT() AS chats FROM cursor_chats; SELECT COUNT() AS plans FROM cursor_plans; SELECT COUNT(*) AS projects FROM cursor_projects;"

    Should return non-zero counts for all three tables.

    1.4 Database Integrity

    psql -U zap_user -d zap -c "
  SELECT 'chats_fts' AS check_name, COUNT(*) FROM cursor_chats WHERE fts_vector IS NOT NULL
  UNION ALL
  SELECT 'plans_fts', COUNT(*) FROM cursor_plans WHERE fts_vector IS NOT NULL
  UNION ALL
  SELECT 'title_history', COUNT(*) FROM cursor_chat_title_history;
"

    1.5 Database Backups

    # Check backup_history table has recent entries
psql -U zap_user -d zap -c "SELECT status, COUNT(*) FROM backup_history WHERE created_at > NOW() - INTERVAL '7 days' GROUP BY status;"
    Verify PostgreSQL backup files exist
    
ls -la /mnt/merlin-backups/databases/postgresql/*.sql.gz | head -5
    Verify SQLite backup directories exist
    
ls -la /mnt/merlin-backups/databases/sqlite/orcus/
    Test backup script manually (dry run)
    
sudo /var/www/zap/cron/backup-pg.sh --dry-run 2>&1 | head -20
    Check cron job is scheduled
    
crontab -l | grep backup-pg

    Expected results:

  • backup_history should show recent success entries
  • PostgreSQL backup files should be < 7 days old
  • SQLite directories should contain .sqlite.gz files
  • Cron should show 0 6 * daily schedule

    • For detailed restore procedures, see /var/www/zap/docs/BACKUP_RESTORE.md

    1.5 Apache

    sudo apachectl -S 2>&1 | grep -E 'zap|projects'
curl -sk https://zap.orcus.lan/ -o /dev/null -w '%{http_code}\n'
curl -sk https://zap.orcus.lan/projects/ -o /dev/null -w '%{http_code}\n'

    Both should return 200.

    1.6 External Access (Tunnel)

    ssh jd@orcus.getzap.co "echo tunnel_ok"
curl -sI https://orcus.getzap.co/ -o /dev/null -w '%{http_code}\n'

    Should return tunnel_ok and 200.

    ---

    2. Zap-Projects

    2.1 Web UI Smoke Tests

    Open in browser and verify visually:

    URLCheck
    ------------
    https://zap.orcus.lan/projects/Projects index loads, card/list toggle works, search filters projects
    https://zap.orcus.lan/projects/cursorCursor Hub loads, Chats tab shows chats with Started/Modified times
    https://zap.orcus.lan/projects/cursor (Plans tab)Plans listed with Created/Modified times, list/card toggle works, "Load more" appears when >50 plans
    https://zap.orcus.lan/projects/cursor (tab persistence)Switch to Plans tab, refresh page -- Plans tab should still be selected
    https://zap.orcus.lan/projects/cursor?tab=plansDeep-link loads Plans tab directly (overrides localStorage)
    https://zap.orcus.lan/projects/cursor (click chat)Clicking a chat title opens transcript viewer with full conversation
    https://zap.orcus.lan/projects/cursor (click plan)Clicking a plan opens plan-viewer with rendered markdown content
    Plan viewer back link"All Plans" link returns to Cursor Hub with Plans tab selected
    https://zap.orcus.lan/projects/cursor (search)Typing in search box filters results in real time
    https://zap.orcus.lan/projects/zapProject detail page loads, shows linked chats and plans
    https://zap.orcus.lan/projects/docsDocumentation viewer renders AGENT-GUIDE.md
    https://zap.orcus.lan/projects/rulesCursor Rules Viewer shows all .mdc rule files

    2.2 REST API Tests

    # List projects
curl -s https://zap.orcus.lan/projects/api/projects.php | python3 -m json.tool | head -20
    List chats (first 5)
    
curl -s 'https://zap.orcus.lan/projects/api/chats.php?limit=5' | python3 -m json.tool | head -20
    Search chats
    
curl -s 'https://zap.orcus.lan/projects/api/chats.php?query=vpn' | python3 -m json.tool | head -20
    List plans
    
curl -s 'https://zap.orcus.lan/projects/api/plans.php' | python3 -m json.tool | head -20
    Search plans (FTS)
    
curl -s 'https://zap.orcus.lan/projects/api/plans.php?query=vpn+pemex' | python3 -m json.tool | head -20
    Global search
    
curl -s 'https://zap.orcus.lan/projects/api/search.php?q=wireguard' | python3 -m json.tool | head -30

    All should return valid JSON with results. The "vpn pemex" plan search should return at least 1 result (VPN VM for PEMEX).

    2.3 MCP Server Tests

    From a Cursor workspace on orcus that has .cursor/mcp.json configured:

  • Ask the AI: "What MCP tools do you have from zap-projects?"
  • Expected: 7 tools listed (list_projects, get_project, list_chats, get_chat, list_plans, search_all, get_cursor_rules)
  • Ask: "Search all zap-projects for vpn" -- should find relevant chats and plans
  • Ask: "List plans about pemex" -- should return VPN VM for PEMEX plan

    • 2.4 File Watcher Test

    # Create a test plan file
echo -e "---\nname: Test Plan\n---\n\nThis is a test plan for watcher verification." > /home/jd/.cursor/plans/test_watcher_verification_00000000.plan.md
    Wait 35 seconds (30s debounce + processing)
    
sleep 35
    Check if it was indexed
    
psql -U zap_user -d zap -c "SELECT name, created_at FROM cursor_plans WHERE path LIKE '%test_watcher%';"
    Clean up
    
rm /home/jd/.cursor/plans/test_watcher_verification_00000000.plan.md
psql -U zap_user -d zap -c "DELETE FROM cursor_plans WHERE path LIKE '%test_watcher%';"

    Should show the test plan in the database after the wait.

    2.5 Title Sync Test

    # Run title sync manually
php /var/www/zap/scripts/sync-cursor-titles.php 2>&1 | tail -5
    Check sync results
    
psql -U zap_user -d zap -c "SELECT chat_id, cursor_title, cursor_synced_at FROM cursor_chats WHERE cursor_synced_at IS NOT NULL ORDER BY cursor_synced_at DESC LIMIT 5;"

    Should show recently synced titles with timestamps.

    2.6 Ingestion Test

    # Dry run (no changes)
php /var/www/zap/scripts/ingest-chat-transcripts.php --skip-llm --verbose 2>&1 | tail -10
    Plans-only mode
    
php /var/www/zap/scripts/ingest-chat-transcripts.php --plans-only --skip-llm 2>&1 | tail -5

    Both should complete without errors.

    ---

    3. Zap-Writer

    3.1 Web UI Smoke Tests

    URLCheck
    ------------
    https://zap.orcus.lan/writer/Project dashboard loads, lists writing projects
    https://zap.orcus.lan/writer/project/mexico-oil-chapterProject workspace loads with tabbed interface (Edits, Fact-Check, Currency, Bibliography, Charts)
    https://zap.orcus.lan/admin/modelsLLM model preferences page loads, shows Ollama models

    3.2 API Tests

    # List edits
curl -s 'https://zap.orcus.lan/writer/api/edits.php?project=mexico-oil-chapter' | python3 -m json.tool | head -20
    List bibliography
    
curl -s 'https://zap.orcus.lan/writer/api/bibliography.php?project=mexico-oil-chapter' | python3 -m json.tool | head -20
    List currency items
    
curl -s 'https://zap.orcus.lan/writer/api/currency.php?project=mexico-oil-chapter' | python3 -m json.tool | head -20

    3.3 CLI Engines (Dry Run)

    # Bibliography ingestion (fast, regex-based)
php /var/www/zap/apps/zap-writer/bin/ingest-bib-direct.php --project=mexico-oil-chapter --dry-run 2>&1 | tail -5

    Should complete without errors and report parsed reference count.

    ---

    4. Recording System

    4.1 Service Check

    sudo systemctl status orcus_tusd --no-pager
curl -sk https://zap.orcus.lan/files/ -o /dev/null -w '%{http_code}\n'

    tusd should be running; /files/ should return 405 (TUS endpoint, not browsable).

    4.2 TUS Upload Test

    # Create a test upload (should return Location header)
curl -sk -X POST https://zap.orcus.lan/files/ \
  -H "Tus-Resumable: 1.0.0" \
  -H "Upload-Length: 100" \
  -H "Upload-Metadata: filename dGVzdC53ZWJt" \
  -D - -o /dev/null 2>&1 | grep -i location

    Should return a Location header with an upload ID.

    4.3 Recordings Page

    Open https://zap.orcus.lan/recordings.php -- should list existing recordings with search, filter, and playback controls.

    4.4 NAS Storage

    ls /mnt/nas/zap-recordings/ 2>/dev/null && echo "NAS mounted" || echo "NAS not mounted"

    ---

    5. Authentication

    5.1 Login Flow

  • Open https://zap.orcus.lan/login.php
  • Log in with valid credentials
  • Verify redirect to home or connect page
  • Check navbar shows user icon with dropdown

    • 5.2 API Tests

    # Check auth status (should return 401 without session)
curl -s https://zap.orcus.lan/api/auth-check.php -w '\n%{http_code}\n'
    Login
    
curl -s -X POST https://zap.orcus.lan/api/auth-login.php \
  -H 'Content-Type: application/json' \
  -d '{"email":"test@example.com","password":"test123"}' | python3 -m json.tool

    5.3 Admin Panel

    Open https://zap.orcus.lan/admin.php (requires admin login) -- should list users with permissions.

    5.4 Detailed Auth Testing

    See docs/testing-auth-and-migration.md for comprehensive auth test procedures including one-time codes, password reset, and registration.

    ---

    6. Messaging

    6.1 Page Load

    Open https://zap.orcus.lan/messages.php (requires login) -- should show room list and message interface.

    6.2 API Tests

    # List rooms (requires auth cookie -- test in browser console)

    fetch('/api/chat-rooms.php?session=test').then(r=>r.json()).then(console.log)

    6.3 Centrifugo Connection

    # Verify Centrifugo is responding
curl -s http://127.0.0.1:25001/health | python3 -m json.tool

    Should return {"status":"ok"} or similar health response.

    ---

    7. Multi-Participant Recording (Host/Guest)

    7.1 Host Page

    Open https://zap.orcus.lan/host (or https://orcus.getzap.co/host for external):

  • Page loads with camera controls
  • Session ID is generated
  • Guest link is displayed

    • 7.2 Guest Page

    Open https://zap.orcus.lan/guest?session={id}:

  • Page loads with camera controls
  • Self-view camera works
  • Host video appears when connected

    • 7.3 WebRTC Connection

    With both host and guest open:

  • Host camera should appear on guest page
  • Guest camera should appear on host page
  • Console should show "ICE connection established" (no "Connection disconnected")

    • Note: Multi-participant recordings are currently WIP -- video streaming works but recording/upload may have issues.

    ---

    8. Shared Libraries

    8.1 LLM Client

    php -r "
require '/var/www/zap/web/bootstrap.php';
require '/var/www/zap/shared/libs/LLMClient.php';
\$c = new LLMClient();
echo 'Model: ' . \$c->getModel() . PHP_EOL;
\$models = \$c->listModels();
echo 'Models available: ' . count(\$models) . PHP_EOL;
"

    Should report the default model and a non-zero model count from delphi.lan.

    8.2 Search Client

    php -r "
require '/var/www/zap/web/bootstrap.php';
require '/var/www/zap/shared/libs/SearchClient.php';
\$s = new SearchClient();
\$r = \$s->search('Mexico oil production 2024');
echo 'Results: ' . count(\$r) . PHP_EOL;
"

    Should return search results from SearXNG on delphi.lan.

    8.3 PostgreSQL Client

    php -r "
require '/var/www/zap/web/bootstrap.php';
require '/var/www/zap/shared/libs/PgClient.php';
\$db = PgClient::connect();
\$r = \$db->query('SELECT COUNT(*) AS c FROM cursor_chats')->fetch();
echo 'Chats in DB: ' . \$r['c'] . PHP_EOL;
"

    ---

    9. VPN Fetch Container

    9.1 Container Status

    ssh root@vpn-fetch.lan "echo container_reachable"

    9.2 VPN Status (Do Not Run Unless Needed)

    # Check VPN status (does NOT activate VPN)
ssh root@vpn-fetch.lan "/root/scripts/vpn-connect.sh status"

    9.3 Downloaded Data

    ls -la /APPS/energystats/pemex-downloads/anuarios/ | head -15

    Should list PEMEX Anuario PDF files.

    ---

    10. Full-Text Search Verification

    10.1 Chat FTS

    psql -U zap_user -d zap -c "
  SELECT chat_id, first_query
  FROM cursor_chats
  WHERE fts_vector @@ plainto_tsquery('english', 'vpn wireguard')
  LIMIT 5;
"

    Should return chats that mention VPN/WireGuard anywhere in their transcript.

    10.2 Plan FTS

    psql -U zap_user -d zap -c "
  SELECT name, path
  FROM cursor_plans
  WHERE fts_vector @@ plainto_tsquery('english', 'vpn pemex')
  LIMIT 5;
"

    Should return the VPN VM for PEMEX plan (and possibly others).

    10.3 Cross-System Search

    curl -s 'https://zap.orcus.lan/projects/api/search.php?q=pemex+anuario' | python3 -m json.tool | head -30

    Should return results from both chats and plans.

    ---

    11. Zap-Writer: Document Ingestion & RAG

    11.1 Google Doc Ingestion

    cd /var/www/zap/apps/zap-writer
php bin/ingest-gdoc-tabs.php --project=hidden-money --dry-run

    Should list all tabs from the Hidden Money Google Doc with word counts. Remove --dry-run to actually ingest.

    11.2 Document Chunk Counts

    PGPASSWORD='ioVYWLKDzEwjgmgWrhxqf/w1jVBdtE3godICRZoUXBE=' psql -U zap_user -h localhost -d zap -c "
  SELECT COUNT(*) as total_chunks, MAX(char_count) as max_chars, AVG(char_count)::int as avg_chars
  FROM writer_document_chunks WHERE project_id = 2;"

    Should show 424 chunks, max ~1700 chars, avg ~1300 chars.

    11.3 Embedding Coverage

    PGPASSWORD='ioVYWLKDzEwjgmgWrhxqf/w1jVBdtE3godICRZoUXBE=' psql -U zap_user -h localhost -d zap -c "
  SELECT
    (SELECT COUNT(*) FROM writer_document_chunks WHERE project_id = 2) as chunks,
    (SELECT COUNT(*) FROM writer_chunk_embeddings WHERE project_id = 2) as embeddings;"

    Both counts should match (424). If embeddings < chunks, re-run php bin/embed-chunks.php --project=hidden-money.

    11.4 RAG Search (API)

    curl -sk 'https://zap.orcus.lan/writer/api/rag-search.php' -X POST \
  -H 'Content-Type: application/json' \
  -d '{"project_slug":"hidden-money","query":"donor advised funds","limit":5}' \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(f'Semantic: {d[\"total_semantic\"]}, Keyword: {d[\"total_keyword\"]}')"

    Should return non-zero results for both semantic and keyword search.

    11.5 Multi-Source RAG Answer (API)

    curl -sk 'https://zap.orcus.lan/writer/api/rag-answer.php' -X POST \
  -H 'Content-Type: application/json' \
  -d '{"project_slug":"hidden-money","query":"what are DAF fees?","model":"qwen3:8b"}' \
  | python3 -c "import json,sys; d=json.load(sys.stdin); parts=d.get('parts',{}); [print(f'[{k}] {v.get(\"duration_ms\",\"?\")}ms') for k,v in parts.items() if isinstance(v,dict)]"

    Should show timing for docs, llm, and web parts (each 5-30s depending on model). External shows placeholder.

    11.6 Models API

    curl -sk 'https://zap.orcus.lan/writer/api/models.php' \
  | python3 -c "import json,sys; d=json.load(sys.stdin); [print(f'{s[\"server_label\"]}: {len(s[\"models\"])} models') for s in d.get('servers',[])]"

    Should show models from delphi (90+), phoebe (2-3), and titan (2-3).

    11.7 Embedding Server Availability

    curl -s http://ollama-titan.lan:11434/api/tags | python3 -c "import json,sys; [print(m['name']) for m in json.load(sys.stdin).get('models',[])]"

    Should include nomic-embed-text:latest. If missing, pull it: curl -s http://ollama-titan.lan:11434/api/pull -d '{"name":"nomic-embed-text","stream":false}'

    11.8 UI Tests (browser)

  • Navigate to https://zap.orcus.lan/writer/project.php?slug=hidden-money&tab=search
  • Verify search box has proper padding on both sides (not edge-to-edge)
  • Verify model dropdown appears next to Search button with models grouped by server
  • Type "what are DAF fees?" and click Search
  • Verify 4 answer cards appear: Your Documents (gold), LLM Knowledge (indigo), Web Search (green), External Library (grey)
  • Verify the Documents card shows source citations with similarity scores
  • Verify clicking a source citation opens the document viewer
  • Verify "Raw Matches" toggle appears below the cards
  • Test collapsing/expanding answer cards by clicking headers

    • ---

    Quick Checklist

    For a fast post-deployment sanity check, run these in order:

    # 1. Services
sudo systemctl is-active orcus_tusd orcus_centrifugo orcus_zap-chat-watcher orcus-tunnel
    2. Web
    
curl -sk https://zap.orcus.lan/ -o /dev/null -w 'home:%{http_code} '
curl -sk https://zap.orcus.lan/projects/ -o /dev/null -w 'projects:%{http_code} '
curl -sk https://zap.orcus.lan/writer/ -o /dev/null -w 'writer:%{http_code} '
curl -sk https://zap.orcus.lan/projects/cursor -o /dev/null -w 'cursor:%{http_code}\n'
    3. API
    
curl -s 'https://zap.orcus.lan/projects/api/search.php?q=test' | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'search: {len(d.get(\"chats\",[]))} chats, {len(d.get(\"plans\",[]))} plans')"
    4. Writer RAG
    
curl -sk 'https://zap.orcus.lan/writer/api/rag-search.php' -X POST -H 'Content-Type: application/json' -d '{"project_slug":"hidden-money","query":"philanthropy","limit":3}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'writer RAG: {d.get(\"total_semantic\",0)} semantic')"
    5. Database
    
psql -U zap_user -d zap -c "SELECT 'chats' AS t, COUNT() FROM cursor_chats UNION ALL SELECT 'plans', COUNT() FROM cursor_plans UNION ALL SELECT 'projects', COUNT() FROM cursor_projects UNION ALL SELECT 'writer_docs', COUNT() FROM writer_documents UNION ALL SELECT 'writer_chunks', COUNT() FROM writer_document_chunks UNION ALL SELECT 'writer_embeddings', COUNT() FROM writer_chunk_embeddings;" -t

    All services should report active. All HTTP codes should be 200. Database should return non-zero counts. Writer RAG should return semantic results.

    ---

    6. Web Documentation Access

    All platform documentation is now accessible via web viewers:

  • Zap Platform Docs: https://zap.orcus.lan/projects/zap-docs
    • - Core documentation: CHANGELOG, CODING_HISTORY, ROADMAP, TESTING, INFRASTRUCTURE - Topic guides: camera-freeze, macbook-tunnel, testing-auth - Cross-references to orcus.lan server documentation

  • Orcus Server Docs: https://orcus.lan/docs.php
- Server documentation: README, ROADMAP, BACKUP_RESTORE - Infrastructure plans and disaster recovery procedures - Hot swap VM and LLM redundancy strategies

These web viewers provide enhanced navigation, search, and mobile-friendly access to all documentation.

← Back to Zap Projects 🖥️ Orcus Server Docs →