From a5030c7fba02654e9e446b63bb4f7fe07514edf5 Mon Sep 17 00:00:00 2001 From: Hermes Bot Date: Thu, 9 Jul 2026 11:39:17 +0800 Subject: [PATCH] =?UTF-8?q?Add=20QA=20Test=20Strategy=20page=20=E2=80=94?= =?UTF-8?q?=20full=20ecosystem=20test=20plan=20with=20agent=20tagging?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tags: @Ody @OC @PI @hermes-contabo 47 test cases across 7 tiers covering: - Hermes Plugin core + API - Ody & PI agent clones - OpenClaw provider - Falah OS CE + Docker - CI/CD pipeline - E2E integration scenarios - Pre-deploy smoke suite --- Home.md | 2 + QA-Test-Strategy.md | 287 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 289 insertions(+) create mode 100644 QA-Test-Strategy.md diff --git a/Home.md b/Home.md index 0f1e0b1..0db2314 100644 --- a/Home.md +++ b/Home.md @@ -4,6 +4,7 @@ Welcome to the Hermes cPanel Agent wiki — documentation for running AI agent c ## Pages +- **[QA Test Strategy](QA-Test-Strategy)** — Comprehensive QA plan for the entire Falah OS ecosystem - **[Cloudflare WAF Bypass Discovery](Cloudflare-WAF-Bypass-Discovery)** — How we discovered Cloudflare wasn't actually blocking the WP REST API - **[Getting Started](Getting-Started)** — Setup guide for new developers - **[Gitea Actions CI/CD](Gitea-Actions-CICD)** — Full CI/CD pipeline documentation @@ -20,3 +21,4 @@ Welcome to the Hermes cPanel Agent wiki — documentation for running AI agent c - Code: https://git.falahos.my/wmj/hermes-cpanel-agent - Issues: https://git.falahos.my/wmj/hermes-cpanel-agent/issues +- Wiki: https://git.falahos.my/wmj/hermes-cpanel-agent/wiki diff --git a/QA-Test-Strategy.md b/QA-Test-Strategy.md new file mode 100644 index 0000000..25e47f7 --- /dev/null +++ b/QA-Test-Strategy.md @@ -0,0 +1,287 @@ +# QA Test Strategy — Falah OS Ecosystem + +> **Status:** Active · **Owner:** @hermes-contabo · **Last Updated:** 2026-07-09 +> +> **Tagged Agents:** @Ody @OC @PI @hermes-contabo + +--- + +## Ecosystem Overview + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Falah OS CE (AWS 13.140.161.244) │ +│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ ┌───────────┐ │ +│ │ Falah │ │ iStore │ │ App Manager │ │ Gateway │ │ +│ │ App:3005 │ │ :3021 │ │ :3022 │ │ :3000 │ │ +│ │ (React) │ │(registry)│ │(docker apps) │ │ │ │ +│ └──────────┘ └──────────┘ └──────────────┘ └───────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + Cloudflare Tunnel + │ +┌─────────────────────────────────────────────────────────────────┐ +│ Hermes Plugin + WordPress (Ummah) │ +│ ┌──────────────────────────────────────────────────────────┐ │ +│ │ Hermes AI Bridge Plugin │ │ +│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────────┐ │ │ +│ │ │REST API │ │AI Prov. │ │Scheduler │ │Nextcloud │ │ │ +│ │ │hermes/v1 │ │(OpenClaw)│ │(WP-Cron) │ │Bridge │ │ │ +│ │ └──────────┘ └──────────┘ └──────────┘ └────────────┘ │ │ +│ └──────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + ┌────────────┴────────────┐ + ▼ ▼ + ┌──────────────┐ ┌──────────────────┐ + │ Ody Clone │ │ PI Clone │ + │ (Content Gen)│ │ (Auto-Reply) │ + │ port 7000 │ │ (in WordPress) │ + └──────────────┘ └──────────────────┘ + │ │ + └──────────┬──────────────┘ + ▼ + ┌──────────────────┐ + │ OpenClaw / OC │ + │ (AI Provider) │ + │ Mac Mini :4000 │ + └──────────────────┘ +``` + +## Components Under Test + +| Component | Agent | Location | Type | Priority | +|-----------|-------|----------|------|----------| +| Hermes WordPress Plugin | @hermes-contabo | cPanel / WordPress | PHP Plugin | P0 | +| Ody Clone (Content Gen) | @Ody | Mac Mini :7000 | MCP Server | P0 | +| PI Clone (Auto-Reply) | @PI | WordPress | Agent | P1 | +| OpenClaw Provider | @OC | Mac Mini :4000 | MCP Server | P0 | +| Falah OS CE Frontend | @hermes-contabo | AWS 13.140.161.244 | React + Express | P1 | +| iStore App Registry | @hermes-contabo | Docker :3021 | Express Service | P1 | +| App Manager | @hermes-contabo | Docker :3022 | Express Service | P1 | +| Gitea Actions CI/CD | @hermes-contabo | Contabo | Pipeline | P0 | + +--- + +## Tier 1: Hermes Plugin (Core Runtime) + +**Owner: @Ody @OC** + +| # | Test | Method | Pass Criteria | Priority | +|---|------|--------|--------------|----------| +| 1.1 | Plugin activation | Activate → check DB tables, options, cron | `hermes_log` table exists, `hermes_api_key` set | 🔴 P0 | +| 1.2 | REST API auth | Call with wrong/missing/valid `X-API-Key` | 401 wrong → 401 missing → 200 valid | 🔴 P0 | +| 1.3 | All 12 REST routes | Health, CRUD posts/users/settings/schedules/logs | Each returns correct shape | 🔴 P0 | +| 1.4 | Content generation | `POST /content/generate` with prompt | Post created with AI content | 🔴 P0 | +| 1.5 | AI provider switching | Switch OpenClaw ↔ OpenAI ↔ Anthropic | API calls route to correct provider | 🟡 P1 | +| 1.6 | WP-Cron scheduling | Schedule future content → tick → verify | Post published at correct time | 🔴 P0 | +| 1.7 | Security: no banned funcs | `exec/shell_exec/system/eval` grep | Zero hits | 🔴 P0 | +| 1.8 | Admin settings UI | Page renders, save works, key regen works | All fields persist, new key invalidates old | 🟡 P1 | +| 1.9 | Logging | All actions logged in `hermes_log` | Entries present with correct levels | 🟡 P1 | + +### API Contract Verification + +``` +hermes/v1: + ├── GET /health → {"status":"ok","version":"1.0.0",...} + ├── POST /content/generate → {"success":true,"post_id":N,"edit_url":"..."} + ├── POST /content/schedule → {"success":true,"task_id":N,"scheduled_date":"..."} + ├── GET /posts → [{id,title,status,date,modified,permalink}] + ├── GET /posts/:id → {id,title,content,excerpt,status,date,...} + ├── PUT /posts/:id → {"success":true,"post_id":N} + ├── DELETE /posts/:id → {"success":true,"deleted":N} + ├── GET /users → [{ID,login,email,display_name,roles}] + ├── GET /users/:id → {id,login,email,display_name,roles} + ├── GET /settings → {site_name,ai_provider,ai_model,...} + ├── PUT /settings → {"success":true} + ├── GET /schedules → [{id,type,scheduled_date,status,created}] + └── GET /logs?level=&limit= → [{id,level,message,context,created_at}] +``` + +--- + +## Tier 2: Ody Clone (Content Generation Agent) + +**Owner: @Ody @OC** + +| # | Test | Method | Pass Criteria | Priority | +|---|------|--------|--------------|----------| +| 2.1 | MCP connectivity | JSON-RPC `initialize` on `mac-mini.local:7000` | Returns `serverInfo` + capabilities | 🔴 P0 | +| 2.2 | Tool discovery | `tools/list` after init | Exposes content-gen tools | 🔴 P0 | +| 2.3 | Content generation | Call MCP `content_generate` tool | Returns markdown content | 🔴 P0 | +| 2.4 | Content quality | Generate 10 Islamic finance posts | No haram content, correct terminology | 🟡 P1 | +| 2.5 | Error handling | Missing API key, rate limits, timeouts | Structured error responses | 🔴 P0 | +| 2.6 | Prompt injection | Malicious prompt payloads | Prompt boundaries respected, no system prompt leak | 🔴 P0 | + +### MCP Tool Contract + +``` +tools: + content_generate: + input: {prompt: string, tone?: string, length?: number} + output: {content: string, title?: string, word_count: number} + + content_schedule: + input: {prompt: string, schedule_date: string, post_type?: string} + output: {task_id: number, scheduled_date: string} + + health_check: + input: {} + output: {status: string, uptime: number, model: string} +``` + +--- + +## Tier 3: PI Clone (Auto-Reply Agent) + +**Owner: @PI** + +| # | Test | Method | Pass Criteria | Priority | +|---|------|--------|--------------|----------| +| 3.1 | Comment auto-reply | Create comment on WordPress post | PI replies within cron window | 🟡 P1 | +| 3.2 | Reply relevance | Test comments about Islamic finance | Reply is contextually appropriate | 🟡 P1 | +| 3.3 | Spam guard | Spam comment → PI should not reply | No reply generated for spam | 🟡 P1 | +| 3.4 | Rate limiting | 50 comments in 1 min → PI behavior | Does not reply to all, respects limits | 🟢 P2 | + +--- + +## Tier 4: OpenClaw Provider (AI Inference) + +**Owner: @OC** + +| # | Test | Method | Pass Criteria | Priority | +|---|------|--------|--------------|----------| +| 4.1 | MCP connectivity | JSON-RPC `initialize` on `mac-mini.local:4000` | Returns capabilities | 🔴 P0 | +| 4.2 | Model availability | `tools/list` | Provider models listed | 🔴 P0 | +| 4.3 | Content generation | Generate via OpenClaw API | Returns text completion | 🔴 P0 | +| 4.4 | Model switching | Switch model in Hermes admin → verify | Different model used | 🟡 P1 | + +--- + +## Tier 5: Falah OS CE + Docker Services + +**Owner: @hermes-contabo** + +| # | Test | Method | Pass Criteria | Priority | +|---|------|--------|--------------|----------| +| 5.1 | Docker build | `docker compose build` | Exit 0, no errors | 🔴 P0 | +| 5.2 | All services healthy | `docker compose up -d` + `/health` on each | All 3 services healthy | 🔴 P0 | +| 5.3 | Falah App UI | Load React app in browser | All modules visible (Zakat, Waqf, Qurban, Sadaqah, Infaq) | 🟡 P1 | +| 5.4 | iStore + Gitea connection | List available apps via iStore | Returns app list | 🟡 P1 | +| 5.5 | App Manager | Install → verify container → uninstall | Clean lifecycle | 🟡 P1 | +| 5.6 | Window manager | Drag, dock, context menu, search bar | All interactions work | 🟢 P2 | +| 5.7 | Casdoor auth | OIDC login → redirect → callback | Authenticated session | 🔴 P0 | + +--- + +## Tier 6: CI/CD Pipeline (Gitea Actions) + +**Owner: @hermes-contabo** + +| # | Test | Workflow | Pass Criteria | Priority | +|---|------|----------|--------------|----------| +| 6.1 | PHP lint | `ci.yml` | All `.php` files parse clean | 🔴 P0 | +| 6.2 | Banned function check | `ci.yml` | No `exec/shell_exec/system/eval` hits | 🔴 P0 | +| 6.3 | Secret scan | `ci.yml` | No hardcoded passwords/tokens | 🔴 P0 | +| 6.4 | File permissions check | `ci.yml` | No world-writable files outside `.git` | 🟡 P1 | +| 6.5 | Schema validation | `ci.yml` | `hermes/v1` namespace declared, all classes in autoloader | 🔴 P0 | +| 6.6 | cPanel deploy | `deploy.yml` | Git push to cPanel succeeds | 🔴 P0 | +| 6.7 | WP API fallback | `deploy.yml` | Plugin zip upload works when cPanel fails | 🟡 P1 | +| 6.8 | Post-deploy verify | `deploy.yml` | Namespace + plugin presence on target | 🔴 P0 | +| 6.9 | Health check | `health.yml` (every 30m) | REST endpoint responds | 🟡 P1 | +| 6.10 | Sync | `sync.yml` (every 6h) | cPanel sync + Nextcloud artifacts | 🟢 P2 | + +--- + +## Tier 7: End-to-End Integration + +**Owner: @Ody @OC @PI @hermes-contabo** + +| # | Scenario | Steps | Expected | Priority | +|---|----------|-------|----------|----------| +| 7.1 | **Full content pipeline** | Dev push → CI → Deploy → WP → Hermes → Ody → OC → WP post | Post published with AI content | 🔴 P0 | +| 7.2 | **Scheduled content** | Schedule via API → WP-Cron tick → Ody → OC → Post | Post created at correct time | 🔴 P0 | +| 7.3 | **Nextcloud sync** | Upload file → Hermes WebDAV → Download | Round-trips successfully | 🟡 P1 | +| 7.4 | **Cloudflare WAF** | Call Hermes API through CF tunnel | Request passes, `ody.falahos.my` reachable | 🔴 P0 | +| 7.5 | **Secret rotation** | Regenerate API key → old calls fail → new calls work | Zero downtime on rotation | 🔴 P0 | +| 7.6 | **Container lifecycle** | App Manager install → run → uninstall → verify | No orphaned containers | 🟡 P1 | +| 7.7 | **Load test** | 50 concurrent content generation requests | No crash, graceful degradation | 🟡 P1 | + +--- + +## 🚀 Pre-Deploy Smoke Suite (< 5 min) + +Run before every production push: + +```bash +# 1. Docker stack +docker compose build && docker compose up -d +for port in 3005 3021 3022; do + curl -sf "http://localhost:$port/health" || echo "FAIL: port $port" +done + +# 2. Hermes API health +curl -sf -H "X-API-Key: $HERMES_KEY" \ + "https://ummah.falahos.my/wp-json/hermes/v1/health" \ + | grep '"status":"ok"' || echo "FAIL: Hermes health" + +# 3. Content generation +curl -sf -X POST -H "X-API-Key: $HERMES_KEY" \ + -H "Content-Type: application/json" \ + -d '{"prompt":"Test: Write 3 sentences about zakat"}' \ + "https://ummah.falahos.my/wp-json/hermes/v1/content/generate" \ + | grep '"success":true' || echo "FAIL: Content gen" + +# 4. MCP - Ody +echo '{"jsonrpc":"2.0","id":1,"method":"initialize"}' | \ + websocat ws://mac-mini.local:7000/mcp || echo "FAIL: Ody MCP" + +# 5. MCP - OC +echo '{"jsonrpc":"2.0","id":1,"method":"initialize"}' | \ + websocat ws://mac-mini.local:4000/mcp || echo "FAIL: OC MCP" +``` + +--- + +## 📊 Test Result Tracking + +| Suite | Tests | Pass | Fail | Blocked | Coverage | +|-------|-------|------|------|---------|----------| +| Plugin Core | 9 | — | — | — | — | +| Ody Agent | 6 | — | — | — | — | +| PI Agent | 4 | — | — | — | — | +| OC Provider | 4 | — | — | — | — | +| Docker/Frontend | 7 | — | — | — | — | +| CI/CD Pipeline | 10 | — | — | — | — | +| E2E Integration | 7 | — | — | — | — | +| **Total** | **47** | — | — | — | — | + +--- + +## ⚠️ Known Blockers (Pre-QA) + +| # | Issue | Owner | Status | +|---|-------|-------|--------| +| B1 | Cloudflare WAF blocks Contabo from WP REST API | @hermes-contabo | Open | +| B2 | WordPress Application Password needed for deploy | @Ody | Open | +| B3 | Openclaw MCP server not running (port 4000 closed) | @OC | Open | +| B4 | Odysseus MCP server not running (port 7000 closed) | @Ody | Open | + +--- + +## 🛠 Testing Tools + +| Tool | Used For | +|------|----------| +| PHPUnit + WP_Mock | PHP unit tests | +| Postman / Bruno | REST API collection + automation | +| MCP Inspector | Ody & PI MCP tool testing | +| `docker compose ps + /health` | Service health verification | +| Playwright | React UI + window manager tests | +| Gitea Actions logs | CI/CD failure inspection | +| k6 | Load testing agent endpoints | +| trufflehog | Secret leak detection | + +--- + +*Page auto-generated by @hermes-contabo · Edit at `https://git.falahos.my/wmj/hermes-cpanel-agent/wiki/QA-Test-Strategy`*