wmj/hermes-cpanel-agent
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity
Page: QA Test Strategy
Pages
Architecture Overview Cloudflare WAF Bypass Discovery Critical Lesson: Never Rename a Production Database via cPanel MySQL API Getting Started Gitea Actions CICD Home QA Test Strategy
Clone
1
QA Test Strategy
Hermes Bot edited this page 2026-07-09 11:39:17 +08:00
Table of Contents

QA Test Strategy — Falah OS Ecosystem

Status: Active · Owner: @hermes-contabo · Last Updated: 2026-07-09

Tagged Agents: @Ody @OC @PI @hermes-contabo

Ecosystem Overview

┌─────────────────────────────────────────────────────────────────┐
│                    Falah OS CE (AWS 13.140.161.244)            │
│  ┌──────────┐  ┌──────────┐  ┌──────────────┐  ┌───────────┐  │
│  │ Falah    │  │ iStore   │  │ App Manager  │  │ Gateway   │  │
│  │ App:3005 │  │ :3021    │  │ :3022        │  │ :3000     │  │
│  │ (React)  │  │(registry)│  │(docker apps) │  │           │  │
│  └──────────┘  └──────────┘  └──────────────┘  └───────────┘  │
└─────────────────────────────────────────────────────────────────┘
                           │
                    Cloudflare Tunnel
                           │
┌─────────────────────────────────────────────────────────────────┐
│                 Hermes Plugin + WordPress (Ummah)               │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  Hermes AI Bridge Plugin                                 │   │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────────┐  │   │
│  │  │REST API  │ │AI Prov.  │ │Scheduler │ │Nextcloud   │  │   │
│  │  │hermes/v1 │ │(OpenClaw)│ │(WP-Cron) │ │Bridge      │  │   │
│  │  └──────────┘ └──────────┘ └──────────┘ └────────────┘  │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
                           │
              ┌────────────┴────────────┐
              ▼                         ▼
      ┌──────────────┐        ┌──────────────────┐
      │ Ody Clone    │        │ PI Clone         │
      │ (Content Gen)│        │ (Auto-Reply)     │
      │ port 7000    │        │ (in WordPress)   │
      └──────────────┘        └──────────────────┘
              │                         │
              └──────────┬──────────────┘
                         ▼
               ┌──────────────────┐
               │ OpenClaw / OC    │
               │ (AI Provider)    │
               │ Mac Mini :4000   │
               └──────────────────┘

Components Under Test

Component Agent Location Type Priority
Hermes WordPress Plugin @hermes-contabo cPanel / WordPress PHP Plugin P0
Ody Clone (Content Gen) @Ody Mac Mini :7000 MCP Server P0
PI Clone (Auto-Reply) @PI WordPress Agent P1
OpenClaw Provider @OC Mac Mini :4000 MCP Server P0
Falah OS CE Frontend @hermes-contabo AWS 13.140.161.244 React + Express P1
iStore App Registry @hermes-contabo Docker :3021 Express Service P1
App Manager @hermes-contabo Docker :3022 Express Service P1
Gitea Actions CI/CD @hermes-contabo Contabo Pipeline P0

Tier 1: Hermes Plugin (Core Runtime)

Owner: @Ody @OC

# Test Method Pass Criteria Priority
1.1 Plugin activation Activate → check DB tables, options, cron hermes_log table exists, hermes_api_key set 🔴 P0
1.2 REST API auth Call with wrong/missing/valid X-API-Key 401 wrong → 401 missing → 200 valid 🔴 P0
1.3 All 12 REST routes Health, CRUD posts/users/settings/schedules/logs Each returns correct shape 🔴 P0
1.4 Content generation POST /content/generate with prompt Post created with AI content 🔴 P0
1.5 AI provider switching Switch OpenClaw ↔ OpenAI ↔ Anthropic API calls route to correct provider 🟡 P1
1.6 WP-Cron scheduling Schedule future content → tick → verify Post published at correct time 🔴 P0
1.7 Security: no banned funcs exec/shell_exec/system/eval grep Zero hits 🔴 P0
1.8 Admin settings UI Page renders, save works, key regen works All fields persist, new key invalidates old 🟡 P1
1.9 Logging All actions logged in hermes_log Entries present with correct levels 🟡 P1

API Contract Verification

hermes/v1:
  ├── GET    /health              → {"status":"ok","version":"1.0.0",...}
  ├── POST   /content/generate    → {"success":true,"post_id":N,"edit_url":"..."}
  ├── POST   /content/schedule    → {"success":true,"task_id":N,"scheduled_date":"..."}
  ├── GET    /posts               → [{id,title,status,date,modified,permalink}]
  ├── GET    /posts/:id           → {id,title,content,excerpt,status,date,...}
  ├── PUT    /posts/:id           → {"success":true,"post_id":N}
  ├── DELETE /posts/:id           → {"success":true,"deleted":N}
  ├── GET    /users               → [{ID,login,email,display_name,roles}]
  ├── GET    /users/:id           → {id,login,email,display_name,roles}
  ├── GET    /settings            → {site_name,ai_provider,ai_model,...}
  ├── PUT    /settings            → {"success":true}
  ├── GET    /schedules           → [{id,type,scheduled_date,status,created}]
  └── GET    /logs?level=&limit=  → [{id,level,message,context,created_at}]

Tier 2: Ody Clone (Content Generation Agent)

Owner: @Ody @OC

# Test Method Pass Criteria Priority
2.1 MCP connectivity JSON-RPC initialize on mac-mini.local:7000 Returns serverInfo + capabilities 🔴 P0
2.2 Tool discovery tools/list after init Exposes content-gen tools 🔴 P0
2.3 Content generation Call MCP content_generate tool Returns markdown content 🔴 P0
2.4 Content quality Generate 10 Islamic finance posts No haram content, correct terminology 🟡 P1
2.5 Error handling Missing API key, rate limits, timeouts Structured error responses 🔴 P0
2.6 Prompt injection Malicious prompt payloads Prompt boundaries respected, no system prompt leak 🔴 P0

MCP Tool Contract

tools:
  content_generate:
    input:  {prompt: string, tone?: string, length?: number}
    output: {content: string, title?: string, word_count: number}
  
  content_schedule:
    input:  {prompt: string, schedule_date: string, post_type?: string}
    output: {task_id: number, scheduled_date: string}
  
  health_check:
    input:  {}
    output: {status: string, uptime: number, model: string}

Tier 3: PI Clone (Auto-Reply Agent)

Owner: @PI

# Test Method Pass Criteria Priority
3.1 Comment auto-reply Create comment on WordPress post PI replies within cron window 🟡 P1
3.2 Reply relevance Test comments about Islamic finance Reply is contextually appropriate 🟡 P1
3.3 Spam guard Spam comment → PI should not reply No reply generated for spam 🟡 P1
3.4 Rate limiting 50 comments in 1 min → PI behavior Does not reply to all, respects limits 🟢 P2

Tier 4: OpenClaw Provider (AI Inference)

Owner: @OC

# Test Method Pass Criteria Priority
4.1 MCP connectivity JSON-RPC initialize on mac-mini.local:4000 Returns capabilities 🔴 P0
4.2 Model availability tools/list Provider models listed 🔴 P0
4.3 Content generation Generate via OpenClaw API Returns text completion 🔴 P0
4.4 Model switching Switch model in Hermes admin → verify Different model used 🟡 P1

Tier 5: Falah OS CE + Docker Services

Owner: @hermes-contabo

# Test Method Pass Criteria Priority
5.1 Docker build docker compose build Exit 0, no errors 🔴 P0
5.2 All services healthy docker compose up -d + /health on each All 3 services healthy 🔴 P0
5.3 Falah App UI Load React app in browser All modules visible (Zakat, Waqf, Qurban, Sadaqah, Infaq) 🟡 P1
5.4 iStore + Gitea connection List available apps via iStore Returns app list 🟡 P1
5.5 App Manager Install → verify container → uninstall Clean lifecycle 🟡 P1
5.6 Window manager Drag, dock, context menu, search bar All interactions work 🟢 P2
5.7 Casdoor auth OIDC login → redirect → callback Authenticated session 🔴 P0

Tier 6: CI/CD Pipeline (Gitea Actions)

Owner: @hermes-contabo

# Test Workflow Pass Criteria Priority
6.1 PHP lint ci.yml All .php files parse clean 🔴 P0
6.2 Banned function check ci.yml No exec/shell_exec/system/eval hits 🔴 P0
6.3 Secret scan ci.yml No hardcoded passwords/tokens 🔴 P0
6.4 File permissions check ci.yml No world-writable files outside .git 🟡 P1
6.5 Schema validation ci.yml hermes/v1 namespace declared, all classes in autoloader 🔴 P0
6.6 cPanel deploy deploy.yml Git push to cPanel succeeds 🔴 P0
6.7 WP API fallback deploy.yml Plugin zip upload works when cPanel fails 🟡 P1
6.8 Post-deploy verify deploy.yml Namespace + plugin presence on target 🔴 P0
6.9 Health check health.yml (every 30m) REST endpoint responds 🟡 P1
6.10 Sync sync.yml (every 6h) cPanel sync + Nextcloud artifacts 🟢 P2

Tier 7: End-to-End Integration

Owner: @Ody @OC @PI @hermes-contabo

# Scenario Steps Expected Priority
7.1 Full content pipeline Dev push → CI → Deploy → WP → Hermes → Ody → OC → WP post Post published with AI content 🔴 P0
7.2 Scheduled content Schedule via API → WP-Cron tick → Ody → OC → Post Post created at correct time 🔴 P0
7.3 Nextcloud sync Upload file → Hermes WebDAV → Download Round-trips successfully 🟡 P1
7.4 Cloudflare WAF Call Hermes API through CF tunnel Request passes, ody.falahos.my reachable 🔴 P0
7.5 Secret rotation Regenerate API key → old calls fail → new calls work Zero downtime on rotation 🔴 P0
7.6 Container lifecycle App Manager install → run → uninstall → verify No orphaned containers 🟡 P1
7.7 Load test 50 concurrent content generation requests No crash, graceful degradation 🟡 P1

🚀 Pre-Deploy Smoke Suite (< 5 min)

Run before every production push:

# 1. Docker stack
docker compose build && docker compose up -d
for port in 3005 3021 3022; do
  curl -sf "http://localhost:$port/health" || echo "FAIL: port $port"
done

# 2. Hermes API health
curl -sf -H "X-API-Key: $HERMES_KEY" \
  "https://ummah.falahos.my/wp-json/hermes/v1/health" \
  | grep '"status":"ok"' || echo "FAIL: Hermes health"

# 3. Content generation
curl -sf -X POST -H "X-API-Key: $HERMES_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Test: Write 3 sentences about zakat"}' \
  "https://ummah.falahos.my/wp-json/hermes/v1/content/generate" \
  | grep '"success":true' || echo "FAIL: Content gen"

# 4. MCP - Ody
echo '{"jsonrpc":"2.0","id":1,"method":"initialize"}' | \
  websocat ws://mac-mini.local:7000/mcp || echo "FAIL: Ody MCP"

# 5. MCP - OC
echo '{"jsonrpc":"2.0","id":1,"method":"initialize"}' | \
  websocat ws://mac-mini.local:4000/mcp || echo "FAIL: OC MCP"

📊 Test Result Tracking

Suite Tests Pass Fail Blocked Coverage
Plugin Core 9 — — — —
Ody Agent 6 — — — —
PI Agent 4 — — — —
OC Provider 4 — — — —
Docker/Frontend 7 — — — —
CI/CD Pipeline 10 — — — —
E2E Integration 7 — — — —
Total 47 — — — —

⚠️ Known Blockers (Pre-QA)

# Issue Owner Status
B1 Cloudflare WAF blocks Contabo from WP REST API @hermes-contabo Open
B2 WordPress Application Password needed for deploy @Ody Open
B3 Openclaw MCP server not running (port 4000 closed) @OC Open
B4 Odysseus MCP server not running (port 7000 closed) @Ody Open

🛠 Testing Tools

Tool Used For
PHPUnit + WP_Mock PHP unit tests
Postman / Bruno REST API collection + automation
MCP Inspector Ody & PI MCP tool testing
docker compose ps + /health Service health verification
Playwright React UI + window manager tests
Gitea Actions logs CI/CD failure inspection
k6 Load testing agent endpoints
trufflehog Secret leak detection

Page auto-generated by @hermes-contabo · Edit at https://git.falahos.my/wmj/hermes-cpanel-agent/wiki/QA-Test-Strategy