# 1. Executive diagnosis The smallest reliable fix is **not** to redesign the watcher, Drive, CI, coding-agent, and webhook stack. The failure is primarily a **completion-semantics problem**: > The workflow currently treats **executor success** as **task success**. Those are not equivalent. The recovery architecture should introduce three things around the existing loop: 1. **Persistent task state with bounded recovery counters** 2. **Independent post-execution verification of expected outputs** 3. **A recovery controller that chooses Levels 0–7 instead of immediately completing, endlessly retrying, or asking a human** The intended success condition should become: **SUCCESS = execution finished + expected observable state exists + verification passes + no unresolved approval-required action remains.** For the recent incident: * Dispatch succeeded. * Executor returned success. * Expected artifact was absent. * Therefore the correct state was **VERIFY_FAILED**, not **COMPLETE**. * The system should then have entered Level 2 or Level 3 recovery depending on whether the artifact location/specification was valid. This architecture can be added around the existing workflow without replacing Google Drive, automation workflows, GitLab CI, coding agents, webhooks, or scheduled jobs. **Readiness today: READY WITH CONDITIONS.** The critical condition is that `COMPLETE` must become unreachable except through an independent verification gate. --- # 2. Current loop map Current effective loop: ```text Cloud task file changes ↓ Watcher detects change ↓ Create intake item ↓ Dispatch coding agent ↓ Wait for executor ↓ Executor returns success ↓ Mark COMPLETE ``` The problematic edge is: ```text EXECUTOR_SUCCESS ──────────────> COMPLETE ``` Replace only that portion with: ```text EXECUTOR_SUCCESS ↓ OBSERVE EXTERNAL STATE ↓ VERIFY EXPECTED RESULT ├── verified ───────────────→ COMPLETE ├── transient failure ──────→ RETRY / REFRESH ├── recoverable mismatch ──→ CORRECT / ALTERNATE PATH ├── partial result ─────────→ PARTIAL + CONTINUE ├── approval required ─────→ HUMAN_GATE └── exhausted/unsafe ──────→ ESCALATE / STOP ``` The target loop is therefore: ```text TRIGGER ↓ GOAL NORMALIZATION ↓ PLAN ↓ ACT ↓ OBSERVE ↓ VERIFY ├─ SUCCESS ├─ RETRY ├─ RECOVER ├─ ESCALATE └─ STOP ↓ STATE UPDATE ↓ NEXT CYCLE ``` A task should also have an explicit **lease/ownership state** so two watcher executions cannot simultaneously process the same task. --- # 3. Failure-mode table | Failure mode | Type | Typical symptom | False-success risk | Recovery | Hard stop? | | ------------------------------------------ | ------------------------------ | ------------------------------------- | -------------------------- | ------------------------------------------- | -------------------------------- | | Executor temporarily unavailable | Transient | Timeout / 502 / runner offline | Low | L1 → L3 → L4 | After bounded exhaustion | | Coding agent waits for input | Usually structural/interaction | Job alive but no progress | High | Detect stall → L2/L5/L6 | If required answer is human-only | | Watcher misses Drive change | Structural/transient | Task remains unseen | High | Reconciliation sweep | No | | Duplicate Drive event | Transient | Same task dispatched twice | High | Fingerprint + idempotency key | No | | Authentication expired | Transient | 401/403/token error | Low | L3 refresh auth | If re-auth requires human | | Executor says success, artifact missing | Structural/result mismatch | `exit=0`, no output | **Critical** | L2/L3/L4 | After bounded recovery | | Artifact created in wrong folder | Structural | Expected path empty | High | Search by task ID/hash → L2 | No | | Artifact exists but is empty/corrupt | Structural | File exists, unusable | High | Verification fails → L2/L4 | Possibly | | Webhook lost | Transient | Executor finished, controller unaware | Medium | Poll/reconciliation | No | | Executor crashes after partial work | Transient | Partial files/checkpoint exist | Medium | Resume from checkpoint | No | | Same bad action repeated forever | Control failure | Infinite retry loop | High | Attempt budgets + recovery level escalation | Yes | | Source task changes during execution | Structural/concurrency | Output based on stale input | High | Compare source revision/hash | Potentially | | Coding agent attempts external send/change | Risk-boundary | Requested action crosses gate | Critical | Pause only gated action | Yes for that action | | Verification service unavailable | Transient | Cannot inspect output | High if completion allowed | Keep `VERIFY_PENDING` | No completion | | Expected-output definition absent | Structural ambiguity | Nothing objective to verify | **Critical** | L5/L6 | Yes before COMPLETE | The most dangerous failure classes are therefore not explicit errors. They are **false positives**: successful API calls, successful CI jobs, or agent statements that do not correspond to the intended external state. --- # 4. Proposed recovery loop The minimum recovery controller can sit after the existing intake/dispatch mechanism. ```text TRIGGER │ ├─ Deduplicate using task_id + source_revision/hash │ ▼ GOAL NORMALIZATION │ ├─ Extract: │ task objective │ expected artifacts/state │ prohibited actions │ verification method │ completion criteria │ ▼ PLAN │ ├─ Select inexpensive primary execution path ├─ Record plan_version ├─ Record expected_output_manifest │ ▼ ACT │ ├─ Acquire lease ├─ Dispatch with idempotency key ├─ Persist execution/run ID before waiting │ ▼ OBSERVE │ ├─ Executor status ├─ heartbeat/progress ├─ produced files/results ├─ source revision │ ▼ VERIFY │ ├─ Artifact/state actually exists? ├─ Correct location? ├─ Correct revision/task ID? ├─ Non-empty / parseable? ├─ Required content or invariant satisfied? ├─ Approval boundary respected? │ ├── PASS → SUCCESS │ ├── FAIL recoverable → recovery ladder │ ├── UNKNOWN → never COMPLETE │ └── unsafe → STOP / HUMAN GATE │ ▼ STATE UPDATE │ ├─ checkpoint ├─ attempt counters ├─ evidence ├─ timestamps └─ next_allowed_action │ ▼ NEXT CYCLE ``` ### Required state distinction Do not collapse everything into `PENDING/RUNNING/COMPLETE/FAILED`. Use at least: ```text NEW NORMALIZED READY DISPATCHED RUNNING STALLED EXECUTION_SUCCEEDED VERIFY_PENDING VERIFY_FAILED RECOVERING PARTIAL HUMAN_GATE COMPLETE FAILED_RECOVERABLE FAILED_FINAL STOPPED_SAFE ``` `EXECUTION_SUCCEEDED` must be explicitly different from `COMPLETE`. ### Stall detection A run is stalled when any one of these occurs: ```text No heartbeat beyond configured timeout No observable state/progress change for N observation windows Agent emits an input-request marker Executor remains "running" beyond task-specific ceiling Same tool/action/result tuple repeats ``` A stall does not immediately require human intervention. First determine whether the requested input can be derived from: * task file, * existing state, * previous executor output, * cloud files, * repository state, * execution logs, * safe defaults explicitly defined by policy. Only genuinely non-derivable or approval-requiring input becomes a Human Gate. --- # 5. Retry policy Use the requested ladder as a monotonic escalation path. | Level | Trigger | Action | Max attempts | Verification | Preserved state | Next | | --------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------- | -----------------: | --------------------------------------- | --------------------------------------- | ------------------------------ | | **L0 Normal execution** | New runnable task | Existing execution path | 1 active run | Full post-action verify | All IDs, plan, source hash | Success or L1 | | **L1 Retry same action** | Timeout, temporary network/service error | Repeat identical safe action | 2 | Same external verifier | Run history + idempotency key | Success or L2/L3 | | **L2 Corrected parameters** | Invalid path, stale parameter, agent recoverable mistake, missing output | Correct path/input/instruction and retry | 2 | Artifact/state verifier | Original + corrected params | Success or L3 | | **L3 Refresh state/context/auth** | Stale Drive view, expired auth, stale repository/context | Re-read task, refresh auth/session, repository state, runner status | 2 | Verify freshness + retry result | Previous checkpoint and source revision | Success or L4 | | **L4 Alternate execution path** | Primary executor/tool unavailable or repeatedly inconsistent | Alternate runner, CI path, agent/tool | 2 paths × 1 | Same canonical verification | Same task ID/output manifest | Success or L5 | | **L5 Reduce scope** | Full task blocked but safe subset can progress | Produce verified partial output/checkpoint | 2 scope reductions | Verify each partial artifact | Completed substeps + unresolved set | Success, next cycle, or L6 | | **L6 Preserve + escalate** | Structural ambiguity, exhausted safe recovery, approval needed | Freeze mutations, package evidence, request minimal human decision | 1 escalation event | Human decision or newly available state | Everything | Resume appropriate level or L7 | | **L7 Safe shutdown** | Unsafe action, unrecoverable inconsistency, repeated systemic failure | Release lease, preserve state, stop automation for task | 1 | Confirm no active mutating run remains | Full forensic bundle | Manual/repaired restart only | ### Global retry ceilings In addition to per-level limits: ```text Maximum executor launches per task: 7 Maximum identical action attempts: 3 total Maximum recovery-level regressions: 1 Maximum task wall-clock recovery window: configurable by task class Maximum model escalations: 2 ``` A higher recovery level must not silently fall back into an unlimited lower-level loop. For example: ```text L4 → L1 → L2 → L3 → L4 → L1 ... ``` must be forbidden. Maintain `highest_recovery_level_reached`; once a level has exhausted its budget, it remains exhausted until materially new external state appears. --- # 6. Verification gates Verification must test **observable results**, not agent confidence. ### Gate A — trigger integrity Before dispatch: ```text task_file_exists = true source_revision/hash = captured task_id = stable duplicate_active_run = false ``` ### Gate B — goal completeness Before execution: ```text objective != null success_criteria != null verification_method != null approval_boundaries identified ``` If an artifact is expected: ```text expected_output_manifest = [ { type, path_or_locator, minimum_conditions, verification_method } ] ``` ### Gate C — execution observation Executor success is evidence only: ```text executor_exit_status = successful ``` This permits transition to: ```text EXECUTION_SUCCEEDED → VERIFY_PENDING ``` It does **not** permit: ```text EXECUTION_SUCCEEDED → COMPLETE ``` ### Gate D — external artifact verification For a file output: ```text file exists AND expected location or canonical locator resolves AND size > minimum AND modified after execution began AND task identifier/revision matches AND file can be opened/parsed where applicable AND required content/invariants pass ``` For a repository change: ```text expected commit/diff exists AND based on correct source revision AND tests/checks pass ``` For an automation change: ```text configuration exists AND enabled/disabled state matches intended state AND next execution path can be resolved ``` ### Gate E — stale-input verification Immediately before completion: ```text current_source_revision == execution_source_revision ``` If not: ```text VERIFY_FAILED(reason="SOURCE_CHANGED_DURING_EXECUTION") ``` Then re-normalize rather than completing stale work. ### Gate F — completion gate Only: ```text all_required_verifications == PASS AND no_required_approval_pending AND no_active_child_execution ``` may write: ```text state = COMPLETE ``` Unknown verification must be treated as `UNKNOWN`, never as `PASS`. --- # 7. Human Gate matrix The goal is to gate the **specific risky action**, not freeze the entire task unnecessarily. | Action | Autonomous? | Gate | | ------------------------------------- | ----------: | -------------- | | Read Drive/task files | Yes | None | | Search files/logs/repos | Yes | None | | Normalize task | Yes | None | | Draft code/content | Yes | None | | Create temporary artifact | Yes | None | | Validate outputs | Yes | None | | Retry jobs | Yes | None | | Refresh executor state | Yes | None | | Refresh non-interactive auth token | Yes | None | | Switch executor/tool | Yes | None | | Re-run safe CI job | Yes | None | | Create checkpoint | Yes | None | | Create non-published working files | Yes | None | | Reduce task scope | Yes | None | | External send/message | **No** | Human approval | | Public/external publishing | **No** | Human approval | | Delete files/data | **No** | Human approval | | Change permissions/access | **No** | Human approval | | Payment/financial transaction | **No** | Human approval | | Accept/sign contract | **No** | Human approval | | Destructive production operation | **No** | Human approval | | Irreversible source-of-truth mutation | **No** | Human approval | A task containing both safe and gated work should behave like: ```text prepare → validate → checkpoint ↓ HUMAN_GATE ↓ approved action ``` not: ```text task detected → HUMAN_GATE → nothing else happens ``` This substantially reduces unnecessary intervention. --- # 8. Cost/model routing policy Routine reliability should come from deterministic controls rather than expensive reasoning. ### Tier 0 — deterministic logic Use for: * file hashes, * task deduplication, * artifact existence, * path checks, * timestamps, * state transitions, * attempt budgets, * schema validation, * Git/CI state, * heartbeat monitoring. This should resolve most failures without an LLM. ### Tier 1 — inexpensive model Use for: * task normalization, * extracting success criteria, * simple coding-agent prompts, * classification of known failure signatures, * choosing among predefined recovery actions. ### Tier 2 — standard capable model Escalate when: ```text 2 unsuccessful corrected attempts OR conflicting observations OR agent repeatedly requests input OR verification failure requires semantic inspection ``` Use for re-planning and structural diagnosis. ### Tier 3 — strongest model Use only when: ```text multiple tools disagree OR task intent is structurally ambiguous OR lower-tier plans repeatedly fail OR recovery requires deep code/repository reasoning ``` A stronger model must **not** bypass approval boundaries. Suggested rule: ```text model escalation ≠ action permission escalation ``` A more capable model gains reasoning scope, not additional authority. --- # 9. State/memory schema A single persistent task record is sufficient. ```json { "task_id": "stable-id", "idempotency_key": "task-id:source-revision", "source": { "file_id": "...", "revision": "...", "hash": "...", "modified_at": "..." }, "goal": { "normalized_objective": "...", "success_criteria": [], "expected_outputs": [], "risk_class": "...", "approval_requirements": [] }, "state": "VERIFY_PENDING", "plan": { "version": 3, "selected_executor": "...", "steps": [] }, "execution": { "run_id": "...", "started_at": "...", "last_heartbeat_at": "...", "reported_status": "...", "reported_result": "..." }, "verification": { "status": "FAIL", "checked_at": "...", "evidence": [], "failures": [] }, "recovery": { "current_level": 2, "highest_level": 2, "attempts_by_level": { "0": 1, "1": 1, "2": 1 }, "total_executor_launches": 3, "last_failure_signature": "...", "same_failure_count": 2 }, "lease": { "owner": "...", "expires_at": "..." }, "checkpoint": { "completed_steps": [], "partial_artifacts": [], "remaining_steps": [] }, "human_gate": { "required": false, "reason": null, "requested_action": null }, "timestamps": { "created_at": "...", "updated_at": "...", "next_check_at": "..." } } ``` Critical persistent fields are: ```text task_id source revision/hash state expected outputs executor run ID verification evidence recovery counters last failure signature checkpoint approval status lease ``` They must survive workflow restarts. --- # 10. Logging plan Use event-style logs rather than one mutable status string. Example: ```text TASK_DETECTED TASK_NORMALIZED LEASE_ACQUIRED PLAN_CREATED DISPATCH_STARTED DISPATCH_ACCEPTED EXECUTOR_HEARTBEAT EXECUTOR_REPORTED_SUCCESS VERIFICATION_STARTED ARTIFACT_NOT_FOUND RECOVERY_LEVEL_CHANGED PARAMETERS_CORRECTED REDISPATCHED ARTIFACT_FOUND VERIFICATION_PASSED TASK_COMPLETED ``` Every event should record: ```text timestamp task_id source_revision workflow_run_id executor_run_id state_before state_after recovery_level attempt_number action tool/path result classification failure signature verification evidence reference cost/model tier human_gate status ``` For agent interactions, retain: ```text prompt hash/version agent response classification whether input was requested reasoning-independent action/result summary ``` Do not require verbose chain-of-thought logs. ### Essential reliability metrics Track: ```text false-success count verification failure rate tasks stalled > threshold recovery success by level average attempts/task duplicate dispatch prevented missed-event tasks recovered by reconciliation human gates/task executor availability auth refresh failures cost/task ``` The primary quality metric should be: ```text Verified Completion Rate = verified completed tasks / tasks marked complete ``` The target must effectively be **100%**, because an unverified task must not be marked complete. --- # 11. Failure-injection tests At minimum run these tests before trusting autonomous completion. | Test | Injection | Expected result | | ---------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | | 1. Missing artifact | Executor returns success but creates nothing | `VERIFY_FAILED`; never COMPLETE; recovery begins | | 2. Executor outage | Primary executor unavailable | L1 bounded retry → L3 refresh → L4 alternate path | | 3. Agent input wait | Agent asks “Which directory?” despite directory existing in task context | Controller resolves context autonomously and resumes | | 4. True human decision | Agent reaches external-send step | Safe work preserved; only send action enters HUMAN_GATE | | 5. Missed Drive event | Disable webhook/change event | Periodic reconciliation detects unprocessed revision | | 6. Duplicate event | Send same task event 5 times | Exactly one active execution due to idempotency/lease | | 7. Expired authentication | Force Drive/GitLab token expiration | L3 refresh; resume from checkpoint; no duplicate task | | 8. Wrong artifact location | Agent produces result in alternate folder | Verification searches deterministic candidate locations/task ID; corrects or redispatches | | 9. Empty artifact | Create zero-byte expected file | Verification fails despite file existence | | 10. Corrupt artifact | Expected JSON/file exists but cannot parse | Verification fails; L2/L4 recovery | | 11. Lost executor webhook | Job completes but completion callback dropped | Observation/reconciliation retrieves actual executor status | | 12. Source changed mid-run | Edit task after dispatch | Original output rejected as stale; task re-normalized | | 13. Infinite failure pattern | Force same deterministic error repeatedly | Retry ceiling reached; transitions to L6, not loop | | 14. Alternate path failure | Primary and secondary executors both unavailable | L5 partial progress or L6 preservation/escalation | | 15. Verification outage | Artifact verifier unavailable | Task stays VERIFY_PENDING; never COMPLETE | | 16. Controller restart | Kill workflow mid-execution | Restart reloads run ID/state and observes existing run rather than blindly redispatching | A release should fail if any test can produce: ```text COMPLETE without verification evidence ``` --- # 12. Implementation backlog with acceptance criteria and rollback ### P0 — Split executor success from task completion **Change** Replace: ```text executor_success → COMPLETE ``` with: ```text executor_success → EXECUTION_SUCCEEDED → VERIFY_PENDING ``` **Acceptance** No workflow path can write `COMPLETE` without a verification record containing `PASS`. **Rollback** Feature flag can restore legacy state handling, but during rollback mark legacy completions as `UNVERIFIED_COMPLETE` rather than silently asserting verification. --- ### P0 — Output manifest For each task normalization, persist expected observable outputs. Example: ```text artifact: type: file location: /expected/path/output.md required: true min_bytes: 1 ``` **Acceptance** Every executable task has either: * at least one objective verifier, or * explicit `NO_EXTERNAL_ARTIFACT` verification logic. **Rollback** Retain manifest fields but permit manual legacy execution; do not delete stored manifests. --- ### P0 — Independent verifier Implement post-execution verification independent of the executor's self-report. **Acceptance** Simulated executor `SUCCESS` plus missing artifact returns `VERIFY_FAILED`. **Rollback** Disable verifier execution but preserve all verification state/evidence. --- ### P1 — Persistent recovery state Add recovery level and attempt counters. **Acceptance** Controller restart does not reset retries or accidentally exceed budgets. **Rollback** Ignore new recovery fields while retaining them for re-enable. --- ### P1 — Idempotency and lease Use: ```text idempotency_key = task_id + source_revision ``` **Acceptance** Repeated watcher events produce one active execution. **Rollback** Disable lease acquisition while preserving event deduplication records. --- ### P1 — Reconciliation scanner Do not rely solely on webhooks/change events. Periodically compare: ```text cloud task revisions vs known intake/task revisions ``` **Acceptance** A deliberately suppressed Drive event is eventually detected and processed exactly once. **Rollback** Disable scheduled scan; watcher remains untouched. --- ### P1 — Stall detector Detect no-progress and agent-input waits. **Acceptance** A stalled agent automatically enters a recovery state rather than remaining RUNNING indefinitely. **Rollback** Disable stall transition but retain heartbeat telemetry. --- ### P2 — Alternate execution route Map primary and fallback executor paths. Example: ```text Primary: existing coding-agent executor Fallback 1: GitLab CI executor Fallback 2: secondary coding agent / runner ``` **Acceptance** Primary-path outage can complete a test task through the fallback with the same verifier. **Rollback** Disable fallback routing. --- ### P2 — Partial-progress checkpoints Allow L5 to preserve completed safe work. **Acceptance** Recovery never repeats already verified expensive work unnecessarily. **Rollback** Ignore checkpoints and rerun whole task, without deleting them. --- ### P2 — Minimal Human Gate packet When L6 is required, present only: ```text what succeeded what failed what was attempted what evidence conflicts exact decision/action required what will resume automatically afterward ``` **Acceptance** Human can resolve the gate with one narrow decision rather than diagnosing the workflow manually. --- # 13. Readiness ## READY WITH CONDITIONS The existing architecture is fundamentally usable. The watcher, Drive, automation workflows, GitLab CI, coding agents, and webhooks do not need wholesale replacement. However, autonomous completion should not be considered fully reliable until these four conditions exist: 1. **Executor success is separated from task success.** 2. **Independent external verification controls `COMPLETE`.** 3. **Retries and recovery levels have persistent bounded counters.** 4. **A reconciliation mechanism exists for missed events and lost callbacks.** The recent incident demonstrates why condition #1 and #2 are P0 rather than optional improvements. Once those are implemented and the missing-artifact test passes, the system can safely make substantially more progress without unnecessary Human Gates. --- # 14. Next autonomous action **Autonomous / no human input required.** The first recovery action should be a **read-only reconciliation pass**, not another system redesign: ```text 1. Find recently COMPLETE tasks. 2. Reconstruct each task's expected output artifact/state. 3. Verify the output independently. 4. For any COMPLETE task without verified output: COMPLETE → VERIFY_FAILED 5. Preserve its original executor run ID and logs. 6. Enter the recovery ladder at: L2 if the expected output/path is known, L3 if state/auth/context may be stale, L6 if no objective success condition can be reconstructed. 7. Do not external-send, delete, change permissions, spend money, alter contracts, or make irreversible production changes. ``` For the specific recent incident: ```text Current classification: EXECUTION_SUCCEEDED → VERIFY_FAILED: EXPECTED_ARTIFACT_MISSING Correct next transition: L2 — validate/correct output target and rerun the artifact-producing step If still absent: L3 — refresh task/source/executor state If still absent: L4 — alternate executor If artifact production can only partly continue: L5 — checkpoint partial progress Only after independent artifact verification: COMPLETE ``` That single change—**making verified external state the authority for completion rather than executor self-report**—removes the highest-risk failure mode while preserving nearly all of the current workflow.