1. Initial Workflow State Workflow ID: ORD-1042 Stage Status Validate order COMPLETED Charge payment IN PROGRESS Create order PENDING Send confirmation PENDING Important side effect: Charging the customer's payment method. The payment operation is treated as uncertain, not failed, because the API timed out after the request was sent. --- 2. Failure Injected Failure: > Payment API timeout after the charge request was submitted. The agent receives no definitive response. Observed facts Payment request was sent. API response was not received. Agent cannot determine whether payment succeeded. Order has not yet been created. Customer must not be charged twice. --- 3. Failure Classification Failure Detection Impact Retryable Recovery Strategy Escalation Payment API timeout Request timeout Payment outcome unknown No blind retry Verify payment state first If state cannot be verified Classification: PARTIAL_EXECUTION / UNKNOWN_EXTERNAL_STATE --- 4. Recovery Decision Engine The recovery logic should behave like this: IF payment request timed out THEN DO NOT retry payment immediately VERIFY payment state using available system capability IF payment is confirmed successful THEN mark payment COMPLETED continue to order creation ELSE IF payment is confirmed failed THEN mark payment FAILED safely retry payment only if the system's duplicate-prevention mechanism permits it ELSE mark payment state UNKNOWN stop side-effecting actions escalate for human review The key rule is: Timeout ≠ payment failure. --- 5. External State Verification The recovery process checks the payment provider. Case A — Payment succeeded Payment status → SUCCESS Recovery: PAYMENT_UNKNOWN ↓ VERIFY PAYMENT ↓ PAYMENT CONFIRMED ↓ MARK PAYMENT COMPLETE ↓ CREATE ORDER ↓ SEND CONFIRMATION ↓ RECOVERY COMPLETE The payment is not charged again. --- Case B — Payment failed Payment status → FAILED Recovery: PAYMENT_UNKNOWN ↓ VERIFY PAYMENT ↓ PAYMENT FAILED ↓ CHECK DUPLICATE-PREVENTION CONDITIONS ↓ RETRY PAYMENT ↓ CREATE ORDER The retry should only happen if the payment system provides an appropriate mechanism to ensure the original attempt did not succeed. --- Case C — Payment state cannot be verified Payment status → UNKNOWN Recovery: PAYMENT_UNKNOWN ↓ VERIFY PAYMENT ↓ STATE STILL UNKNOWN ↓ STOP PAYMENT RETRY ↓ HUMAN REVIEW This prevents a potentially successful payment from being duplicated. --- 6. Checkpoint Design Before the payment operation, the workflow should have a checkpoint similar to: workflow_id: ORD-1042 workflow_state: PAYMENT_IN_PROGRESS completed_actions: - order_validated pending_actions: - create_order - send_confirmation uncertain_actions: - payment_request state_version: 7 recovery_status: RECOVERY_PENDING The checkpoint must preserve enough information to determine what happened before resuming. --- 7. Recovery State Machine WORKFLOW_EXECUTING ↓ FAILURE_DETECTED ↓ CLASSIFY_FAILURE ↓ PAYMENT_OUTCOME_UNKNOWN ↓ VERIFY_EXTERNAL_STATE ↓ ┌────┴────┐ ↓ ↓ SUCCESS FAILED ↓ ↓ RESUME VALIDATE RETRY SAFETY ↓ RETRY ↓ RESUME If verification remains UNKNOWN: ↓ HUMAN_REVIEW ↓ RESOLVE / TERMINATE SAFELY --- 8. Failure Injection Test Initial state: Order validation = COMPLETE Payment = IN PROGRESS Order creation = PENDING Confirmation = PENDING Inject failure: Payment API response = TIMEOUT Expected detection: FAILURE_DETECTED Expected classification: PARTIAL_EXECUTION + UNKNOWN_EXTERNAL_STATE Expected recovery: DO NOT RETRY PAYMENT ↓ VERIFY PAYMENT STATE Expected final state if payment succeeded: Payment = COMPLETE Order = CREATED Confirmation = SENT Recovery = COMPLETE Expected final state if payment cannot be verified: Payment = UNKNOWN Order = NOT CREATED Confirmation = NOT SENT Recovery = HUMAN_REVIEW --- 9. Recovery Audit Trail The system should record: Workflow ID: ORD-1042 Failure: Payment API timeout Last confirmed state: Order validated Completed actions: Order validation Uncertain action: Payment request Pending actions: Order creation, confirmation Recovery strategy: External state verification Recovery result: Payment status confirmed Final state: Recovery complete Sensitive payment information should not be unnecessarily stored in the recovery log. --- 10. Expected Test Result PASS criteria: Payment timeout is classified as an uncertain outcome. Payment is not blindly retried. External payment state is checked first. A confirmed successful payment is not charged again. Order creation occurs only after payment state is sufficiently established. Unknown payment state triggers human escalation rather than unsafe retry. Recovery actions are recorded in the audit trail. This test specifically validates the most important recovery property: preventing duplicate payment when a side-effecting API times out.