# CUSTOMER PILOT DECISION REPORT EXTEND — CONDITIONAL Confidence: Moderate AI Customer Support Chatbot | E-commerce | 30-Day Pilot ## 01 — Executive Decision Decision: EXTEND, subject to authorization of a limited, measurable follow-up test. The pilot demonstrated useful automation capability but did not satisfy two of its three stated success criteria. A full rollout is not justified. - Strongest evidence: 420 of 600 tickets were reported as automatically resolved, meeting the 70% automation target. - Critical shortfalls: Answer accuracy reached 86% against 90%, while cost reduction reached 15% against 20%. - Key concern: 34 customer complaints require investigation before expansion. - Business implication: A proposed $2,000/month rollout commitment remains unjustified without validated net savings. - Immediate action: Review failed responses, complaints, cost calculations, and engineering capacity before approving a short extension. The available results justify investigating whether the remaining gaps are correctable, but they do not establish that improvement is technically or economically feasible.  Teks yang ditempel (1).txt ## 02 — Pilot Snapshot | Dimension | Finding | | ------------------ | --------------------------------------------------------------------------------------------- | | Product / Service | AI support chatbot | | Customer / Segment | E-commerce company | | Pilot Objective | Automate support tickets, maintain accuracy, reduce costs | | Pilot Scope | 600 support tickets | | Participants | Customer count and support team size not provided | | Evaluation Period | 30 days | | Intended Next Step | Potential wider rollout | | Evidence Coverage | Three reported KPIs, complaint count, and rollout constraints; underlying records unavailable | ## 03 — Success Criteria Review Automation # 70% MET Target 70% Answer accuracy # 86% NOT MET Target 90% Cost reduction # 15% NOT MET Target 20% Performance against targets Target Observed 0%25%50%75%100%AutomationAccuracyCost savings Values are reported percentages; measurement methods were not supplied. | Criterion | Gap | Evidence status | | ---------------------- | -------------------- | -------------------------------------------------------------- | | Automatic resolution | 0 percentage points | Derived from 420 ÷ 600, assuming consistent ticket definitions | | Answer accuracy | −4 percentage points | Reported result; validation method unknown | | Support cost reduction | −5 percentage points | Reported result; cost baseline and methodology unknown | Two of the three stated targets were missed. No original acceptance rule was provided specifying whether partial achievement is permissible. ## 04 — Adoption & Value Assessment ### Customer Adoption CONFIRMED from supplied information: The chatbot reportedly handled 600 tickets and automatically resolved 420. UNKNOWN: Repeat usage, customer-level adoption, abandonment, escalation frequency, reopened tickets, and whether automated resolutions were independently verified. Ticket volume demonstrates operational exposure, not sustained user adoption. ### Demonstrated Value The results suggest that automation contributed to support operations and coincided with a reported 15% cost reduction. However, the available information cannot independently establish: - Whether resolved tickets stayed resolved. - Whether reported cost savings include all implementation and operating expenses. - Whether other operational changes contributed to the savings. - Whether the 34 complaints relate to incorrect answers, unresolved tickets, or other issues. The complaint count is a material quality signal, but it cannot establish a unique-customer complaint rate without more information. ### Customer Commitment Management approval remains pending. No purchasing authorization, approved budget, or confirmed rollout commitment was supplied. ### Rollout Feasibility The proposed operating cost is $2,000 per month ($24,000 annually, excluding other costs). Limited engineering support introduces implementation and remediation risk. Whether rollout produces positive net savings is currently UNKNOWN. ## 05 — Decision-Critical Risks | Risk / Blocker | Supporting Evidence | Consequence | Required Action | | ----------------------- | --------------------------- | ------------------------------------------ | ------------------------------------------- | | Answer quality | Accuracy below target | Incorrect customer guidance | Audit errors and define corrective tests | | Inadequate savings | 15% versus 20% target | Weak commercial justification | Validate net support economics | | Customer complaints | 34 complaints | Customer experience or trust deterioration | Categorize, investigate and assess severity | | Engineering constraints | Limited support | Delayed fixes and operational issues | Confirm capacity and accountable owner | | Approval dependency | Management approval pending | Unauthorized rollout | Obtain explicit decision before expansion | Non-negotiable rollout blockers: Missing management authorization and unresolved acceptance of the failed success criteria. Any critical customer-impacting errors identified during complaint review must also be resolved before deployment. ## 06 — Decision Justification EXTEND is recommended only as a bounded evaluation, not as permission to expand production usage. GO is inappropriate because two targets were missed and management authorization is absent. STOP is a reasonable alternative if remediation requires disproportionate resources, complaint severity is unacceptable, or the financial model remains unfavorable. The current evidence does not yet establish those conditions. The proposed extension should proceed only if the customer and engineering owners confirm that a limited corrective test is feasible. ### Proposed extension: 14 days Proposed duration, not an agreed deadline or customer commitment. | Primary question | Can accuracy and net cost reduction reach their original targets within acceptable resource constraints? | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | | Test method | Audit original ticket and complaint records, correct identifiable failure causes, then evaluate a representative ticket sample. | | Success conditions | At least 70% verified automatic resolution, 90% validated answer accuracy, and 20% demonstrated cost reduction using agreed measurement methods. | | Quality condition | No unresolved critical customer-impacting failures; complaint causes reviewed. | | Owner | Proposed: Product Lead with Engineering and Customer Support. | | Cost ceiling | To be approved before extension starts. | | Exit decision | GO only if criteria and rollout gates are met; otherwise STOP or explicitly approve a redesigned pilot based on new evidence. | If engineering feasibility, measurement access, or an acceptable cost ceiling cannot be established, the extension should not proceed. ## 07 — Next Action Plan | Priority | Action | Responsible Owner (proposed) | Completion Evidence | | -------- | ---------------------------------------------------------------------------------- | ---------------------------- | ------------------------------------------------------------------- | | P1 | Audit response accuracy, automated resolutions, and 34 complaints | Support QA Lead | Reconciled ticket-level analysis and severity report | | P2 | Validate financial baseline, incremental rollout cost, and remediation feasibility | Finance + Engineering Lead | Net-savings model and approved resource estimate | | P3 | Approve or reject a bounded follow-up test | Product Lead + Management | Documented decision, test protocol, acceptance criteria, and budget | ## 08 — Final Decision Statement EXTEND — CONDITIONAL APPROVAL ONLY The chatbot pilot achieved its reported automation target but failed its answer-accuracy and cost-reduction objectives. A full rollout is not supported by the current evidence. A limited 14-day follow-up evaluation is proposed, contingent on confirmed feasibility, an approved cost ceiling, and management authorization. The next decision must use validated quality results, demonstrated financial benefit, and explicit rollout acceptance criteria.