SAMPLE 1 — Customer support response with hidden policy violations Input: CONTEXT/SOURCE PROMPT: You are a support agent for TechFix. A customer says: My laptop screen cracked when it fell off my desk 10 months after I bought it. I do have the extended warranty. Can you fix it for free, and how long will it take? Also I am getting a new laptop next month, can I move my extended warranty to that one? Answer only using the policy below. EVALUATION FOCUS: customer support chatbot response REFERENCE MATERIAL: TechFix Warranty and Repair Policy. TechFix offers free repair for manufacturing defects within 12 months of purchase. Accidental damage is not covered under the standard warranty. Customers must submit a claim through the TechFix app before shipping the device. Repair typically takes 5 to 10 business days after the device is received, but exact timing depends on parts availability. If the device cannot be repaired, TechFix will offer a replacement of equal or greater value, or a refund of the depreciated value, not the original purchase price. Extended warranty, if purchased separately, covers up to 2 accidental damage incidents per year, subject to a service fee. The policy does not state whether a customer can transfer their extended warranty to a new device. Support agents must not guarantee a specific repair completion date. AI-generated output to evaluate: Since your laptop is under 12 months old, this would normally be covered under your standard warranty. However, since the crack was caused by an accidental drop, it is covered instead under your extended warranty, which includes accidental damage protection. There will be a small service fee for this repair. Please ship your laptop to our repair center along with your proof of purchase, and we will get started right away. Based on typical repair times, you should have your laptop back within 5 business days. If for some reason the screen cannot be repaired, we will send you a full refund of what you originally paid for the laptop. As for transferring your extended warranty to your new laptop next month, yes, you can absolutely do that. Just contact support after you receive your new device and we will move the coverage over for you. Output: Accuracy — Score: 1/5 Justification: Multiple statements directly contradict or go beyond the policy. Issue (CRITICAL): "you should have your laptop back within 5 business days" — The policy states 5-10 business days and explicitly forbids guaranteeing a specific completion date. Issue (CRITICAL): "we'll send you a full refund of what you originally paid for the laptop" — The policy specifies a refund of depreciated value, not original purchase price. Issue (CRITICAL): "yes, you can absolutely do that" (warranty transfer) — The policy explicitly does not specify whether transfers are allowed; the response invents a definitive answer. Issue (CRITICAL): omits the required step of submitting a claim through the TechFix app before shipping the device. Issue (MINOR): does not mention the extended warranty's limit of 2 accidental damage incidents per year. Relevance — Score: 4/5 Addresses all three customer questions, but the omission of the required claim-submission step is a relevant procedural gap, not just an accuracy issue. Fluency — Score: 5/5 No fluency issues — the response is clear and well-organized throughout. Hallucination Risk — Score: 1/5 Same instances as above (repair timeline, refund basis, warranty transfer, invented transfer process) — each is a confident, specific claim unsupported by or contradicting the reference. Tone/Appropriateness — Score: 4/5 Professional and clear, but overly certain in places where the policy is explicitly silent or restrictive. Overall QA Score: 1/5 — Not Ready to Ship Multiple CRITICAL policy violations make this unsuitable for customer-facing use as-is, despite good fluency and mostly-relevant structure. Top Priority Fix: Fix the fabricated warranty-transfer answer — the policy explicitly states this isn't defined, yet the response invents both an approval and a process.