Evaluation Goal Determine whether the new system prompt produces a reliable and practically meaningful improvement without introducing unacceptable regressions. Evidence Check The new version is 1.7 percentage points higher overall: 89.1% − 87.4% = +1.7 percentage points However, the aggregate scores alone are insufficient to conclude that the new version is better. Before making a release decision, verify: Per-category results for billing, refunds, account access, and general support Critical failure rates Test-set composition and representativeness Whether both versions were evaluated on the same 800 cases Evaluation criteria and scoring method Evaluator/judge configuration and calibration Sample-size and uncertainty analysis Any changes to model, retrieval, decoding, tools, or other system components Regression Analysis Do not invent category-level results from the overall scores. Compare the two versions across each category and inspect whether any important subgroup or failure mode regressed. If the same cases were evaluated by both versions, analyze the paired outcomes where the evaluation design supports it rather than treating the two aggregate percentages as independent measurements. Reliability Check Validate the evaluator against a representative human-reviewed subset when model-based judging is used. Inspect evaluator disagreements and determine whether the observed 1.7-point difference is statistically and practically meaningful. Decision MORE EVIDENCE REQUIRED The new version has an observed overall improvement of 1.7 percentage points, but the supplied evidence does not establish that the improvement is reliable or that no unacceptable regression exists. Next Action Provide the category-level results, critical failure data, evaluation configuration, and raw or case-level comparison where available. Then perform the regression, uncertainty, and failure analysis before making the release decision. Do not declare the new version better solely from the 89.1% versus 87.4% aggregate scores.