PromptBase
PromptBase SelectGet Select
Close icon
General
Home
Marketplace
Create
Hire
Login
Chat
Sell
Explore

Prompt Details

Model
Chat - (gpt-5.6)
Token size
2,860
Example input
[AGENT_ARCHITECTURE_AND_SYSTEM_INSTRUCTIONS]: Scheduling agent manages meetings. It can search calendars, create events, update events, and cancel events. It must check availability before creating a meeting. It must never create duplicate events. [AVAILABLE_TOOLS_AND_SCHEMAS]: search_calendar( user_id: string, start_time: string, end_time: string ) create_event( user_id: string, title: string, start_time: string, end_time: string, attendees: array ) update_event( event_id: string, start_time: string, end_time: string ) cancel_event( event_id: string ) [AGENT_TASKS_AND_USER_REQUESTS]: Schedule a 30-minute project meeting with Alex tomorrow at 3 PM. [TOOL_RESULTS_AND_ERROR_RESPONSES]: search_calendar: available=true First create_event: status=SUCCESS event_id=EVT-5521 Second create_event: status=SUCCESS event_id=EVT-5522 [EVALUATION_METRICS_LOGS_AND_KNOWN_FAILURES]: 200 scheduling requests 12 duplicate meetings 18 unnecessary calendar searches 7 incorrect attendee mappings 4 incorrect time-zone conversions
Example output
πŸ”Phase 1 - Agent & Tool Environment Reconstruction The agent's objective is: Understand request ↓ Determine meeting duration ↓ Determine requested time ↓ Check availability ↓ Create meeting ↓ Confirm successful creation ↓ Stop The agent has four relevant capabilities: search_calendar β†’ availability verification create_event β†’ event creation update_event β†’ modify existing event cancel_event β†’ remove existing event For this request, only search_calendar and create_event are required. --- Phase 2 - Tool Selection Audit πŸ› οΈ Call 1: search_calendar search_calendar(...) Classification: 🟒 Correct Selection The system instruction explicitly requires an availability check before creating a meeting. The tool is appropriate because the agent needs to determine whether the requested slot is free. --- Call 2: create_event create_event(...) Classification: 🟒 Correct Selection The availability result was: available=true Therefore, creating the event is appropriate. --- Call 3: create_event create_event(...) Classification: πŸ”΄ Incorrect / Redundant The previous call already returned: status=SUCCESS event_id=EVT-5521 The second creation is not verification. It is another side-effecting creation operation. Because it generated: event_id=EVT-5522 the customer now has two meetings. 🚨 Critical reliability failure. --- Phase 3 - Argument Construction Audit 🧩 The arguments are mostly structurally valid. However, two risks exist. Time representation "tomorrow 15:00" This is ambiguous unless the agent has an explicit timezone context. A safer representation would be: 2026-09-07T15:00:00+05:30 or another explicitly resolved timezone. Attendee attendees=["Alex"] The schema accepts an array, but "Alex" may not uniquely identify a person. A stronger system should resolve Alex to a unique attendee identifier before event creation. --- Phase 4 - Schema Compliance Analysis πŸ“ The calls technically comply with the basic schema. However, the schema itself is too weak. It does not specify: Timezone requirements Attendee identifier format Duplicate-event protection Idempotency Whether event creation is safely retryable Maximum attendee count Valid timestamp format Schema weakness The most important missing field is: idempotency_key A stronger event-creation schema could conceptually be: create_event( user_id: string, title: string, start_time: ISO8601 datetime, end_time: ISO8601 datetime, attendees: array, idempotency_key: string ) --- Phase 5 - Unnecessary Tool-Call Detection πŸ—‘οΈ There are two major inefficiencies. Duplicate event creation The second create_event was unnecessary. Verification misunderstanding The agent apparently treats: "create succeeded" as: "Maybe creation didn't happen, so create again." That is incorrect. Verification should be read-only. For example: create_event ↓ SUCCESS ↓ get/read event if such a read-only verification tool exists. It should never be: create_event ↓ create_event again --- Phase 6 - Tool-Call Sequence Analysis πŸ”„ Actual sequence search_calendar ↓ create_event ↓ create_event Correct sequence search_calendar ↓ create_event ↓ SUCCESS ↓ STOP Or, if explicit verification is required: search_calendar ↓ create_event ↓ read_event ↓ SUCCESS ↓ STOP The current architecture has a verification-via-reexecution problem. --- Phase 7 - Context & Decision Consistency 🧠 After receiving: event_id=EVT-5521 the agent should update its internal state: event_created=true event_id=EVT-5521 task_status=COMPLETED Instead, it behaves as though: event_created=false This demonstrates a task-state consistency failure. --- Phase 8 - Tool-Result Interpretation Audit πŸ“€ Actual result status=SUCCESS event_id=EVT-5521 Correct interpretation Meeting successfully created. Event ID = EVT-5521. No further creation call required. Agent behavior create_event again This is a: πŸ”΄ Result Interpretation Failure The agent failed to interpret the successful side-effect response as completion. --- Phase 9 - Failure Pattern Classification 🚨 Failure Category Severity Duplicate event creation Redundant Call Critical Reusing create as verification Sequence Failure Critical Missing idempotency protection Schema Failure Critical Ambiguous attendee identity Argument Failure High Ambiguous timezone Argument Failure High Incorrect success-state handling Context Failure High Generic verification logic Retry Failure High Recurring production pattern 12 duplicate meetings / 200 requests That equals a 6% duplicate-event rate in the supplied sample. This is a significant reliability problem for a scheduling system. --- Phase 10 - Retry Behavior Audit πŸ” Current logic: IF creation appears successful β†’ create the event again This logic should be removed. The agent should distinguish: VERIFICATION from: RETRY Safe retry model create_event ↓ TIMEOUT ↓ Check whether event exists ↓ If exists β†’ SUCCESS ↓ If confirmed absent β†’ retry with same idempotency key Unsafe model create_event ↓ TIMEOUT ↓ create_event again The first request may have succeeded despite the timeout. --- Phase 11 - Recovery Strategy πŸš‘ Scenario A: Timeout create_event ↓ TIMEOUT ↓ Search for matching event ↓ Event exists? β”œβ”€β”€ YES β†’ Treat as successful └── NO β†’ Retry safely Scenario B: Duplicate event detected Duplicate detected ↓ Keep canonical event ↓ Cancel duplicate ↓ Record recovery ↓ Notify user if necessary Scenario C: Unknown attendee "Alex" ↓ Multiple possible matches ↓ Do not create event ↓ Ask user to identify correct Alex Scenario D: Timezone ambiguity Requested 3 PM ↓ Timezone unresolved ↓ Resolve timezone from user/calendar context ↓ Only then create event --- Phase 12 - Schema & Validation Improvements βš™οΈ The event creation tool should require: user_id title start_time end_time attendees idempotency_key Validation should enforce: start_time < end_time timestamps must be ISO-8601 timezone must be explicit attendees must use valid identifiers idempotency_key must be unique per logical request Tool metadata should also expose: side_effect: CREATE_EVENT idempotent: YES_WITH_KEY safe_to_retry: CONDITIONAL verification_method: READ_ONLY_LOOKUP This makes the tool's operational behavior much clearer to the agent. --- Phase 13 - Reliability Testing Strategy πŸ§ͺ The improved system should be tested with: 1. Normal meeting creation 2. Calendar slot unavailable 3. Ambiguous attendee 4. Unknown attendee 5. Invalid timestamp 6. Missing timezone 7. Calendar search timeout 8. Event creation timeout 9. Event creation success 10. Duplicate creation attempt 11. Same idempotency key 12. Different idempotency key 13. Partial tool response 14. Malformed event response 15. Existing duplicate event 16. User changes meeting time 17. User cancels before creation 18. Calendar becomes unavailable after availability check --- Phase 14 - Tool-Calling Reliability Score πŸ“Š Estimated Score: 64/100 Confidence: Medium Dimension Score Tool Selection Accuracy 78/100 Argument Validity 70/100 Call Necessity 50/100 Sequence Correctness 48/100 Result Interpretation 45/100 Retry Quality 35/100 Recovery Quality 58/100 Schema Robustness 52/100 Major score penalties -15 Duplicate event creation -10 Unsafe verification strategy -6 Missing idempotency mechanism -5 Ambiguous attendee handling --- Phase 15 - Reliability Improvement Roadmap πŸ›£οΈ πŸ”₯ IMMEDIATE Problem: Duplicate events. Change: Add idempotency keys and prohibit repeated create calls after success. Expected impact: Very high. Effort: Medium. Risk: Low. Validation: Run 100 duplicate/timeout scenarios. --- πŸš€ SHORT TERM Problem: Ambiguous time and attendee arguments. Change: Require canonical attendee IDs and ISO-8601 timestamps with timezone. Expected impact: High. Effort: Low/Medium. Risk: Low. Validation: Adversarial argument tests. --- πŸ—οΈ MEDIUM TERM Problem: Verification is implemented as another creation. Change: Separate write operations from read-only verification. Expected impact: Very high. Effort: Medium. Risk: Low. Validation: Workflow-level testing. --- 🧠 LONG TERM Problem: Weak task-state management. Change: Introduce explicit execution states: REQUESTED ↓ VALIDATING ↓ AVAILABILITY_CONFIRMED ↓ CREATING ↓ CREATED ↓ VERIFIED ↓ COMPLETED And for uncertain execution: CREATING ↓ TIMEOUT ↓ VERIFYING ↓ FOUND β†’ COMPLETED NOT_FOUND β†’ SAFE_RETRY UNKNOWN β†’ ESCALATE --- 🎯 Final Recommendation The highest-impact fix is to separate verification from execution. The agent must never use a side-effecting create_event call as its own verification mechanism. The ideal architecture is: User Request ↓ Intent Extraction ↓ Argument Validation ↓ Availability Check ↓ Create Event + Idempotency Key ↓ Tool Result ↓ State Update ↓ Optional Read-Only Verification ↓ COMPLETED Final verdict: ⚠️ Medium reliability with a critical duplicate-execution vulnerability. The biggest opportunity is to combine idempotent tool design + explicit task states + read-only verification + timezone/attendee validation.
GPT

Agent Toolcalling Failure Auditor

Add to Cart
Instant accessInstant access
Usage rightsCommercial use
Money-back guaranteeMoney‑back
By purchasing this prompt, you agree to our terms of service
GPT-5.6
Tested icon
Guide icon
4 examples icon
Free credits icon
AI agents often fail because they choose the wrong tool, send invalid arguments, call tools unnecessarily, or misinterpret tool results. 🎯 Detect wrong tool-selection decisions 🧩 Find invalid, missing & malformed arguments πŸ—‘οΈ Identify unnecessary tool calls πŸ”„ Analyze tool-call sequences 🚨 Classify recurring failure patterns πŸ›‘οΈ Build retry & recovery strategies βš™οΈ Improve tool schemas & validation πŸ“Š Score overall tool-calling reliability Find exactly where your agent's tool-calling pipelin
...more
Added over 1 month ago
Report
Browse Marketplace