### 1\. 🚀 Phase 1 — Prototype Readiness & Repository Reconstruction ### Current-state findings * Confirmed in sample: TypeScript, Node.js 22, NestJS, BullMQ, Redis, pnpm, and a Linux deployment target. * Confirmed in sample: The application processes uploaded PDFs using extraction, OCR, and AI. * Confirmed in sample: API and worker containers are intended to run separately. * Inferred: The API accepts uploads and submits document-processing jobs to BullMQ. * Proposed: Store document metadata and processing status in a durable database, if not already available. * Unknown: Database technology, ORM, existing endpoints, actual OCR package, model provider, and authentication implementation. * Unknown: Whether uploaded files are stored temporarily on disk before being transferred to object storage. ### Deployment blockers * Confirm that all native OCR dependencies work on the target Linux image. * Verify that `pnpm-lock.yaml` matches `package.json`. * Identify where job status and extracted results are persisted. * Confirm whether the worker can recover a job after an unexpected process termination. * Verify that PDF parsing and OCR libraries handle malformed files safely. Required evidence: Repository tree, package manifests, source files, Docker configuration, queue setup, and existing tests. ### 2\. 🏗️ Phase 2 — Deployment Target & Runtime Architecture ### Recommended architecture Business User Authenticated document upload HTTPS Reverse Proxy TLS, request-size and connection limits NestJS API Container Authentication, upload validation, job submission Redis + BullMQ Job queue and retry scheduling Object Storage Private original documents and outputs Dedicated Worker Container PDF extraction → OCR → AI extraction → validation Persistent Metadata Store Job status, ownership, results and audit records ### Architectural decisions * Use asynchronous jobs because OCR and document processing can exceed ordinary HTTP request timeouts. * Keep the API responsive while workers process files. * Store uploaded documents in private object storage rather than relying on a container's writable filesystem. * Use Redis for queue coordination, not as the only permanent record of business-critical results. * Use a separate database for durable document and job metadata if the existing application does not already provide one. * Use Docker Compose for an initial single-VM deployment, but recognize that a single VM is a potential availability bottleneck. * Avoid Kubernetes unless operational scale or deployment requirements justify it. ### 3\. 📦 Phase 3 — Dependency & Build Reproducibility ### Runtime specification * Node.js: Version 22, as provided in the sample. * Package manager: pnpm with the committed lockfile. * Application framework: NestJS. * Queue: BullMQ with a compatible Redis deployment. * Container platform: Linux. * Build approach: Compile TypeScript into production JavaScript and run the compiled entry point. * Native dependencies: Confirm OCR engine binaries, PDF libraries, and required system packages before selecting the final runtime image. ### Illustrative verification commands ```bash node --version corepack pnpm --version corepack pnpm install --frozen-lockfile corepack pnpm run build corepack pnpm test ``` These commands assume the relevant package scripts exist and Corepack is available. Verify the repository before relying on them. ### Build requirements * Pin dependency versions and use the committed lockfile. * Keep development tools out of the final runtime image when practical. * Compile TypeScript during the build stage. * Generate a software bill of materials where supported. * Scan dependencies and container images for known vulnerabilities. * Avoid downloading OCR models or executables dynamically at application startup unless explicitly designed and secured. Acceptance criterion: The application builds from a clean checkout without modifying the committed lockfile. ### 4\. 🐳 Phase 4 — Docker & Containerization Engine ### Proposed multi-stage Dockerfile This example assumes a `build` script creates `dist/main.js`, and that runtime dependencies can be installed from the lockfile. Verify the actual project structure and OCR system requirements before use. ```dockerfile FROM node:22-bookworm-slim AS build WORKDIR /app RUN corepack enable COPY package.json pnpm-lock.yaml ./ RUN pnpm install --frozen-lockfile COPY . . RUN pnpm run build FROM node:22-bookworm-slim AS runtime ENV NODE_ENV=production WORKDIR /app RUN corepack enable \ && groupadd --system appgroup \ && useradd --system --gid appgroup appuser COPY package.json pnpm-lock.yaml ./ RUN pnpm install --prod --frozen-lockfile \ && pnpm store prune COPY --from=build --chown=appuser:appgroup \ /app/dist ./dist USER appuser EXPOSE 3000 CMD ["node", "dist/main.js"] ``` Important: If OCR requires native libraries, install only the verified runtime packages in the final stage. The example must also be adapted if the application needs additional runtime assets or migrations. ### Proposed `.dockerignore` ```text .git .github node_modules dist coverage .nyc_output .env .env.* !.env.example *.pem *.key uploads temporary-documents logs npm-debug.log* pnpm-debug.log* ``` ### Build and verification ```bash docker build -t document-agent:local . ``` ```bash docker image inspect document-agent:local ``` * Confirm the image uses the intended Node.js runtime. * Confirm no local uploads, credentials, or private documents entered the build context. * Run API and worker containers with separate commands or service definitions. * Do not mount the Docker socket or run the application as root without a justified need. * Configure shutdown grace periods and writable temporary directories explicitly. ### 5\. 🤖 Phase 5 — AI Agent Runtime & Execution Lifecycle ### Document processing lifecycle Validate user identity and upload authorization Validate file type, size and content Store original file in private object storage Create durable document and job records Enqueue processing job Extract embedded PDF text Run OCR when required Extract structured information with the AI model Validate output schema and business rules Persist result and processing status Notify authorized client or expose result endpoint ### Reliability requirements * Assign a unique document ID and processing job ID. * Record the upload and job state before returning a successful job-submission response. * Use bounded retries for transient model, storage, and OCR-service failures. * Distinguish retryable failures from invalid documents and permanent validation errors. * Route repeatedly failing jobs to a dead-letter mechanism with diagnostic metadata. * Set maximum file size, page count, OCR duration, memory usage, and model input size. * Track the processing stage so interrupted jobs can be resumed or safely reprocessed. * Use idempotency keys to avoid duplicate job submissions. * Treat extracted text as untrusted content that may contain prompt injection attempts. * Validate AI-generated output against a schema before storing or returning it. Critical decision: A retry must not create duplicate billing, duplicate records, or repeated external actions. If a stage has uncertain completion, reconcile its state before rerunning it. ### 6\. 🌐 Phase 6 — API Server & Request Handling These are proposed routes for the sample application. ### Endpoint design * `POST /v1/documents` * Purpose: Upload and register a document. * Authentication: Required. * Validation: File size, type, content, user quota, and authorization. * Response: `202 Accepted` when asynchronous processing is successfully scheduled. * Failure: Reject invalid uploads and avoid creating incomplete job records. * `GET /v1/documents/{document_id}` * Purpose: Retrieve document status and permitted metadata. * Authentication: Required. * Authorization: Confirm that the requester can access that document. * Response: Status, timestamps, and processing stage. * `GET /v1/documents/{document_id}/result` * Purpose: Retrieve extracted data. * Authentication: Required. * Validation: Confirm ownership and successful completion. * Failure: Return an appropriate error if the result is unavailable. * `POST /v1/documents/{document_id}/cancel` * Purpose: Request cancellation. * Authentication: Required. * Behavior: Cancel only if the current processing stage supports safe cancellation. * `GET /health/live` * Purpose: Determine whether the API process is functioning. * Behavior: Avoid expensive OCR or AI requests. * `GET /health/ready` * Purpose: Verify that the API has initialized the dependencies needed to accept work. * Behavior: Use bounded dependency checks. ### Example task response ```json { "document_id": "doc_example_456", "status": "queued" } ``` This is an illustrative response, not a verified implementation. ### Request-handling safeguards * Configure reverse-proxy and application-level upload limits. * Do not trust filename extensions or client-provided MIME types alone. * Validate actual file signatures and parsing behavior. * Store files under generated identifiers instead of user-supplied paths. * Return generic public errors and retain sanitized internal diagnostics. * Define whether cancellation stops only future processing or also attempts to terminate active OCR/model calls. * Use a separate authenticated download mechanism for large results or original files. ### 7\. 🔐 Phase 7 — Environment Configuration & Secret Management ### Configuration inventory * `NODE_ENV` * Purpose: Runtime environment. * Secret: No. * Required: Yes. * `PORT` * Purpose: API listening port. * Secret: No. * Required: Yes. * `REDIS_URL` * Purpose: Queue connectivity. * Secret: Treat as sensitive if it includes credentials. * Required: Yes for BullMQ. * `DATABASE_URL` * Purpose: Persistent metadata and job state. * Secret: Yes if it contains authentication information. * Required: If a database is used. * `MODEL_API_KEY` * Purpose: AI provider authentication. * Secret: Yes. * Required: Yes for model-based extraction. * `MODEL_NAME` * Purpose: Select the approved model. * Secret: No. * Required: Yes. * `OBJECT_STORAGE_ENDPOINT` * Purpose: Storage endpoint. * Secret: Usually no, but restrict access to the configuration. * Required: Depending on provider. * `OBJECT_STORAGE_ACCESS_KEY` * Purpose: Storage authentication. * Secret: Yes. * Required: Yes when key-based authentication is used. * `OBJECT_STORAGE_SECRET_KEY` * Purpose: Storage authentication. * Secret: Yes. * Required: Yes when key-based authentication is used. * `OCR_TIMEOUT_SECONDS` * Purpose: Bound OCR execution. * Secret: No. * Required: Yes. * `MAX_UPLOAD_BYTES` * Purpose: Limit uploaded document size. * Secret: No. * Required: Yes. * `MAX_WORKER_CONCURRENCY` * Purpose: Limit simultaneous processing jobs. * Secret: No. * Required: Yes. ### Secret management procedure * Store production credentials in a protected secret store or restricted deployment configuration. * Use separate credentials for development, staging, and production. * Grant the worker only the storage and model permissions it needs. * Rotate credentials without logging their values. * Prevent secrets from entering Docker build layers, error responses, logs, and source control. * Validate required configuration at startup. * Fail safely if mandatory configuration is missing. ### 8\. ❤️ Phase 8 — Health Checks & Graceful Operations ### Health-check policy * Liveness: Confirms the Node.js process and event loop are responsive. * Readiness: Confirms essential configuration and API dependencies are initialized. * Worker health: Reports worker heartbeat, processing progress, and recent failures. * Queue health: Monitors queue accessibility, waiting jobs, failed jobs, and oldest job age. * Storage health: Checks the required object-storage operation with a bounded timeout where appropriate. ### Shutdown behavior * Mark the API instance unready. * Stop accepting new uploads and job submissions. * Allow active HTTP requests to finish within a defined deadline. * Stop workers from claiming new jobs. * Allow active jobs to finish or persist recoverable state. * Close Redis, database, and storage clients. * Exit before the container termination deadline. A liveness failure may justify a process restart, but a temporary model-provider outage alone should not automatically cause repeated container restarts. ### 9\. 📊 Phase 9 — Structured Logging & Execution Observability ### Recommended log fields * `timestamp` * `severity` * `request_id` * `document_id` * `job_id` * `processing_stage` * `attempt_number` * `ocr_duration_ms` * `model_duration_ms` * `result_status` * `error_category` ### Lifecycle events * `document_upload_accepted` * `document_validation_failed` * `job_enqueued` * `pdf_extraction_completed` * `ocr_started` * `model_extraction_failed` * `retry_scheduled` * `job_dead_lettered` * `document_processing_completed` * `job_cancelled` ### Metrics and alerts * Upload failure rate. * Queue depth and oldest waiting job. * Average and percentile processing duration. * OCR and model-provider failure rates. * Worker memory consumption. * Dead-letter queue growth. * Storage failures and database errors. * Estimated model usage and cost. Do not log complete documents, extracted financial details, access tokens, private model reasoning, or unrestricted OCR output. Define retention periods and access restrictions according to the sensitivity of processed documents. ### 10\. 🔄 Phase 10 — CI/CD Pipeline Architecture ### Proposed release workflow * Developer opens a pull request. * CI installs dependencies using the frozen pnpm lockfile. * Linting, type checking, unit tests, and integration tests run. * Tests cover malformed PDFs, retry behavior, duplicate submissions, and interrupted jobs. * Dependency and container scans run. * The build produces versioned API and worker images. * Images are published to a private registry with commit-SHA tags and immutable digests. * A staging deployment uses isolated credentials and test documents. * Smoke tests verify upload, queue processing, OCR, model extraction, and result retrieval. * Production release requires approval. * The same tested image digests are promoted to production. * Deployment verification checks API health, worker progress, queue depth, and error rates. Required configuration: Repository permissions, private registry, deployment identity, VM access mechanism, and secret provisioning. Do not store SSH private keys or cloud credentials directly in workflow files. Prefer short-lived deployment credentials and least-privilege access. ### 11\. 🛡️ Phase 11 — Security & Production Hardening ### High-priority risks * Unauthorized document access: Enforce authentication and document-level ownership checks. * Malicious file uploads: Validate file signatures, enforce limits, isolate parsers, and reject malformed inputs. * Prompt injection in document content: Treat extracted text as untrusted data and restrict agent tools. * Credential exposure: Use a secret store, restrict runtime access, and rotate credentials. * Resource exhaustion: Limit pages, file size, OCR runtime, memory, concurrency, and queue growth. * Duplicate processing: Use idempotency and durable processing state. * Sensitive result leakage: Apply authorization to every result and download request. * Container compromise: Run as non-root, minimize privileges, and restrict network access where feasible. * Data retention violations: Define deletion policies for originals, intermediate files, and extracted results. ### Security acceptance requirements * No public access to the object-storage bucket by default. * No user-controlled path used directly for filesystem writes. * No unrestricted model-generated tool execution. * No unvalidated AI-generated data written into business systems. * No secret values or sensitive documents in routine logs. * No release before critical vulnerabilities and authorization failures are addressed. ### 12\. 🚢 Phase 12 — Deployment Execution & Configuration ### Step 1: Prepare the source * Action: Verify the TypeScript source, scripts, dependency lock, and build output. * Command: `corepack pnpm install --frozen-lockfile` * Expected result: Dependencies install without lockfile changes. * Verification: Review installation output and run the build. * Failure response: Fix dependency inconsistencies before proceeding. ### Step 2: Configure persistent services * Action: Provision Redis, metadata storage if needed, and private object storage. * Inputs: Connection details, access policies, retention requirements. * Expected result: API and worker identities can access only their required resources. * Verification: Run authenticated connection checks. * Failure response: Diagnose networking and permissions before deploying application containers. ### Step 3: Build images * Action: Build separate production images if the application uses distinct API and worker entry points. * Command: `docker build -t document-agent:release-candidate .` * Expected result: A reproducible image. * Verification: Inspect image configuration and scan for vulnerabilities. * Failure response: Resolve the build issue before publishing. ### Step 4: Configure Docker Compose * Action: Define API, worker, and reverse-proxy services with explicit resource limits where supported. * Inputs: Image tags, runtime secrets, network rules, persistent dependencies. * Expected result: Services use the intended configuration and communicate over restricted networks. * Verification: Review the resolved Compose configuration without printing secret values. * Failure response: Correct invalid references, service dependencies, or permissions. ### Step 5: Deploy to staging * Action: Start the release candidate in an isolated environment. * Expected result: API starts and workers can consume jobs. * Verification: Inspect container health, logs, and queue processing. * Failure response: Stop promotion and collect evidence before changing production configuration. ### Step 6: Test a document end to end * Action: Upload a non-sensitive test PDF. * Expected result: The document is stored, processed, and produces a validated result. * Verification: Confirm task state, output integrity, and access control. * Failure response: Identify the failed processing stage and avoid blindly resubmitting the job. ### Step 7: Release to production * Action: Promote the verified image digest and approved configuration. * Expected result: The service becomes ready and processes authorized jobs. * Verification: Monitor errors, queue depth, worker health, and storage operations. * Failure response: Stop the rollout and execute the tested rollback procedure. No commands have been executed against a real cloud VM in this sample. ### 13\. 🔧 Phase 13 — Troubleshooting & Failure Diagnosis * PDF extraction fails * Evidence: Parser logs, file metadata, sanitized error category. * Possible causes: Malformed PDF, unsupported encryption, missing native library. * Corrective action: Validate the file and test the parser in an isolated environment. * Verification: Process a known-good fixture and a malformed fixture. * OCR worker crashes * Evidence: Memory metrics, exit code, OCR duration, container logs. * Possible causes: Oversized pages, memory exhaustion, incompatible native libraries. * Corrective action: Enforce page limits and tune worker concurrency based on measurements. * Verification: Run representative documents under resource limits. * Jobs remain queued * Evidence: Queue depth, worker heartbeat, Redis connectivity. * Possible causes: Worker stopped, queue misconfiguration, Redis outage. * Corrective action: Restore the underlying dependency or worker configuration. * Verification: Confirm the oldest queued job begins processing. * Model requests are rate-limited * Evidence: Provider status codes, retry metadata, request timestamps. * Possible causes: Quota exhaustion or excessive concurrency. * Corrective action: Apply bounded exponential backoff and enforce provider-aware concurrency. * Verification: Confirm retries are bounded and the queue remains controlled. * Duplicate document results appear * Evidence: Document IDs, idempotency keys, job history, database records. * Possible causes: Duplicate submissions or non-idempotent retries. * Corrective action: Add deduplication and stage-level completion records. * Verification: Repeat the same submission and confirm it does not create duplicate business effects. * Object-storage access fails * Evidence: Sanitized provider error, identity permissions, endpoint configuration. * Possible causes: Expired credentials, incorrect policy, networking issue. * Corrective action: Correct the identity or network configuration. * Verification: Upload and retrieve a controlled test object. * Deployment fails after an image update * Evidence: Container exit status, startup logs, health checks, image digest. * Possible causes: Missing native dependency, invalid runtime configuration, incompatible build artifact. * Corrective action: Halt promotion and restore the previous compatible image. * Verification: Confirm readiness and process a controlled test job. ### 14\. ♻️ Phase 14 — Release, Rollback & Recovery ### Release approach Use a staged rollout with a verified previous image. On a single VM, a replacement deployment may briefly interrupt service; avoid promising zero downtime without appropriate redundancy. ### Rollback triggers * API startup or readiness fails. * Workers cannot consume jobs. * Document-processing failures rise materially above the agreed baseline. * Sensitive data is exposed. * Results become corrupted or inconsistent. * A release introduces incompatible database changes. ### Recovery procedure * Stop the rollout. * Record the current image digest and sanitized failure evidence. * Restore the previous known-good API and worker images. * Verify database and queue compatibility. * Inspect active, queued, and failed jobs. * Reconcile processing stages before re-enqueuing uncertain work. * Verify document access controls and storage integrity. * Run end-to-end smoke tests. * Record the incident and corrective action. Do not delete Redis data, clear queues, or roll back database schemas without evidence and an approved recovery plan. Application rollback does not automatically undo already completed document processing or external side effects. ### 15\. 🧪 Phase 15 — Production Readiness & Acceptance Testing Every test starts as NOT TESTED. Actual results must be filled in after execution. * Test AADDE-ST2-01: Clean installation * Preconditions: Clean checkout and supported Node.js version. * Procedure: Install dependencies using the frozen lockfile. * Expected result: Installation completes without lockfile modification. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-02: Production build * Preconditions: Valid dependency lock and build scripts. * Procedure: Run the production build and build the Docker image. * Expected result: Image starts with the expected entry point. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-03: Upload validation * Preconditions: Staging API and test credentials. * Procedure: Submit a valid PDF, an oversized file, and a malformed file. * Expected result: Valid files are accepted; invalid files are safely rejected. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-04: Authorization * Preconditions: Two separate test users. * Procedure: Attempt to retrieve another user's document and result. * Expected result: Unauthorized access is rejected. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-05: End-to-end processing * Preconditions: Working OCR and model credentials. * Procedure: Upload a known test PDF and retrieve its result. * Expected result: Processing completes and output passes schema validation. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-06: Duplicate submission * Preconditions: Idempotency support implemented. * Procedure: Submit the same logical job repeatedly. * Expected result: Duplicate business effects are prevented. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-07: Worker interruption * Preconditions: A recoverable test job. * Procedure: Interrupt a worker during processing. * Expected result: Job state remains consistent and recovery follows the retry policy. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-08: Retry and dead-letter handling * Preconditions: Configured retry policy and dead-letter mechanism. * Procedure: Simulate a controlled permanent failure. * Expected result: Retry limits are respected and the job is recorded for investigation. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-09: Secret exposure * Preconditions: Built image and staging logs. * Procedure: Inspect image layers, configuration, and logs for secrets. * Expected result: No credentials are exposed. * Actual result: Not tested. * Status: NOT TESTED. * Test AADDE-ST2-10: Rollback * Preconditions: Previous compatible release available. * Procedure: Roll back a staging release and reprocess a controlled job. * Expected result: Service recovers without losing or duplicating business-critical state. * Actual result: Not tested. * Status: NOT TESTED. ### Production acceptance criteria The deployment should not be approved until authorization, file validation, persistent state, worker recovery, secret handling, and rollback have been verified with evidence. ### 16\. ✅ Phase 16 — Final AI Agent Deployment & DevOps Blueprint ### Final deployment package * Architecture: NestJS API + BullMQ worker + Redis + private object storage + durable metadata store where required. * Runtime: Node.js 22 and pnpm, subject to repository validation. * Build artifacts: Reproducible API and worker images with immutable digests. * Configuration: Runtime variables and protected credentials for the model, queue, storage, and OCR integrations. * Execution model: Asynchronous document processing with persistent job state, bounded retries, and safe cancellation. * Security: Authenticated uploads, document-level authorization, file validation, private storage, least privilege, and prompt-injection safeguards. * Health checks: Separate API liveness, API readiness, worker health, and queue monitoring. * Observability: Structured logs, processing-stage events, queue metrics, error categories, and model usage tracking. * CI/CD: Locked dependency installation, automated tests, image scanning, staging validation, approval, and controlled production rollout. * Recovery: Previous image retention, state reconciliation, controlled reprocessing, and verified rollback. ### Final readiness status * READY: Sample scenario and proposed architecture are defined. * BLOCKED: Actual repository inspection and platform-specific deployment decisions. * NOT VERIFIED: Container builds, OCR compatibility, model connectivity, queue durability, authentication, production security, monitoring, and rollback. * NOT APPLICABLE: None can be conclusively classified without reviewing the actual project. Final verdict: Sample Test 2 provides a distinct, implementation-oriented deployment blueprint for an AI document-processing agent. It is not evidence that the software has been built, deployed, or production-tested.