# 1. EXECUTIVE SCALE DIRECTION **Status: HARDEN BEFORE GROWTH** The current modular-monolith architecture is not presently disproven by the supplied scale. The strongest near-term strategy is: **KEEP THE MODULAR MONOLITH → ESTABLISH CAPACITY BASELINES → HARDEN DATABASE / WORKER / DEPENDENCY BEHAVIOR → ENABLE HORIZONTAL API SCALING WHEN MEASURED SATURATION REQUIRES IT → SCALE WORKERS INDEPENDENTLY → INTRODUCE DATABASE READ SCALING OR SERVICE EXTRACTION ONLY WHEN specific workload evidence justifies them.** 3,000 registered users and approximately 150 peak concurrent users do **not** establish CPU, throughput, database, worker, or storage limits. The main current risk is therefore not insufficient architecture complexity. It is **insufficient capacity evidence combined with single-deployment concentration**. Microservices, sharding, specialized databases, and complex orchestration should be deferred. --- # 2. CURRENT ARCHITECTURE & CAPACITY FACTS ## CURRENT FACTS ### User-facing compute * Next.js application. * Node.js application layer. * One API deployment. * Modular-monolith architecture. ### Stateful services * PostgreSQL. * Redis. * Object storage. ### Background processing * Two background workers. ### Deployment * Docker-based deployment. ### Scale * 3,000 registered users. * Approximately 150 peak concurrent users. ### Team constraints * Five-person engineering team. * Limited DevOps capacity. * Moderate infrastructure budget. * Explicit preference to avoid unnecessary microservices. ## CURRENT RISKS 1. **Single API deployment concentration** * Potential availability and compute bottleneck. * Actual saturation is UNKNOWN. 2. **Database concentration** * PostgreSQL may eventually become a shared contention point across synchronous requests and background jobs. * No current evidence shows this is happening. 3. **Worker capacity uncertainty** * Two workers exist, but arrival rate, processing rate, queue age, retries, and utilization are unknown. 4. **Operational concentration** * A small team could be harmed more by premature distributed-system complexity than by temporary infrastructure constraints. 5. **Insufficient scaling telemetry** * Current capacity cannot be inferred from registered users or concurrency alone. ## UNKNOWN CAPACITY PARAMETERS * API request rate. * Request mix. * p50 / p95 / p99 latency. * CPU utilization. * Memory utilization. * event-loop delay. * database query latency. * database connection utilization. * lock contention. * cache hit rate. * worker utilization. * job arrival rate. * job processing rate. * queue depth. * oldest-job age. * retry rate. * dead-letter rate. * storage growth. * object-transfer volume. * external dependency latency. * external dependency rate limits. * deployment resource limits. --- # 3. UNKNOWN CAPACITY VARIABLES The following must be measured before architecture escalation decisions: | Variable | Why It Matters | | -------------------------------------- | ---------------------------------------------- | | Requests/sec by endpoint | Determines compute demand | | Concurrent in-flight requests | Reveals actual runtime pressure | | p95/p99 latency | Detects degradation before outright failure | | CPU/memory saturation | Indicates application resource constraints | | Node.js event-loop delay | Detects blocking or runtime saturation | | DB active connections | Reveals pool pressure | | Slow-query distribution | Identifies database bottlenecks | | DB lock/wait time | Detects contention | | Read/write ratio | Determines whether read scaling may help | | Cache hit/miss rate | Determines whether Redis is useful | | Job arrival/completion rate | Determines worker capacity | | Queue age | Shows whether backlog is operationally harmful | | Retry/dead-letter rate | Detects hidden processing failure | | Object-storage throughput | Identifies transfer bottlenecks | | External dependency latency/error rate | Detects non-local constraints | | Growth rate of stored data | Supports storage/data-layout planning | --- # 4. WORKLOAD MODEL The current workload model is incomplete. ## Interactive workload Measure: **REQUEST ARRIVAL RATE** → requests per second by route. **CONCURRENCY** → simultaneous requests and open connections. **REQUEST COST** → CPU, database queries, external calls, cache calls, and object transfers per request class. **LATENCY** → p50 / p95 / p99 by endpoint. ## Database workload Measure: * reads/sec; * writes/sec; * connection-pool utilization; * expensive queries; * table/index growth; * transaction duration; * lock waits; * buffer/cache behavior. ## Background workload Measure: * jobs created/time; * jobs completed/time; * oldest pending job; * average and tail processing time; * worker CPU/memory; * retries; * dependency waits; * failed jobs. ## Storage workload Measure: * new objects/time; * stored bytes growth; * upload/download volume; * object size distribution. ## External workload Measure: * calls by provider; * latency; * error rate; * timeout rate; * rate-limit events. --- # 5. SCALING DOMAIN ASSESSMENT ## API / COMPUTE **Current state:** One API deployment. **Assessment:** Potential future scaling boundary, but insufficient evidence exists to justify additional replicas solely from user count. **Priority:** Instrument first, then prepare stateless horizontal scaling. --- ## BACKGROUND PROCESSING **Current state:** Two workers. **Assessment:** Must be treated separately from API capacity. Two workers may be excessive, adequate, or insufficient depending on job arrival rate and processing duration. **Priority:** Measure queue age and processing capacity. --- ## DATABASE **Current state:** Single PostgreSQL data platform assumed from supplied description. **Assessment:** Highest-probability shared scaling dependency over time, but no current bottleneck evidence exists. **Priority:** Query visibility, connection management, indexing, transaction analysis. --- ## CACHE **Current state:** Redis exists. **Assessment:** Existing infrastructure should not automatically receive more responsibility. Measure whether Redis materially reduces repeated database or computational work. --- ## STORAGE Object storage is already an appropriate abstraction for binary/static artifacts. No architectural change currently justified. --- ## SEARCH No dedicated search requirement or bottleneck supplied. **Status: NO CHANGE JUSTIFIED** --- ## EXTERNAL DEPENDENCIES Potential scaling limits may appear outside the application. Track provider: * latency; * quotas; * throttling; * failures; * timeout behavior. --- ## OBSERVABILITY Currently the biggest architectural prerequisite. Scaling decisions cannot be evidence-driven without consistent metrics. --- ## DEPLOYMENT Docker is sufficient as an application packaging mechanism. No evidence currently justifies Kubernetes or a more complex orchestration platform. --- ## OPERATIONS TEAM The five-person team is a primary architectural constraint. The architecture should minimize: * services; * deployment units; * stateful technologies; * bespoke infrastructure; * operational runbooks; * cross-service debugging. --- # 6. ARCHITECTURE EVOLUTION LADDER ## STAGE 0 — CURRENT MODULAR MONOLITH Next.js → Node.js modular monolith → PostgreSQL → Redis → object storage → background workers ### Purpose Establish actual system capacity. ### Exit evidence Observed saturation, reliability weakness, or operational bottleneck. --- ## STAGE 1 — HARDEN CURRENT ARCHITECTURE Add: * request metrics; * structured tracing; * database query visibility; * worker/queue metrics; * dependency metrics; * resource dashboards; * timeout policies; * retry boundaries; * idempotency where required. ### Trigger Current architecture lacks sufficient evidence for safe scaling decisions. ### Validation Baseline load + endurance tests. --- ## STAGE 2 — HORIZONTAL API SCALING Potential move: **ONE API INSTANCE → MULTIPLE STATELESS API INSTANCES** ### Trigger Sustained or recurring API saturation correlated with unacceptable latency/error behavior after application-level inefficiencies are addressed. **Threshold:** BASELINE-RELATIVE THRESHOLD REQUIRED. ### Preconditions * stateless request handling; * shared session/state moved appropriately; * safe database connection pooling; * health checks; * load balancing. --- ## STAGE 3 — INDEPENDENT WORKER SCALING Increase worker concurrency or worker count independently of API compute. ### Trigger Job arrival rate persistently exceeds completion capacity or queue age violates operational requirements. ### Validation Queue saturation and recovery test. --- ## STAGE 4 — DATABASE OPTIMIZATION Sequence: query analysis → indexes → connection tuning → transaction/data-access changes → resource adjustment ### Trigger Database metrics show measurable contribution to request or worker latency. --- ## STAGE 5 — SELECTIVE READ SCALING Potential addition: **PostgreSQL primary + read replica(s)** ### Trigger Read workload materially contributes to primary database saturation after query and schema optimization. ### Do not introduce if writes, locking, poor query design, or connection pressure are the actual bottleneck. --- ## STAGE 6 — DATA PARTITIONING Potential: * table partitioning; * workload-specific data segmentation. ### Trigger Demonstrated data volume/access-pattern problem not adequately solved through query/index/schema/resource improvements. No current justification. --- ## STAGE 7 — SELECTIVE SERVICE EXTRACTION Extract only a workload with a specific independent operational requirement. Possible future candidates might include: * heavy background processing; * document/media processing; * high-volume notification handling. No candidate can presently be classified as justified from the supplied evidence. --- # 7. SCALING TRIGGER MATRIX | Signal | Measurement | Trigger Condition | Action | Why | Trade-off | Validation | Reversibility | | ------------------------- | -------------------------------------- | ---------------------------------------------------------------- | ---------------------------------------- | ---------------------------------- | ----------------------------------------- | ------------------------ | ------------- | | API saturation | CPU, memory, event-loop delay, latency | Sustained resource pressure correlates with unacceptable latency | Add API replicas | Increase compute capacity | More deployment/load-balancing complexity | Load + autoscaling test | High | | Growing API latency | p95/p99 by route | Latency worsens with load and compute is limiting | Horizontal scale or optimize hotspot | Address actual constraint | More infrastructure | Comparative load test | High | | DB latency | query latency + waits | DB dominates critical request latency | Optimize queries/indexes | Cheapest DB scaling move | Index/write overhead | DB load test | High | | DB connection pressure | pool utilization | Connection waits contribute to latency/errors | Tune pooling / concurrency | Prevent connection exhaustion | Pool tuning risk | Saturation test | High | | Read-heavy DB saturation | reads, CPU, I/O | Optimized read workload still limits primary | Add read replica | Separate eligible reads | Replication lag + routing complexity | Read-load test | Medium | | Worker backlog | queue age/depth | Arrival persistently exceeds completion | Increase workers/concurrency | Match processing capacity | More compute/dependency pressure | Queue saturation test | High | | Dependency throttling | rate-limit events | Provider limits constrain workload | Rate limiting / buffering / backpressure | Protect downstream dependency | Added latency | Dependency simulation | High | | Cache inefficiency | hit/miss + workload cost | Repeated expensive reads have measurable reuse | Adjust caching strategy | Reduce repeated work | Consistency/invalidation complexity | Cache effectiveness test | High | | Data-access concentration | workload distribution | One bounded workload independently drives scaling | Consider extraction | Independent scale/failure domain | Distributed-system overhead | Isolated workload test | Medium | | Availability risk | deployment failures/outages | Single instance materially violates required availability | Add redundancy | Remove single application instance | Higher infrastructure cost | Failure/recovery test | High | All numeric trigger values require baseline-derived thresholds. --- # 8. DATABASE EVOLUTION PLAN ## STEP 1 — QUERY OPTIMIZATION Collect: * slow-query logs; * execution plans; * query frequency; * latency distribution. Move further only if database pressure remains material. --- ## STEP 2 — INDEXES Add indexes where actual access patterns justify them. Validate: * read improvement; * write overhead; * index size; * maintenance cost. --- ## STEP 3 — CONNECTION MANAGEMENT Measure: * active connections; * idle connections; * queue/wait duration; * API vs worker connection demand. Avoid solving connection pressure by simply increasing pool sizes without validating PostgreSQL behavior. --- ## STEP 4 — SCHEMA / DATA ACCESS Investigate: * N+1 patterns; * unnecessarily wide queries; * high-frequency joins; * long transactions; * excessive round trips; * inefficient pagination. --- ## STEP 5 — RESOURCE SCALING Increase database resources only if workload measurements demonstrate resource saturation. --- ## STEP 6 — CACHING Use Redis selectively for: * frequently reused; * sufficiently stable; * expensive-to-obtain data. Do not cache merely because Redis already exists. --- ## STEP 7 — READ REPLICAS Introduce only if: 1. reads are a dominant source of pressure; 2. queries are already optimized; 3. replication lag is acceptable; 4. read-after-write semantics are understood. --- ## STEP 8 — PARTITIONING Require evidence of: * large-table operational impact; * access locality; * maintenance problems; * query patterns benefiting from partition elimination. --- ## STEP 9 — SPECIALIZED STORES Add only when PostgreSQL is demonstrably mismatched to a specific workload. No evidence currently supports this move. --- # 9. WORKER / QUEUE EVOLUTION Current worker count: **2** Capacity conclusion: **UNKNOWN** Required model: **ARRIVAL RATE** vs. **PROCESSING RATE** vs. **OLDEST JOB AGE** Observe: * queue depth; * job age; * worker utilization; * processing duration; * retries; * dead letters; * dependency waits. ## Evolution sequence ### A. Optimize worker processing Remove unnecessary serial work or expensive repeated operations. ### B. Adjust worker concurrency Only where dependency and database capacity permit it. ### C. Increase worker replicas When processing demand requires parallel capacity. ### D. Introduce workload separation Separate queues only when workloads differ materially in: * urgency; * resource consumption; * retry policy; * dependency behavior; * isolation requirements. ### E. Add backpressure Prevent workers from overwhelming PostgreSQL or external providers. --- # 10. SERVICE DECOMPOSITION TEST ## API modules **Extraction driver:** none currently demonstrated. **Current evidence:** modular monolith already provides internal boundaries. **Benefit:** limited without independent scaling/deployment need. **New complexity:** * network calls; * deployment coordination; * tracing; * distributed failure; * versioning; * operational ownership. **STATUS: NOT JUSTIFIED** --- ## Background processing **Extraction driver:** potentially different compute and scaling profile. **Current evidence:** two workers exist, but no workload pressure data. **Benefit:** possible independent scaling and fault isolation. **New complexity:** separate deployment and operational surface. **STATUS: PREPARE ONLY** --- ## High-load future bounded context **Extraction driver:** independent scaling or operational ownership. **Current evidence:** none supplied. **STATUS: NOT JUSTIFIED** --- # 11. RELIABILITY EVOLUTION Before scaling compute, establish: ## REDUNDANCY Multiple API instances when availability requirements or saturation justify them. ## TIMEOUTS Explicit boundaries around: * database work; * external calls; * worker dependencies. ## RETRIES Retry only transient failures. Use bounded retries with backoff. ## IDEMPOTENCY Required for retry-sensitive operations such as: * payments; * order mutation; * notifications; * background jobs. ## QUEUE DURABILITY Ensure accepted jobs survive worker restarts where business semantics require it. ## RECOVERY Document restoration and restart behavior. ## ROLLBACK Deployments must allow application rollback while respecting schema compatibility. ## LOAD SHEDDING Protect critical paths during overload rather than permitting total resource exhaustion. ## GRACEFUL DEGRADATION Potentially degrade: * non-critical enrichment; * analytics; * secondary notifications; * optional asynchronous work before core transactional functionality. --- # 12. OBSERVABILITY REQUIRED FOR SCALING Minimum instrumentation: ### API * request rate; * error rate; * p50/p95/p99; * CPU; * memory; * event-loop lag. ### PostgreSQL * query latency; * slow queries; * connections; * lock waits; * transaction duration; * CPU/I/O. ### Redis * hit rate; * miss rate; * memory; * evictions; * latency. ### Workers * arrival rate; * completion rate; * processing duration; * queue depth; * oldest-job age; * retries; * failures. ### External dependencies * latency; * failures; * timeouts; * rate-limit events. ### Deployment * deployment frequency; * failed deployments; * rollback events; * resource utilization by release. --- # 13. COST & OPERATIONAL COMPLEXITY ## Harden current system **Infrastructure:** low incremental increase. **Operational burden:** low. **Monitoring burden:** moderate. **Team expertise:** existing stack. **New failure modes:** limited. **Recommendation:** immediate. --- ## Multiple API instances **Infrastructure:** additional compute/load balancing. **Operational burden:** low to moderate. **Monitoring:** per-instance + aggregate. **New failure modes:** instance routing and state assumptions. **Reversibility:** high. --- ## More workers **Infrastructure:** additional compute. **Operational burden:** low. **Risk:** increased pressure on PostgreSQL and external services. **Reversibility:** high. --- ## Read replicas **Infrastructure:** additional database capacity. **Operational burden:** moderate. **New failure modes:** * replication lag; * stale reads; * routing errors. **Reversibility:** medium. --- ## Microservices **Infrastructure:** materially higher. **Operational burden:** high. **Monitoring:** distributed tracing and service-level telemetry required. **Team expertise:** distributed systems and deployment coordination. **Failure modes:** substantially expanded. **Current cost/benefit:** unsupported. --- ## Sharding **Operational burden:** very high. **Failure modes:** routing, rebalancing, cross-shard operations, migrations. **Current need:** unsupported. --- # 14. SCALE VALIDATION TESTS ## BASELINE LOAD TEST Purpose: Determine current throughput/latency/resource relationships. Supports decisions about: * API replicas; * DB optimization; * connection management. --- ## STRESS TEST Increase workload beyond normal operating range until degradation occurs. Determine: * first bottleneck; * degradation shape; * recovery behavior. --- ## SPIKE TEST Rapidly increase request volume. Validate: * queueing; * connection management; * overload behavior; * recovery. --- ## ENDURANCE TEST Run sustained representative workload. Detect: * memory leaks; * pool exhaustion; * queue accumulation; * gradual database degradation. --- ## DATABASE LOAD TEST Replay representative query mixes. Validate whether: * indexes; * query changes; * pooling; * resource changes actually improve the limiting workload. --- ## QUEUE SATURATION TEST Increase job production above worker processing capacity. Observe: * queue growth; * age; * recovery; * retries; * dependency pressure. --- ## DEPENDENCY FAILURE TEST Introduce: * slow responses; * transient failures; * rate limiting. Validate: * timeouts; * retry boundaries; * backpressure; * graceful degradation. --- ## HORIZONTAL-SCALE TEST Run the same workload against one and multiple API instances. Confirm whether additional replicas materially increase useful capacity before relying on horizontal scaling. --- ## RECOVERY TEST Terminate: * API instance; * worker; * dependency connection. Verify recovery without duplication, data corruption, or uncontrolled retry storms. --- # 15. CHANGES TO DEFER Defer: * microservices; * Kubernetes solely for expected growth; * database sharding; * multiple specialized databases; * event-driven decomposition across every module; * complex service mesh infrastructure; * dedicated search infrastructure without a search requirement; * multi-region deployment without corresponding availability requirements; * aggressive autoscaling without validated workload signals. These changes increase complexity before the supplied evidence demonstrates a need. --- # 16. ARCHITECTURE EVOLUTION HANDOFF ## KEEP * Modular monolith. * Next.js. * Node.js. * PostgreSQL. * Redis. * Object storage. * Docker. * Separate background-worker execution. ## HARDEN * Application statelessness where horizontal scaling may be required. * Database query visibility. * Connection management. * worker retry/idempotency behavior. * timeout and dependency-failure handling. * deployment recovery. * API availability strategy. ## MEASURE * request rate; * concurrency; * tail latency; * CPU/memory; * event-loop lag; * database query and connection pressure; * cache effectiveness; * worker arrival/completion rate; * queue age; * external dependency behavior; * storage growth. ## NEXT EVOLUTION **Build the capacity-observability baseline and harden the existing architecture before introducing another architecture tier.** If application compute subsequently proves to be the first constraint, the smallest likely next architectural move is: **ONE API DEPLOYMENT → MULTIPLE STATELESS API REPLICAS** without decomposing the modular monolith. ## DO NOT ADD YET * microservices; * sharding; * specialized data stores; * service mesh; * complex orchestration solely for anticipated growth; * read replicas without measured read pressure. ## REVISIT TRIGGER Reopen architecture evolution when measured workload demonstrates a persistent constraint such as: * API saturation correlated with unacceptable latency/errors; * worker arrival rate exceeding processing capacity; * queue-age SLA violation; * database pressure remaining material after optimization; * independent workload requiring distinct scaling or isolation. Where no existing baseline defines acceptable limits: **BASELINE-RELATIVE THRESHOLD REQUIRED** ## STATUS **HARDEN BEFORE GROWTH** No supplied evidence demonstrates that the current architectural model has reached its scalability limit. Scalability has **not** been validated because no load-test or production-capacity evidence was supplied.