# Senior Backend Engineer Interview Rubric: Distributed Rate Limiter ## 1. Interview Overview & Scoring Weights - **Role:** Senior Backend Engineer - High-Throughput Ingestion - **Format:** 60-minute practical coding and architecture session. Suggested split: 0-10 min requirements and capacity math, 10-20 min architecture, 20-45 min core implementation, 45-60 min failure-mode probing. - **Stack:** Java 21, Spring Boot, Kafka, Redis, DynamoDB - **Problem:** API rate limiter with distributed sliding windows and burst handling at 100k RPS - **Core Evaluation Dimensions:** - Problem Solving & Architecture (30%) - Code Quality & Reliability (25%) - Edge-Case Resilience (25%) - Pragmatism & Debt Management (20%) - Communication and trade-off articulation is scored inside each dimension, not separately. ## 2. Standardized Scoring Matrix | Dimension | Does Not Meet (1) | Partially Meets (2) | Meets Senior Bar (3) | Exceeds Bar (4) | | :--- | :--- | :--- | :--- | :--- | | **Problem Solving & Architecture** | Starts coding without clarifying requirements. Proposes an in-memory counter or a DynamoDB write per request. Does no throughput math. | Chooses Redis but cannot justify the sliding-window algorithm. Throughput math is partial (e.g., ignores key cardinality or per-node load). Treats burst as an afterthought. | Clarifies limits, keys (per user/API key/tenant) and fail behavior up front. Estimates Redis ops/sec and picks an algorithm (sliding window counter or log) with stated accuracy and memory trade-offs. Uses Redis for hot-path state, DynamoDB for policy config, Kafka for async usage events. Adds token-bucket burst allowance explicitly. | Everything in "Meets", plus a layered design (local pre-check, then Redis). Proposes sharding or hash-tagging strategy and a plan for hot tenants. Quantifies the accuracy-vs-cost trade-off between the sliding log and the weighted two-window approximation. Anticipates multi-region behavior. | | **Code Quality & Reliability** | Check-then-increment across separate Redis calls. Shared mutable state without synchronization. No timeouts. Untestable static or global usage. | Code works on the happy path. Uses a Lua script or atomic op, but the interface is tightly coupled to Redis. Timeouts are missing or unbounded. Few or no tests. | Atomic Redis operation (Lua script or equivalent) for check-and-update. Clean `RateLimiter` interface with injected clock and store. Explicit timeouts and a defined fail-open/fail-closed policy. Correct HTTP semantics (429, `Retry-After`). Writes at least one deterministic test for the window logic. | Everything in "Meets", plus idiomatic Java 21 use (records for decisions, sealed result types, virtual-thread awareness). Script SHA caching and `NOSCRIPT` handling. Metrics hooks (allowed/denied/fallback counters, latency histograms). Tests include concurrency and clock-edge cases. | | **Edge-Case Resilience** | Identifies no failure modes unprompted. Collapses when asked "what if Redis is down?" or "what if one tenant is 40% of traffic?" | Identifies one or two failure modes (usually Redis outage) but offers shallow mitigations. Misses clock, boundary, and hot-key issues. | Independently raises at least three of: Redis outage, hot key, window-boundary double burst, clock skew, retry storms. Gives a concrete mitigation for each. | Raises most of the above unprompted. Discusses second-order effects (a fallback limiter creating a thundering herd on recovery, local-limit drift across N instances). Defines how they would test or chaos-test each one. | | **Pragmatism & Debt Management** | Either builds nothing shippable in 60 min or gold-plates a single component. No acknowledgment of what was skipped. | Makes shortcuts without labeling them, or over-engineers (adds Kafka to the hot path). Debt is acknowledged only when asked. | Ships a working core, then names deliberate shortcuts (static limits, single region, approximate accuracy) with a mitigation or follow-up plan. Keeps boundaries clean so shortcuts are replaceable. | Everything in "Meets", plus sequences a rollout (shadow mode, then log-only, then enforce). Identifies which shortcuts are safe at 10k RPS but not at 100k RPS. Defines explicit exit criteria for each debt item. | ## 3. Targeted Technical Edge Cases - **Case 1: Hot Tenant Saturates a Single Redis Key** - *Trigger Condition:* One API key generates 30-40% of the 100k RPS. All traffic for that key hits one Redis key, and therefore one shard and one thread. - *Expected Senior Response:* Recognizes the single-key ceiling. Proposes splitting the budget across sub-keys (e.g., `key:{tenant}:{0..N}`), or leasing batches of tokens to each app instance and syncing periodically. States the accuracy cost of each approach. - *Red Flag:* Says "Redis is fast enough" without computing per-key ops/sec, or proposes scaling Redis Cluster nodes without noticing that one key maps to one slot. - **Case 2: Check-Then-Increment Race** - *Trigger Condition:* Two or more instances read the current count, both see it under the limit, and both increment, so the limit is exceeded under concurrency. - *Expected Senior Response:* Names the race immediately. Moves the read, evaluation, and write into a single atomic Lua script (or equivalent atomic command sequence). Explains why `MULTI/EXEC` alone does not give conditional logic. - *Red Flag:* Uses `GET` then `INCR` as separate calls, or adds a JVM-level `synchronized` lock and believes it solves a distributed problem. - **Case 3: Redis Outage or Latency Spike** - *Trigger Condition:* Redis failover takes several seconds, or p99 latency jumps above the request budget. The interviewer asks "what happens to the API now?" - *Expected Senior Response:* Sets a tight client timeout and a circuit breaker. Makes an explicit, justified fail-open or fail-closed choice per endpoint class. Proposes a coarse local limiter as a fallback. Considers recovery behavior (avoiding a thundering herd or counter reset exploit). - *Red Flag:* No timeouts, so request threads (including virtual threads) pile up behind a dead dependency. Or defaults to fail-open for everything without discussing abuse risk. - **Case 4: Window Boundary Burst and Clock Skew** - *Trigger Condition:* Fixed-window counters allow 2x the limit across a boundary. Separately, instances compute window start from their own clocks, which drift. - *Expected Senior Response:* Explains why the sliding window fixes the boundary problem. Uses the sorted-set log (exact, memory-heavy) or the weighted two-window counter (approximate, cheap) and states the trade-off. Takes time from Redis (`TIME` inside the script) rather than from app servers. - *Red Flag:* Uses `System.currentTimeMillis()` on each app node as the source of truth, or cannot explain why the fixed-window approach fails. - **Case 5: Retry Storm After Throttling** - *Trigger Condition:* Clients receive 429s and retry immediately and in sync, amplifying load. Also tests whether the candidate's design handles bursts without punishing legitimate spikes. - *Expected Senior Response:* Returns `Retry-After` and rate-limit headers. Recommends client backoff with jitter. Combines a sliding window for the sustained rate with a token bucket for burst capacity. Notes that rejected requests should be cheap (reject before expensive work). Mentions Kafka only for async usage and audit events, not for the decision path. - *Red Flag:* Treats burst handling as "raise the limit". Puts Kafka or DynamoDB in the synchronous decision path. Has no plan for synchronized client retries. ## 4. Technical Debt Tolerance Indicators - **Healthy Pragmatism (Positive Indicators):** - Says, for example, "I'll hardcode per-tier limits now and load them from DynamoDB later. The `PolicyProvider` interface lets me swap that in without touching the limiter." The shortcut is named, scoped, and replaceable. - Defines clear seams (`RateLimiter`, `WindowStore`, `PolicyProvider`, `Clock`) so Redis, the algorithm, and the config source can each change independently. - Proposes shadow mode or log-only enforcement before blocking real traffic. - **Reckless Shortcuts (Negative Indicators):** - Ignores non-functional requirements: no latency budget, no memory estimate for the sliding log at 100k RPS, no metrics. - Omits timeouts, the Redis failure path, or the fail-open/fail-closed decision, and does not acknowledge the omission. - Leaves non-atomic check-and-update in place with "we'll fix it later". - **Over-Engineering (Negative Indicators):** - Puts Kafka consumers or DynamoDB conditional writes in the per-request path. - Builds a generic multi-algorithm plugin framework, multi-region consensus, or a custom protocol before the single-region core works. - Spends more than 15 minutes on one component (e.g., the Spring configuration or a perfect abstraction) and leaves the core algorithm unfinished. ## 5. Decision Heuristic - **Clear Hire Threshold:** - Weighted score of 3.0 or higher. - No dimension below 2, and at least two dimensions at 3 or higher, one of which must be Problem Solving & Architecture or Edge-Case Resilience. - The core atomic check-and-update logic is working or correctly specified by the end of the session. - A weighted score of 3.5 or higher with Edge-Case Resilience at 3 or higher should be flagged for consideration at a higher level. - **Hard Disqualifiers:** - Cannot identify or fix the check-then-increment race even after a direct hint. - Proposes a per-request DynamoDB write or a Kafka round trip on the hot path and cannot be moved off it after discussion. - No fail-open/fail-closed reasoning at all after being asked about a Redis outage. - Fabricates or misrepresents technical claims (e.g., asserts `MULTI/EXEC` gives conditional atomicity) and does not correct after probing.