# Content Moderation System Design: Second-Hand Marketplace **Assumptions (all starting points to tune):** - 2M listings/day with ~4 photos each, so ~8M images/day. - Chat volume is ~10M messages/day, not stated in the brief. - Peak is 4x average: ~90 listings/s, ~370 images/s, ~460 chat messages/s at peak. - The 300 ms budget is p99 for the synchronous listing-submit decision. Chat gets a tighter 100 ms text-only budget. Heavier analysis runs asynchronously. - Overall violation prevalence is ~1-2% of listings. Prevalence for the rarest severe categories is far lower than 0.1%. - Chat is not end-to-end encrypted and the privacy notice permits automated scanning. **Confirm both with qualified counsel.** - The team is 4 engineers plus an outsourced labeling and review vendor. --- ## 1. Policy-to-label mapping ### Label taxonomy | Group | Label | Surface | Tier | |---|---|---|---| | Scams | SCAM-DIVERT (steering payment or conversation off-platform) | chat, listing | S2 (S1 when paired with account risk) | | | SCAM-PAYFRAUD (fake payment, overpayment, fake escrow or shipping-proof schemes) | chat | S1 | | | SCAM-FAKELIST (non-existent item, bait pricing, reused or stock imagery) | listing | S1 | | | SCAM-PHISH (credential-harvesting or malicious links) | chat, listing | S1 | | | SCAM-ACCT (fake, duplicate, or compromised accounts) | account-level, behavioral | S2 | | Prohibited items | PROH-CSAM (child sexual abuse material) | image, text | S0 | | | PROH-WEAPON (firearms, explosives, restricted weapon parts) | listing | S0/S1 | | | PROH-DRUG (illegal drugs, paraphernalia, controlled medication) | listing | S1 | | | PROH-COUNTERFEIT | listing | S1/S2 | | | PROH-STOLEN | behavioral signal only | S2 | | | PROH-REGULATED (age-restricted, recalled, wildlife, medical; jurisdiction-dependent) | listing | S2 (geo-gated) | | Chat harassment | HARASS-THREAT | chat | S0/S1 | | | HARASS-HATE (identity-based abuse) | chat | S1 | | | HARASS-SEXUAL | chat, image | S1 | | | HARASS-DOX (sharing private personal information) | chat | S1 | | | HARASS-ABUSE (insults, sustained bullying) | chat | S2 | | | SPAM (bulk or repetitive unsolicited messaging) | chat, listing | S3 | Each label has a definition document with an in-scope/out-of-scope decision tree, per-jurisdiction notes, and a version number. Labels are multi-label, not mutually exclusive: one listing can be both SCAM-FAKELIST and PROH-COUNTERFEIT. ### Severity tiers and enforcement | Tier | Meaning | Automated action | Human involvement | Account consequence | |---|---|---|---|---| | **S0 Critical** | Severe real-world harm, zero tolerance | Block or hide instantly, preserve evidence | Specialist review, SLA ≤15 min | Immediate suspension pending review | | **S1 High** | Clear policy violation, high harm | Auto-block or hide above threshold, hold in the review band | General queue, SLA ≤2 h | Strike: 1st is removal plus warning, 2nd within 90 days is a 7-day suspension, 3rd is a ban (starting points) | | **S2 Medium** | Likely violation, contextual | Friction: inline warning, contact-info masking, hold with limited visibility, or hold for review | Queue or sampled, SLA ≤24 h | Soft signal fed into the account risk score | | **S3 Low** | Nuisance or borderline | Label, downrank, rate-limit, nudge | Audit sampling only | None | **Trade-off:** I deliberately gave SCAM-DIVERT S2 friction (a warning to the buyer) rather than S1 removal. Many legitimate users share contact details, and a hard block would create high false positives. The precision of the warning improves a lot once account-risk features are fused in. **Legal flags for counsel:** - Child-safety detection and reporting obligations. - Regional notice-and-appeal and transparency requirements for platforms. - Which items count as prohibited or regulated in each operating jurisdiction. - Any duty to preserve evidence or cooperate with law enforcement. I make no legal conclusions here. --- ## 2. Data strategy ### Sourcing | Source | Use | Caveat | |---|---|---| | Historical moderation decisions and removals | Primary positives | Biased toward what past systems caught | | User reports | Candidate positives, hard negatives | Noisy; reporting behavior varies by community | | Buyer disputes, chargebacks, refund outcomes | Delayed scam labels | Lag of days to weeks; use for training the scam fusion model, not for real-time evaluation | | Random sample of live traffic | Unbiased prevalence and negatives | Needed for honest FPR measurement | | Public toxicity and abuse datasets | Pretraining or warm start for harassment | Check licenses; domain mismatch with marketplace chat | | LLM-assisted synthetic data | Augment rare classes, red-team variations | Generated at category level, human-verified, never used in test sets | | Hosted-API pre-labels | Bootstrap labeling to cut annotator cost | Humans verify; treat as weak labels | ### Labeling guidelines - One decision tree per label, with 20-30 adjudicated edge cases per label. Examples of legitimate items that superficially resemble prohibited categories (for instance, lawful antiques or hobby equipment) are kept in an internal, access-controlled guideline doc, not here. - Tag every label with context needed (chat window, seller history, listing category, price relative to category median). - Version guidelines. Any change triggers relabeling of the gold set. ### Annotator agreement - Triple-label the first ~1,000 items per label, then double-label with adjudication on disagreement. - Target Krippendorff's alpha ≥0.70 per label (≥0.80 for S0/S1). If a label stays below 0.60, fix the guideline or merge or split the label before collecting more data. - Inject a gold set (200-500 items) at 5-10% of annotator work for ongoing quality tracking. ### Class imbalance - Weighted loss or focal loss for training. Never resample the test set. - Hard-negative mining: pull high-scoring benign items from production audits and appeal overturns. - Active learning: weekly pull of ~2,000 items near the decision boundary for labeling. - Initial labeling budget (starting point): ~25k listings and ~25k chat items, oversampled toward rare labels, at an assumed vendor cost of $0.05-0.30 per item. ### Bias audit (before every model promotion) - Slice by language, dialect and code-mixing, region, item category, seller tenure, and community. - Check for vocabulary-driven false positives where a word is normal in a legitimate product category (for example hobby, hunting, or kitchen goods) but triggers a prohibited-item label. - Check for harassment-model over-flagging of dialects, reclaimed in-group language, or heated but legitimate price negotiation. - Compare FPR across slices (see Section 4 for gap thresholds). ### Preprocessing **Text (listings and chat):** - Unicode NFKC normalization, zero-width and control character stripping, homoglyph folding, and whitespace collapse. Keep the original alongside the normalized text. - Language ID, then per-language routing. - Extract structured signals: URLs, phone and email patterns, payment-app mentions. - Chat: use a rolling window of the last 5-10 messages with speaker roles (buyer or seller). **Images:** - Strip EXIF, resize to 224-384 px for models, and compute a perceptual hash (PDQ or pHash) plus a CLIP/SigLIP-style embedding. - Run OCR for text embedded in photos, since it is a common evasion route. - Run exact and near-duplicate detection to catch relisted removed content and stock-image reuse. **Privacy:** redact phone numbers, emails, and addresses before data goes to annotators or hosted APIs. **Annotator wellbeing:** exposure limits, rotation, and support for reviewers of severe content are required. Confirm obligations with counsel and HR. --- ## 3. Model architecture ### Layered pipeline ``` Upload-time (async, parallel to user filling the form): images → hash + embedding + OCR + image heads → cached scores Submit-time (sync, p99 ≤300 ms): L0 rules/hash/blocklists → L1 fast classifiers → L2 fusion (GBM + account features) → policy engine → action ↓ (grey band only) L3 heavy model (async) → review queue / auto-resolve ``` ### L0: Rules, hash matching, blocklists (5-20 ms) | Option | Cost | Latency | Accuracy | |---|---|---|---| | **A. In-house exact + perceptual hash (PDQ/pHash) against removed-content index, URL and domain blocklists, regex for structured patterns** | Very low | 1-10 ms | Very high precision on known-bad content, near-zero recall on novel content | | **B. Industry hash-sharing programs and third-party URL reputation feeds** | Low-moderate, plus membership process | 10-30 ms (API) or local index | High precision on known severe content; needed for PROH-CSAM | **Choice:** both. A is the backbone and B is non-negotiable for severe-harm hash matching. Membership and reporting-workflow requirements should be confirmed with counsel. L0 handles roughly 5-10% of violations with essentially no false positives. ### L1: Fast classifiers **Text** | Option | Cost | Latency | Accuracy | |---|---|---|---| | A. TF-IDF/fastText linear models | Lowest | 1-3 ms CPU | Moderate; fails on obfuscation and multilingual nuance | | B. Distilled multilingual transformer (e.g., a MiniLM/DistilBERT/XLM-R-base class model, int8 ONNX) | Low-moderate | 10-40 ms CPU, 3-8 ms on a small GPU with batching | High; best accuracy/cost balance | | C. Hosted moderation or LLM API per message | High at this volume | 150-600 ms | High on generic toxicity, weaker on marketplace-specific scams | **Choice:** B as the main model, A as a cheap pre-filter and fallback. C is not viable in the synchronous path. At ~12M texts/day, hosted pricing of even $0.20-1.00 per 1,000 short texts costs $2.4k-12k per day (assumed pricing, check current rates). **Image** | Option | Cost | Latency | Accuracy | |---|---|---|---| | A. Frozen CLIP/SigLIP-class embeddings + small linear/MLP heads per label | Low | ~10-25 ms/img on GPU batched | Good; cheap to add labels | | B. Fine-tuned small vision model (EfficientNet/ViT-S class) | Moderate | 10-30 ms/img GPU | Higher on specific classes, needs more labeled data | | C. Hosted vision moderation API | ~$1-2 per 1,000 images (assumed) | 100-400 ms | Strong on generic categories, weak on marketplace-specific ones | **Choice:** A first (fastest to ship with 4 engineers, shares one embedding across all labels and the near-duplicate index), B for the highest-volume or worst-performing labels once there is data. At 8M images/day, C would cost ~$8-16k/day, so it is used only for escalations (<1-2% of images). ### L2: Fusion layer | Option | Cost | Latency | Accuracy | |---|---|---|---| | A. Hand-written rules combining scores | Lowest | <1 ms | Brittle, hard to maintain | | B. LightGBM stacker on model scores + account features (account age, listing velocity, price vs. category median, device and IP reuse, link and contact-pattern counts, prior strikes, dispute history) | Low | 2-5 ms | Much higher on scams, where behavior beats content | **Choice:** B, with rules kept only as hard overrides. For scams, account and behavioral features are typically the strongest signals, which is why this layer matters. ### L3: Heavy model for ambiguous cases (async, grey band ~2-5% of traffic) | Option | Cost | Latency | Accuracy | |---|---|---|---| | A. Hosted multimodal LLM/VLM API with structured output | ~$3-10 per 1,000 calls (assumed) | 1-5 s | High on nuance, flexible to policy changes | | B. Self-hosted fine-tuned open VLM or large cross-encoder | GPU-heavy, needs MLOps | 0.5-2 s | High after fine-tuning, cheaper at high volume, better privacy | **Choice:** A for the first 90 days (no GPU ops burden), revisit B if L3 volume exceeds ~300-500k calls/day or privacy requirements tighten. **Security note:** listing and chat content can contain text aimed at manipulating an LLM. Treat content strictly as data, enforce structured JSON output, and give the L3 model no tools or side effects. ### Fine-tune versus hosted API | Situation | Choice | |---|---| | High volume (>~500k inferences/day), sync latency <100 ms, stable labels, plenty of in-domain data | Fine-tune and self-host | | Cold start, low-volume escalation, fast-changing policy, long-tail languages | Hosted API | | Privacy-sensitive content (chat) | Prefer self-hosted, or hosted with zero-retention contract and redaction | | Need labels fast | Hosted LLM as pre-labeler with human verification | ### Latency budget (listing submit, p99 target 300 ms) | Step | Budget | |---|---| | Gateway, auth, request parsing | 15 ms | | Fetch cached image scores/hashes | 15 ms | | Text normalize + language ID | 10 ms | | L0 rules, hash and blocklist lookup | 15 ms | | L1 text model | 40 ms | | Account features from Redis | 10 ms | | Fusion + policy engine | 10 ms | | Write decision + emit events | 20 ms | | **Subtotal** | **~135 ms** | | Headroom for tails and retries | ~165 ms | If photos are not yet scored at submit (fast users), the listing enters "pending visibility" (published but not surfaced in search or feeds) until scoring completes, typically within seconds. ### Capacity sketch (rough, to tune) - Text: ~700 req/s peak on roughly 8-10 CPU inference pods, or 2 small GPUs with batching. - Images: ~370 img/s peak on ~3 L4-class GPUs with autoscaling, minus ~20% skipped via duplicate-hash caching. --- ## 4. Evaluation ### Metrics (per label, never just aggregate) - **Precision** = TP / (TP + FP) - **Recall** = TP / (TP + FN) - **F1** = 2PR / (P + R). It is a summary only; decisions use precision at a fixed recall for S0/S1 and recall at a fixed precision for auto-action thresholds. - **False positive rate** = FP / (FP + TN). At this scale FPR is the number that matters operationally. - Also track PR-AUC (ROC-AUC is misleading under heavy imbalance), calibration error (ECE), and per-label review-band volume. ### Why false positives matter here - Sellers are the supply side. A wrongly blocked listing costs the seller time, and repeated wrongful blocks drive churn of exactly the users who make the marketplace liquid. - At 2M listings/day with ~1.98M benign, **a 0.1% FPR is ~2,000 wrongly blocked listings per day.** Even 0.01% is ~200. - False strikes can lead to wrongful suspensions, and appeals consume review capacity. - Legitimate listings often superficially resemble prohibited categories, so precision problems concentrate on specific communities and item categories. - Starting-point targets: auto-action precision ≥0.95 on S1 labels and FPR ≤0.05% on auto-block. S0 prioritizes recall, accepting more review load because the action is hold-then-review, not permanent punishment, except on exact hash matches. ### Building the test set | Component | Purpose | Starting size | |---|---|---| | Random live sample (stratified by language and category, with sampling weights) | Unbiased prevalence, FPR, and negatives | 20-30k items | | Targeted positives per label | Enough positives to measure recall | ≥300 per label (≥500 for S0/S1) | | Hard negatives (legitimate lookalikes, appeal overturns) | Stress precision | ~2-5k | | Temporal holdout | Mimic deployment; train up to t-14 days, test the last 14 | rolling | | Adversarial set (from red-teaming) | Robustness | versioned, grows each quarter | | Frozen golden set | Regression gate across versions | 1-2k, adjudicated by senior reviewers | **Sizing logic:** recall of ~0.90 with a ±0.05 95% CI needs about 140 positives (n ≈ 1.96²·p(1−p)/d²), so 300+ gives margin for slice analysis. If you observe zero false positives in n clean negatives, the 95% upper bound on FPR is about 3/n (rule of three), so verifying FPR ≤0.05% requires at least ~6,000 clean negatives and ~20k gives comfort. Use bootstrap CIs on every reported metric. Keep the test set out of labeling-guideline iteration, and refresh it quarterly, because the frozen version becomes stale. ### Slice analysis - Slices: language (top ~10 plus code-mixed), region and dialect, item category, seller tenure (new vs. established), seller type (individual vs. professional), listing price band, and communities defined by category or locale. - Minimum slice size ~200 positives and ~2,000 negatives; otherwise report as "insufficient data" rather than a number. - **Flag a slice** if FPR exceeds 1.5x overall FPR, or recall is more than ~10 points below the overall figure. A flagged slice blocks promotion unless counsel and the policy lead sign off on an exception. - Do not infer protected attributes from user data to run slices. Use language, region, and category proxies, and consult counsel on fairness and discrimination considerations. ### Offline versus online | Stage | What it answers | Limits | |---|---|---| | Offline | Does the new model beat the old on frozen and fresh sets, per label and slice? | Test set staleness, label noise | | Shadow mode (1-2 weeks) | How does it behave on live traffic with no user impact? Score distribution, latency, disagreement with the current model | No feedback on outcomes | | Canary (1% then 5-10%) | Real action-rate, appeal-rate, and report-rate changes | Must have an automatic rollback | | Continuous audit | Ground truth in production: 0.1-0.5% random sample of auto-approved and auto-actioned items sent to human review | Costs reviewer time but is the only unbiased recall estimate | **Online proxy metrics:** appeals overturn rate (biased proxy for FPR, because most wrongly actioned users don't appeal), buyer scam reports per 1,000 transactions, dispute and chargeback rates, time-to-removal, listing completion rate, seller churn after enforcement, and GMV impact. Do not estimate recall from user reports alone, since they only see what escaped, not what was never noticed. Don't A/B test with a no-enforcement holdout on S0/S1 labels; use audit sampling instead. --- ## 5. Thresholds and human review ### Per-label thresholds (calibrated probabilities; all numbers are starting points to tune) Calibrate with isotonic regression on a recent held-out set, then choose thresholds to hit the precision target for auto-action. Review-band lower bounds are set to cap review volume. | Label | Auto-action ≥ | Review band | Below band | |---|---|---|---| | PROH-CSAM | Exact hash match: auto | Classifier ≥0.50: hold + specialist | Allow | | PROH-WEAPON | 0.97 block | 0.55-0.97 | Allow | | PROH-DRUG | 0.95 block | 0.60-0.95 | Allow | | PROH-COUNTERFEIT | 0.97 block (brand-resale false positives are costly) | 0.65-0.97 | Allow | | PROH-STOLEN | No content auto-action; behavioral only | Account-risk-driven review | Allow | | PROH-REGULATED | 0.90 age-gate or restrict by region | 0.65-0.90 | Allow | | SCAM-DIVERT | 0.95 warn buyer + mask contact info | 0.60-0.95 inline warning only; queue only on repeat offenders | Allow | | SCAM-PAYFRAUD | 0.95 hide message + warn | 0.65-0.95 | Allow | | SCAM-FAKELIST | 0.97 and account-risk high: hold | 0.70-0.97 | Allow | | SCAM-PHISH | Blocklist exact match, or model ≥0.95 | 0.70-0.95 | Allow | | HARASS-THREAT | 0.90 hide + warn sender | 0.50-0.90 specialist | Allow | | HARASS-HATE / SEXUAL / DOX | 0.95 hide + warn | 0.70-0.95, or on user report | Allow | | HARASS-ABUSE | 0.97 warn only | Review only if reported or repeated | Allow | | SPAM | 0.90 rate-limit | None | Allow | **Escalation by account risk:** lower the auto-action threshold by ~0.03-0.05 for accounts with high fused risk scores (new account, high velocity, prior strikes), and raise it for established sellers with clean history. This is the biggest lever for protecting legitimate sellers without losing recall on scams. ### Review capacity and volume control Assumed capacity: 3,000-5,000 review decisions/day from 8-12 contract reviewers at ~250-600 fast decisions each (10-25 s per item with good tooling). That is 0.15-0.25% of listings. Without control, a naive 3% grey band would be ~60k items/day, which is not feasible. So: - Tune review bands to hit the daily capacity target, with a dynamic cutoff that tightens when the queue backs up. - Items that do not fit the queue get a graceful default: S2 grey items receive limited visibility or friction with auto-expiry after 24-48 h, not indefinite holds. - Chat messages go to review only when reported or when scored S0/S1 in the band. Everything else is handled with friction. ### Queue prioritization Priority score = severity weight × P(violation) × exposure × SLA-age factor, where: - Exposure is the predicted views or reach (price, category popularity, seller follower count). - SLA-age escalates priority as an item nears its deadline. SLAs: S0 ≤15 min on a dedicated specialist queue; S1 ≤2 h; S2 ≤24 h. Separate queues by label family and language, with the reviewer tool showing the listing, photos, chat window, and account history. ### Reviewer quality control Double-review 10% of S1 decisions and 100% of S0 permanent actions. Maintain a 5% QA sample with gold-item injection. Track reviewer-vs-reviewer agreement and reviewer-vs-model disagreement as a drift signal. ### Appeals flow 1. User receives a notice with the violation category, the action taken, and a one-click appeal. 2. A different reviewer (not the original) handles the appeal, with a 48 h SLA (starting point), faster for suspensions. 3. Outcome is communicated with a reason category. A second-level escalation is available for suspensions and bans. 4. Overturns restore content, reverse the strike, and are logged as hard negatives for retraining. 5. Overturn-rate spikes per label or reviewer trigger a threshold review. **For counsel:** statement-of-reasons content, notice timing, appeal rights, and transparency reporting requirements vary by region. This design is not a determination of what applies. --- ## 6. Adversarial robustness and drift ### Evasion tactics (by category) | Tactic category | Targets | Mitigation | |---|---|---| | Character-level obfuscation (substitution, spacing, homoglyphs, zero-width characters) | Text models, blocklists | Aggressive normalization, character-n-gram features, train on augmented variants | | Text embedded in images | Text-only pipelines | OCR on every image, with OCR text routed through the text model | | Image perturbation (crops, flips, filters, border padding, re-encoding) | Hash matching | Perceptual hashes with Hamming-distance tolerance, multi-crop and flipped hashing, embedding kNN against removed content (ANN index) | | Evolving coded vocabulary and euphemisms | Keyword rules, stale models | Embedding-based similarity, weekly mining of new terms from reviewer decisions, multilingual and transliteration coverage | | Moving conversation off-platform through indirect phrasing | SCAM-DIVERT | Behavioral features (message velocity, link or contact patterns, new account plus high-value item), conversation-level scoring | | Benign-first, bait-and-switch edits | Submit-time-only checks | Re-scan on every edit to title, description, price, or photos; periodic re-scan of live high-exposure listings | | Split content across listing and chat | Single-surface models | Cross-surface account-level risk score | | Mass-posting with templated variation, account farming, repost-after-removal | Per-item classifiers | Near-duplicate clustering, device and IP linkage, rate limits, velocity features | | Threshold probing (iterating until a post passes) | Any fixed threshold | Limit feedback detail (broad reason categories only), rate-limit rejected resubmissions, apply account-level escalation after repeated near-misses | | Prompt-manipulation aimed at the heavy LLM layer | L3 | Content treated as untrusted data, structured outputs, no tool access, a classifier on L3 output, and L3 never solely decides S0/S1 | ### Retraining cadence (starting points) | Cadence | What | |---|---| | Daily | Blocklists, hash index, ANN index of removed content | | Weekly | Rule and keyword refresh, active-learning labeling batch, fusion-model refresh if audit shows degradation | | Monthly | Full retrain of text and image heads on fresh labels, with shadow then canary | | Quarterly | Evaluate a larger base-model upgrade, refresh test sets, bias audit | | Triggered | Retrain or hotfix when PSI on score or embedding distributions exceeds ~0.2, audit precision drops more than ~3 points, or a new abuse pattern spikes | ### Drift monitoring Track input drift (embedding centroid shift, language mix, category mix), score drift (per-label score histograms, PSI), and label drift (audit-sample prevalence), alongside performance drift measured on the continuous audit sample. ### Red-teaming - Quarterly, 2 people for ~3 days (an engineer plus a T&S analyst). Test sets are built from category-level prompts and mutation tooling, never published in docs. - Automated mutation suite in CI: character-level and image transformations applied to the frozen positives, with a hard gate on recall degradation (e.g., ≤5 points drop). - Keep red-team sets versioned as an eval slice, and rotate fresh sets so the models do not overfit them. - Between quarters, feed reviewer-discovered evasions back into the suite. --- ## 7. Deployment and monitoring ### Architecture (cloud-agnostic) ``` Client → API gateway → Moderation service (sync path) │ ├─ Redis: feature/score cache, rate-limit counters │ ├─ Model serving (Triton or KServe): text, image-embedding, fusion │ └─ Policy engine (config-driven thresholds + actions) │ Upload service → image pre-scoring workers (GPU) → cache │ Event stream (Kafka/Kinesis/PubSub) ├─ L3 async workers (hosted VLM/LLM API, with redaction proxy) ├─ Review queue service (Postgres) → reviewer tool → appeals ├─ Graph/velocity aggregation (account features) └─ Data lake (object storage, Parquet) → warehouse → training pipelines ``` Design points: - **Decouple image work from submit.** Image scoring starts at upload, so the sync path just reads cached results. - **Stateless sync service** with autoscaling on CPU and queue depth. Autoscale the GPU pool with a CPU-model fallback. - **Idempotent decisions** keyed on content ID and version, so edits and retries are safe. - **Degradation policy** when L1 exceeds ~250 ms or fails: fall back to L0 rules plus the cheap linear model. For high-risk accounts or S0 signals, fail closed (pending visibility), and for low-risk traffic, fail open with async re-scan within minutes. Fail-closed and fail-open choices are per label, and need sign-off from the policy lead. - **Cost (rough monthly, to tune):** GPUs ~$2-4k, CPU and storage ~$3-5k, hosted L3 API ~$3-8k, labeling ~$5-10k, excluding reviewer vendor cost, which dominates and depends heavily on region. ### Logging Every decision logs: content ID, surface, model and rule versions, policy version, per-label scores, fusion inputs summary, threshold applied, action, latency per stage, and reviewer outcome when present. Log raw content separately with short retention, and keep decision metadata longer for audits and appeals. Retention periods are a counsel and privacy decision, so I give none here. ### Dashboards - Volume and action rate per label and surface, plus score histograms. - Latency p50/p95/p99 per stage, error and timeout rates, fallback rate. - Queue depth, age, and SLA attainment per tier. - Audit-sample estimates of precision and recall with CIs, per label. - Appeal rate and overturn rate, per label and per reviewer. - Slice FPR and recall for the major language and category slices. - Cost per 1,000 items, hosted API spend. ### Alerting (starting points) | Alert | Trigger | |---|---| | Latency | p99 >250 ms for 5 consecutive minutes | | Errors/timeouts | >1% over 5 minutes | | Action-rate anomaly | Per-label action rate beyond 3σ of the same-hour 7-day baseline | | Score drift | PSI >0.2 on any label | | Queue breach | S0 queue over 10 min of oldest-item age, S1 over 1.5 h | | Appeals | Overturn rate for a label above 2x the 4-week baseline | | Audit | Estimated precision drops more than 3 points | ### Rollback - Thresholds and actions live in versioned config, so a change or revert takes effect in seconds without a deploy. - Models roll out behind flags: shadow, then 1%, 10%, 50%, 100%, with automatic rollback if guardrail metrics (action rate, latency, appeal rate) breach. - Per-label kill switch to downgrade from auto-action to review-only, and a global "safe mode" that reverts to the last known-good model and thresholds. - Keep the previous two model versions warm. ### Privacy controls - Data minimization: log scores and hashes by default, and keep raw content only as long as needed. - Redact phone numbers, emails, and addresses before sending anything to hosted APIs or annotators, and use contracts with zero data retention and no training on the data. - Role-based access control for reviewers, with masked personal data in the reviewer tool where not needed to decide. - Encryption in transit and at rest, audit logs on all content access, and regional data residency where required. - Deletion workflow that propagates to training data and logs. **For counsel:** lawfulness of scanning private messages, user notice and consent language, data retention periods, cross-border transfers to hosted vendors, and data-subject request handling. --- ## 8. 30/60/90-day roadmap **Team (assumed):** Eng 1 ML (text and chat), Eng 2 ML (vision and fusion), Eng 3 backend and infrastructure (serving, policy engine, logging), Eng 4 tooling and data (review tool, labeling pipeline, dashboards). ### Days 0-30: Foundations and fast wins - Finalize the taxonomy, tiers, and actions with policy and legal. Start the counsel review of the legal flags above. - Ship L0: hash index, blocklists, URL reputation, structured-pattern rules, and a severe-harm hash-matching integration (contingent on program membership). - Build the decision engine skeleton, config-driven thresholds, and full decision logging. - Stand up labeling with a vendor, guidelines v1, and a gold set. Label the first ~10k items. - Use a hosted API for bootstrap pre-labeling and as a temporary L1 for the highest-risk labels (S0/S1 prohibited items) on async paths. - Minimal review tool, bought or built, with queues for S0 and S1. - Run everything in shadow mode, with no user-facing enforcement except L0 exact matches. **Exit criteria:** shadow decisions logged for 100% of listings; the random audit sample is flowing; guidelines reach alpha ≥0.70 on the first labels. ### Days 31-60: Models and review - Fine-tune the distilled multilingual text model and the image embedding heads, and ship the account-feature GBM fusion for scams. - Image pre-scoring at upload, plus the OCR path and near-duplicate index. - Calibrate thresholds, then set review bands to fit capacity. - Build the offline eval harness, frozen golden set, and first slice analysis. - Canary at 1-5% for S1 prohibited items and scam listings, with the latency budget validated under load. - Onboard reviewers with QA sampling and double-review. **Exit criteria:** meet precision targets on S1 labels in canary with no flagged slice gaps; sync p99 ≤300 ms at 2x peak load test. ### Days 61-90: Chat, appeals, and hardening - Chat pipeline (100 ms budget) covering harassment and SCAM-DIVERT/PAYFRAUD with friction-based actions. - Appeals flow, user notices, and the strike system. Counsel review of the notice wording before launch. - Dashboards, alerting, rollback runbook, and a rollback drill. - First red-team sprint and the mutation test suite in CI. - Automate the monthly retraining pipeline and weekly active learning. - Cost review, including whether L3 volume justifies self-hosting. **Exit criteria:** full listing path live behind canary-to-100% rollout with automated rollback; chat live for at least the top three harassment labels; first retrain completed end to end. **Deliberately deferred beyond 90 days:** graph-based ring detection, a self-hosted fine-tuned VLM, deep long-tail language coverage, and advanced seller reputation scoring. Four engineers cannot do these safely alongside the above. --- ## Five biggest risks and mitigations 1. **False positives hurting honest sellers at scale.** Even a 0.1% FPR means ~2,000 wrongful blocks per day. *Mitigation:* high precision targets for auto-action, account-risk-adjusted thresholds, a friction-first approach for S2, per-slice FPR gates, a fast appeals path, and appeal overturns fed back as hard negatives. 2. **Label scarcity and quality for rare, severe categories.** Positives are scarce, definitions are fuzzy, and annotator exposure is a wellbeing concern. *Mitigation:* targeted sampling plus active learning, hosted-LLM pre-labels with human verification, alpha-gated guidelines, hash matching for the most severe content (not ML), specialist reviewers with wellbeing support, and counsel review of reporting duties. 3. **Latency or availability failures in the 300 ms path.** A slow moderation call blocks listing creation. *Mitigation:* image scoring at upload so the sync path reads cache, a lean text model, strict per-stage budgets, per-label fail-open and fail-closed degradation rules, and load testing at 2x peak before each rollout. 4. **Adversaries adapting faster than a 4-person team can retrain.** Evasion and vocabulary drift erode recall quietly. *Mitigation:* behavior-based features that are harder to evade, normalization, OCR, and embedding-based matching, a daily-refreshed removed-content index, drift alerts, a continuous random audit for unbiased recall, and quarterly red-teaming with CI mutation tests. 5. **Review capacity, legal exposure, and team bandwidth.** Overflowing queues cause SLA misses on S0/S1, while privacy and regional compliance are easy to underinvest in. *Mitigation:* dynamic band tuning to fit capacity, graceful auto-expiring defaults for S2, strict prioritization, a roadmap that defers nonessential work, and early counsel engagement on chat scanning, data retention, child-safety reporting, and notice-and-appeal obligations.