### 📋 SOP: Production Database Password Rotation * **Document ID:** SOP-AUTO-F8B2 * **Primary Owner/Role:** Database Administrator (DBA) / SecOps Engineer * **Technical Depth:** Advanced Engineering * **Compliance Framework:** ISO 27001 (Control A.9 - Access Control & Control A.18 - Compliance) * **Process Objective & DoD:** Ensure the secure, zero-downtime rotation of production database credentials to mitigate unauthorized access risks. The Definition of Done (DoD) is achieved when credentials are completely updated across the database engine, applications, and secrets manager, verification logs are generated, and connection continuity is validated without performance degradation. * **Pre-requisites & Tools Required:** * **AWS Secrets Manager / HashiCorp Vault** (Type: Centralized Secrets Store) -> Dependency/Access Level: Admin / Write Access * **Production Database Cluster (e.g., PostgreSQL/MySQL)** (Type: Target System) -> Dependency/Access Level: Superuser / ALTER USER Privileges * **CI/CD Pipeline Deployment Tool (e.g., GitHub Actions/Jenkins)** (Type: Automation System) -> Dependency/Access Level: Trigger / Operator Access * **IAM Execution Role** (Type: Access Credential) -> Dependency/Access Level: Least Privilege policy allowing secret updates and database connection * **Sequential Workflow Steps:** 1. Retrieve the existing database root/application credentials safely from the secure centralized secrets store without printing them to local environment logs. 2. Generate a new cryptographically secure password utilizing an automated generator, enforcing a minimum of 32 characters, uppercase, lowercase, numeric, and special character constraints. 3. Create a temporary secondary database user matching the active user's permissions, or leverage native database multi-user rotation algorithms to prevent connection drop-offs during transition. 4. Inject the newly generated password into the target database engine using an encrypted transport layer (SSL/TLS). 5. Update the primary application connection strings inside the centralized secrets manager to reference the new credential payload. 6. Trigger a rolling restart of all dependent application microservices or containers via the CI/CD deployment platform to force them to fetch the updated configuration cache. 7. Monitor active application connection pools and API error rates synchronously during the deployment phase to verify proper handshakes. 8. Revoke or deprecate the legacy credentials inside the database engine once all application nodes report stable connections. * **Dynamic Decision Tree & Error-Handling:** * **Step ID: 2.4** - Main Action: Injecting the new password into the database engine * *Potential Failure / Condition:* Database connection timeout or authentication failure with the master administrative account during execution. * *Contingency Protocol:* IF connection fails, THEN abort the rotation pipeline immediately, flag an alert to the SecOps incident channel, and maintain the existing active credentials in the applications to guarantee zero service disruption. * **Step ID: 2.6** - Main Action: Application rolling restart configuration fetch * *Potential Failure / Condition:* Microservices crash loop (CrashLoopBackOff) due to immediate access denied errors with the new credentials. * *Contingency Protocol:* IF application instances fail to authenticate, THEN immediately rollback the secrets store configuration to the legacy credentials, trigger an immediate emergency redeployment of the apps, and initiate a full diagnostic audit on user permission syncs. > ⚠️ **CRITICAL SAFETY BOUNDARY:** Under no circumstances should database credentials be hardcoded in application source code repositories, deployment scripts, or environment variables. All rotations must execute through an encrypted, memory-only execution context via the Secrets Manager. Failure to do so exposes production infrastructure to credential harvesting. > * **Compliance Checklist:** * [ ] Generate cryptographic 32+ character random alphanumeric string. * [ ] Stage password within the secrets manager prior to database modifications. * [ ] Verify successful database authentication utilizing the new credentials from an isolated network node. * [ ] Confirm zero authentication error spikes across production application monitoring dashboards. * [ ] Purge any cached memory instances containing the legacy password artifacts. * **Exception Logging & Audit Trail:** All transaction operations, API calls to the secrets manager, and execution telemetry must be recorded natively into the secure SIEM log stream (e.g., AWS CloudTrail or central syslog system). Captured data payloads must include the automated Execution Timestamp, Triggering IAM User ID, Document ID (SOP-AUTO-F8B2), and Success/Failure Status. Raw passwords must never be captured or printed within any log files or system outputs.