During our scheduled maintenance window, we experienced a temporary disruption in service availability. This resulted in database connectivity errors for some of our services.
The disruption was caused by an unexpected behavior change in our database connection pooling layer after a routine component update. The update enforced stricter SSL parameter negotiation than previous versions, which prevented the pooler from establishing secure connections to the backend database.
This was an undocumented upstream edge case and was not caught by our pre-deployment testing.
We detected the issue quickly and reverted the update, after which connectivity was restored. We are tracking the upstream fix and will roll it out only after additional validation.
Impact was limited because maintenance is performed during low-traffic windows, but we apologize for the disruption.
To reduce the likelihood of similar incidents, we are:
Further aligning staging and production configuration for database connectivity and SSL behavior.
Strengthening our maintenance readiness checks, including reviewing relevant upstream issues and release notes before changes are applied.
As always, we are committed to improving reliability and continuously refining our processes.
Resolved
During our scheduled maintenance window, we experienced a temporary disruption in service availability. This resulted in database connectivity errors for some of our services.
The disruption was caused by an unexpected behavior change in our database connection pooling layer after a routine component update. The update enforced stricter SSL parameter negotiation than previous versions, which prevented the pooler from establishing secure connections to the backend database.
This was an undocumented upstream edge case and was not caught by our pre-deployment testing.
We detected the issue quickly and reverted the update, after which connectivity was restored. We are tracking the upstream fix and will roll it out only after additional validation.
Impact was limited because maintenance is performed during low-traffic windows, but we apologize for the disruption.
To reduce the likelihood of similar incidents, we are:
Further aligning staging and production configuration for database connectivity and SSL behavior.
Strengthening our maintenance readiness checks, including reviewing relevant upstream issues and release notes before changes are applied.
As always, we are committed to improving reliability and continuously refining our processes.
Identified
An incident occurred during routine maintenance.
