Platform Outage
Incident Report for Universe
Postmortem

On Friday, November 2 at 11:22AM EST, migrations were deployed to run on our PostgresQL database. One of the migrations added an index to a new column on a large table. This index was not added concurrently and at 11:33AM EST caused an outage due to a locked table. During this time, we paused the checkout queue to prevent user checkout errors for the duration of the outage. At 11:40AM EST, the migrations were completed and the platform outage was resolved. We are currently investigating further actions to prevent this happening in the future and are in the process of adding automated safety checks on possibly dangerous migrations.

Posted 7 months ago. Nov 02, 2018 - 13:12 EDT

Resolved
Service has been restored.
Posted 7 months ago. Nov 02, 2018 - 11:41 EDT
Identified
We believe we have identified the issue, and are taking remediation steps now. We believe a resolution is pending in approximately 5 minutes. We have paused the checkout queue inflow, to preserve waiting buyers.
Posted 7 months ago. Nov 02, 2018 - 11:40 EDT
Investigating
We are investigating a platform outage.
Posted 7 months ago. Nov 02, 2018 - 11:36 EDT
This incident affected: API and Ticket Processing Queue.