Primary database outage

Incident Report for Universe

Resolved

GCP reports all issues have now been resolved.

Posted Mar 27, 2020 - 17:23 EDT

Update

Our upstream cloud provider (GCP) is continuing their work to resolve degraded performance.

Progress can be tracked here: https://status.cloud.google.com/incident/cloud-networking/20004

Posted Mar 27, 2020 - 17:07 EDT

Update

We are continuing to closely monitor the root cause: https://status.cloud.google.com/incident/cloud-networking/20004

Posted Mar 27, 2020 - 16:49 EDT

Monitoring

Database access has been mostly restored - we're continuing to closely monitor performance as we ensure a smooth recovery to 100%

Posted Mar 27, 2020 - 16:24 EDT

Identified

We've isolated the issue to a widespread outage at our upstream database provider, who is working hard to restore service for many affected clients (universe included)

Posted Mar 27, 2020 - 16:11 EDT

Update

We're currently working with our upstream service providers to identify & resolve the issue

Posted Mar 27, 2020 - 15:55 EDT

Investigating

We are currently investigating this issue.

Posted Mar 27, 2020 - 15:47 EDT

This incident affected: API, Ticket Processing Queue, and Primary DB.