DrPublish incident

Incident Report for Aptoma AS

Postmortem

Our investigations into the incident Monday revealed a certain use of DrPublish that caused the front-end to create an exponentially increasing amount of requests back to DrPublish's APIs. This in turn revealed a weakness in the error-logging mechanisms, which further contributed to the problems.

These two factors together created a load on the backend beyond what autoscaling could mitigate. This caused a major outage for some customers, and decreased service operations for others.

We have now corrected everything that played a direct role in this incident to prevent incidents from the same or similar causes in the future.

We are sorry for the problems caused by this incident.

Posted Sep 26, 2024 - 11:49 CEST

Resolved

This incident has been resolved.

We will later publish a public postmortem here once we have concluded.

Posted Sep 23, 2024 - 14:18 CEST

Update

All DrPublish users should expect DrPublish to function as normal.

We are still monitoring the situation.

Posted Sep 23, 2024 - 12:13 CEST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Sep 23, 2024 - 11:05 CEST

Identified

The issue has been identified, and we are working to resolve it. It will take some more time for all systems to fully recover.

Posted Sep 23, 2024 - 10:56 CEST

Update

We are still investigating this issue, and we expect the incident to last for some more time.

We'll post updates immediately when we have new information, and we'll update you every 30 minutes regardless of status.

Posted Sep 23, 2024 - 10:26 CEST

Investigating

We are currently investigating this issue.

Posted Sep 23, 2024 - 09:57 CEST

This incident affected: DrPublish - GUI and DrPublish - API /io.