System outage - FireServiceRota and Brandweerrooster are down
Incident Report for FireServiceRota
Postmortem

Post-Mortem of Outage of Thursday 26th of August

What happened?

At 00:25 we received an alert from our automated liveness checks that our systems were down.
The team immediately mobilized to establish comms and update our StatusPage. An investigation
was initiated. Symtoms were similar to those seen during the previous outages caused by
instabilities from our hosting provider. A few minutes afterwards, we received confirmation
from our host provider that they were experiencing issues at their London Data Center.

Brandweerrooster was offline for 0h57m on 26/8/2022.
FireServiceRota was offline for 0h57m on 26/8/2022.

For more technical information about the issue, please refer to the postmortem from
our provider.

What we will do to prevent this?

Since the last outages caused by the host provider, we've initiated migrating actions
to have redundancy accross multiple host providers. We aim to have the new setup
ready before the end of the year.

For more information about the actions we are taking, please refer the this link.

Posted Sep 01, 2022 - 17:55 UTC

Resolved
This incident has been resolved - Postmortem will be available soon.

More updates regarding Linode - https://status.linode.com/incidents/qcb6ksmsmz40
Posted Aug 26, 2022 - 00:17 UTC
Update
FireServiceRota and Brandweerrooster are back online, we are continuing to monitor. We are waiting for Linode to update the status of the fix.
Posted Aug 25, 2022 - 23:27 UTC
Monitoring
We have successfully switched over FireServiceRota to the Frankfurt data centre. FireServiceRota is now operational and we will continue monitoring this issue.
Posted Aug 25, 2022 - 23:12 UTC
Update
We are continuing to work on a fix for this issue.
Posted Aug 25, 2022 - 22:49 UTC
Update
We are implementing a switch over to the Frankfurt data centre for FireServiceRota. Linode Data Centres in London and Frankfurt are unreachable at the moment.
Posted Aug 25, 2022 - 22:45 UTC
Identified
The issue has been identified, our data centre in London is experiencing a connectivity issue.

https://status.linode.com/incidents/qcb6ksmsmz40?u=d7g1g3j5q0z6
Posted Aug 25, 2022 - 22:38 UTC
Investigating
We are currently investigating the issue.
Posted Aug 25, 2022 - 22:29 UTC
This incident affected: Datacenters (Linode EU-West (London), Linode EU-Central (Frankfurt)), FireServiceRota (FireServiceRota Primary Systems, FireServiceRota Standby Systems, FireServiceRota Home Page), and Brandweerrooster (Netherlands) (Brandweerrooster Primary Systems, Brandweerrooster Standby Systems, Brandweerrooster Home Page).