IN-355 - Network Issues

Incident Report for Checkit

Resolved

A network configuration change resulted in a network routing issue within our infrastructure. This caused periods of degraded connectivity between application services and backend databases.

As a result, some customers experienced intermittent monitoring issues and a temporary interruption to data collection services. During the same period, the CAM+ website was also unavailable for approximately three hours because application services were unable to establish connections to the required databases.

Core alarm processing remained operational during the incident. However, the impact to monitoring functionality was initially limited to a subset of customers and was not immediately identified.

As network connectivity was progressively restored between 17:12 BST and 20:45 BST, queued alarm traffic was released and a significant volume of alarms required processing. This temporary increase in traffic placed additional demand on the alarm receiving infrastructure, resulting in delays to alarm processing and intermittent failures for remote alarm calls on WARP devices.

Alarm processing returned to normal operation at 22:08 BST.

Customer Impact

Customers may have experienced the following during the incident:

CAM+ website: Unavailable for approximately three hours.
Data collection: Intermittent or failed data collection from DC2 between approximately 14:50 BST and 20:45 BST.
CAM+ Backfill: Degraded performance while outstanding data was being recovered.
Alarm processing: Intermittent processing delays and delayed alarm notifications between approximately 15:23 BST and 22:08 BST.
WARP remote alarm calls: Some calls may have failed or experienced delays while the alarm processing infrastructure was handling the backlog.
Services Not Impacted

The following services remained operational throughout the incident:

Automated backfill services.
Core alarm processing capabilities.
Recovery and Remediation

Once network connectivity was restored, recovery activities were initiated to process outstanding data and ensure services returned to normal operation.

A backfill process was initiated overnight and remains in progress to restore any outstanding data affected during the period of degraded connectivity. Engineering teams are continuing to monitor the recovery process and validate the integrity and completeness of the recovered data.

Next Steps

Our engineering teams will continue to monitor the platform closely while recovery and validation activities are completed. We will also review the incident in detail to identify further preventative actions and improvements to our change management and monitoring processes.
Posted Aug 28, 2026 - 11:17 UTC

Identified

All customer-facing services have now been restored and are operating normally.

We are continuing work behind the scenes to fully restore network connectivity and resilience. A backfill process was initiated overnight and remains in progress to recover any data affected during the incident.

We’re sorry for the disruption caused and will continue to monitor service stability closely.
Posted Aug 26, 2026 - 09:57 UTC

Update

Services have been restored through temporary measures while we continue to investigate the underlying network issue.

Alarm processing and call notifications are operating normally, automated backfill for the outage period has been initiated, and data logging has been restored.

The CAM+ Backfill app, accessed through the CAM+ website, continues to experience degraded performance.

Our Networking Team remains on site and is continuing to investigate the underlying issue. We’re sorry for the inconvenience caused and will provide a further update by 10:30 BST on August 2026.
Posted Aug 25, 2026 - 19:04 UTC

Update

We’re continuing to work towards a resolution, and we’re sorry for the inconvenience this is causing. Our Networking Team remains on site, and all previously reported services are still affected.

Please continue to monitor sensor readings and alarms manually via the WARP display panel, as alarm notifications are not currently being delivered.

We’ll share a further update as soon as we have meaningful progress.
Posted Aug 25, 2026 - 17:04 UTC

Update

We are continuing to investigate this issue.
Posted Aug 25, 2026 - 15:05 UTC

Investigating

Due to an overrun on planned maintenance, we are experiencing network issues.

Issue Description
The network issues are causing degraded performance for several services.

Impact
• Access to the CAM+ website may be affected.
• Logging of CAM+ data may be affected. Backfill will be initiated as soon as is practical.
• Alarm and alert notifications will not be sent via call, email or SMS.

Next Steps
• Our Networking Team are on-site and currently attending to the issue.

What You Should Do
• Continue to monitor sensor readings and alarms using the WARP display panel.
• Be aware that alarm notifications will not be delivered until the issue has been resolved.

Next Update
We will aim to issue an update before 18:00 BST today.
Posted Aug 25, 2026 - 15:04 UTC
This incident affected: CAM+.