Earlier, we experienced an issue where improperly formatted watch probability data caused the alert ingest to hang, preventing alerts from being processed normally.
The corrupted string has been identified and removed, and the system has fully recovered. Weather Watch 79 has also been successfully processed and posted to affected groups.
We are implementing additional input validation and error handling to prevent unexpected formats from causing similar issues in the future.
All systems are now operating normally.
Resolved
Earlier, we experienced an issue where improperly formatted watch probability data caused the alert ingest to hang, preventing alerts from being processed normally.
The corrupted string has been identified and removed, and the system has fully recovered. Weather Watch 79 has also been successfully processed and posted to affected groups.
We are implementing additional input validation and error handling to prevent unexpected formats from causing similar issues in the future.
All systems are now operating normally.
Monitoring
The system is now fully recovering after a corrupted string entered the alert ingest pipeline and caused processing failures across both processors.
The corrupted data has been identified and removed, and Weather Watch 79 has now been successfully processed and posted to all affected groups.
We believe this issue was triggered by a settings change earlier today to ingest watch probability data. This introduced a format that was outside what the processors currently support, which caused the ingest pipeline to fail.
To prevent similar issues in the future, we are implementing additional input validation and error handling so unexpected formats cannot cause system-wide processing interruptions.
Short-term alerts are fully operational, and long-term processing is continuing to recover as queued alerts are processed.
We apologize for the disruption and appreciate your patience as we worked to restore full functionality.
Identified
We are currently working on a fix for a critical alert ingest error caused by a corrupted string that impacted both processors.
Short-term alerts have now been restored and are operational. However, Weather Watch 79 remains unable to post, and some long-term processing may still be affected while we continue implementing a full resolution.
Our team is actively working to restore full functionality as quickly as possible. Additional updates will be provided as progress continues.
We appreciate your patience and apologize for the disruption.
Investigating
We are currently investigating a critical issue where a corrupted string entered the alert ingest pipeline, causing errors across both processors. As a result, Weather Watch 79 is currently unable to post, and processing of all other alerts has been halted.
Our team is actively working to resolve this issue as quickly as possible. Updates will be provided as soon as more information becomes available.
We apologize for the disruption and appreciate your patience while we work to restore full functionality.