Fixed

Alerts are flowing at normal rates. We believe the incident is over.

Email alert delivery has been fully restored for all users. We appreciate your patience while we investigated and resolved this issue.

At 5:56 PM EDT, we were notified that our email hosting account had been placed into a suspended state due to a suspected security breach. As a precaution, we immediately disabled all email alert pipelines and isolated the affected systems to prevent any information from being transmitted through our internal APIs while the issue was investigated.

During our investigation, we discovered that email alerts had begun experiencing widespread delivery failures approximately 30 minutes earlier. Messages were being rejected by multiple recipient domains through our email hosting provider's infrastructure.

By 6:07 PM EDT, we were able to determine that no breach had occurred. Instead, a filtering system within our provider's platform had been triggered following a large burst of alert traffic, causing legitimate weather alerts to be flagged and rejected as spam.

After contacting our hosting provider, we were informed that recent changes to their email filtering policies likely contributed to the issue and could potentially result in similar disruptions in the future.

Fortunately, we were already preparing a migration to a new email delivery provider later this month. To expedite restoration efforts, we accelerated that migration and redirected our alert delivery systems to the new provider.

Service was successfully restored once the migration was completed.

Additionally, due to the impact this incident had on the reputation of our primary domain, we have elected to isolate alert delivery traffic from our main domain going forward.

All Forecastix email alerts will now be sent from:

alerts@alerts.forecastix.xyz

This change improves the reliability and resilience of our alert delivery infrastructure while helping protect the reputation of our primary domain and services.

We apologize for the disruption and thank you for your continued support of Forecastix.

Resolved

Slack has declared that messages and webhooks are stable.

At this time, we believe the Facebook publishing issue has been resolved. Posts are now flowing normally and delivery rates have significantly improved following changes implemented by our third-party provider.

We will continue monitoring the situation closely and will reopen this incident if additional issues are observed.

Thank you for your patience while we worked through this issue.

The issue causing intermittent Error 100 responses from our third-party Facebook posting provider has been resolved. Posting performance and delivery times have returned to normal operation.

We will continue monitoring the platform closely to ensure stability. We apologize for any delays this may have caused.

Resolved

We are now back on the primary National Weather Service NWWS-OI feed.

All systems are operating normally, and full alert coverage has been restored.

Resolved

Fixed

The issue causing incorrect expiration times in GPT-rewritten Special Weather Statements has been resolved. After implementing additional mitigation, outputs have returned to normal and expiration times are now generating correctly.

We will continue to monitor the system to ensure continued stability. Thank you for your patience while we worked through this issue.

Earlier, we experienced an issue where improperly formatted watch probability data caused the alert ingest to hang, preventing alerts from being processed normally.

The corrupted string has been identified and removed, and the system has fully recovered. Weather Watch 79 has also been successfully processed and posted to affected groups.

We are implementing additional input validation and error handling to prevent unexpected formats from causing similar issues in the future.

All systems are now operating normally.

Fixed

The issue affecting group assignment for some new user signups has been corrected. New accounts should now receive the proper group creation callback during signup. We will continue to monitor the system to ensure everything remains stable.

We’ve completed a major (and expensive!) backend infrastructure upgrade. Users can now expect alerts to be posted to most platforms within about 5 seconds of issuance. WxThing Flash users can now receive alerts in as little as 2 seconds.

This upgrade significantly improves speed and reliability, helping ensure you receive critical alerts as quickly as possible.