Email alert delivery has been fully restored for all users. We appreciate your patience while we investigated and resolved this issue.
At 5:56 PM EDT, we were notified that our email hosting account had been placed into a suspended state due to a suspected security breach. As a precaution, we immediately disabled all email alert pipelines and isolated the affected systems to prevent any information from being transmitted through our internal APIs while the issue was investigated.
During our investigation, we discovered that email alerts had begun experiencing widespread delivery failures approximately 30 minutes earlier. Messages were being rejected by multiple recipient domains through our email hosting provider's infrastructure.
By 6:07 PM EDT, we were able to determine that no breach had occurred. Instead, a filtering system within our provider's platform had been triggered following a large burst of alert traffic, causing legitimate weather alerts to be flagged and rejected as spam.
After contacting our hosting provider, we were informed that recent changes to their email filtering policies likely contributed to the issue and could potentially result in similar disruptions in the future.
Fortunately, we were already preparing a migration to a new email delivery provider later this month. To expedite restoration efforts, we accelerated that migration and redirected our alert delivery systems to the new provider.
Service was successfully restored once the migration was completed.
Additionally, due to the impact this incident had on the reputation of our primary domain, we have elected to isolate alert delivery traffic from our main domain going forward.
All Forecastix email alerts will now be sent from:
This change improves the reliability and resilience of our alert delivery infrastructure while helping protect the reputation of our primary domain and services.
We apologize for the disruption and thank you for your continued support of Forecastix.
Resolved
Email alert delivery has been fully restored for all users. We appreciate your patience while we investigated and resolved this issue.
At 5:56 PM EDT, we were notified that our email hosting account had been placed into a suspended state due to a suspected security breach. As a precaution, we immediately disabled all email alert pipelines and isolated the affected systems to prevent any information from being transmitted through our internal APIs while the issue was investigated.
During our investigation, we discovered that email alerts had begun experiencing widespread delivery failures approximately 30 minutes earlier. Messages were being rejected by multiple recipient domains through our email hosting provider's infrastructure.
By 6:07 PM EDT, we were able to determine that no breach had occurred. Instead, a filtering system within our provider's platform had been triggered following a large burst of alert traffic, causing legitimate weather alerts to be flagged and rejected as spam.
After contacting our hosting provider, we were informed that recent changes to their email filtering policies likely contributed to the issue and could potentially result in similar disruptions in the future.
Fortunately, we were already preparing a migration to a new email delivery provider later this month. To expedite restoration efforts, we accelerated that migration and redirected our alert delivery systems to the new provider.
Service was successfully restored once the migration was completed.
Additionally, due to the impact this incident had on the reputation of our primary domain, we have elected to isolate alert delivery traffic from our main domain going forward.
All Forecastix email alerts will now be sent from:
This change improves the reliability and resilience of our alert delivery infrastructure while helping protect the reputation of our primary domain and services.
We apologize for the disruption and thank you for your continued support of Forecastix.
Identified
We have identified the issue and have begun migrating to a new backend service. This migration will take place over the next several hours.
During this time, email alerts will remain temporarily disabled. All other Forecastix services continue to operate normally.
We appreciate your patience as we work to restore full functionality.
Investigating
Email alerting is temporarily unavailable while we investigate an increased reject rate from iCloud. This is necessary to protect domain reputation and prevent spam filtering. We apologize for any inconvenience. SMS, and all other platforms are unaffected.
