Write-up published
Resolved
Fixed
Monitoring
We have corrected an issue that affected Advisory 3 and caused some GPT-rewritten posts to include technical forecast information, such as Z time and wind speeds in knots, instead of the normal public-facing wording.
The issue was caused by a rare timing condition where the technical Forecast Advisory arrived before the Public Advisory. The system mistakenly determined that the full advisory package was ready and sent it for processing, causing GPT Rewrite to receive the technical product instead of the public-facing product.
Safeguards were already in place for incomplete and incorrect advisory data, but this specific timing edge case did not occur during our multi-day testing and was not accounted for. A secondary GPT safeguard also should have rejected the unexpected product and fallen back to the standard non-GPT format, but did not trigger as expected.
Both issues have now been corrected. We have added additional checks to ensure the complete advisory package has arrived before processing begins, along with stronger validation before information is provided to GPT Rewrite. If those checks fail, the system will fall back to the standard non-GPT format.
Testing confirms the fixes are working as expected. We will closely monitor the 2:00 AM intermediate advisory to verify everything continues to process correctly with live advisory data.
Identified
We have identified the cause as a race condition where the TCM arrived before the TCP Public Advisory. The advisory package was incorrectly marked as ready and sent to the processor before the full package had arrived, causing GPT Rewrite to receive the Forecast Advisory instead of the Public Advisory.
We already had safeguards in place to prevent incomplete or incorrect advisory data from being processed. However, this particular timing edge case was not accounted for, as it did not occur during our multi-day testing prior to release.
Additionally, GPT Rewrite should have recognized that it received the wrong product and fallen back to the standard non-GPT format. That secondary safeguard did not trigger as expected.
We are correcting both issues and adding additional validation throughout the pipeline. The system will now triple-check that the complete advisory package has arrived before marking it as ready for processing, and GPT Rewrite will independently verify that it received the correct product before generating a post.
Investigating
We are investigating an issue with Advisory 3 where the GPT rewrite was incorrectly provided the technical forecast discussion instead of the public advisory discussion. This resulted in some posts containing technical units and terminology, such as Z time and knots (kt).