Resolved -
This incident has been resolved.
Our investigation determined that one of our Internet Service Providers (ISPs) was experiencing significant performance degradation. Because the connection remained partially operational rather than failing completely, our automated failover did not redirect traffic to the secondary ISP as designed. We have identified an improvement to our failover configuration that will better detect this type of degraded service and help ensure traffic automatically transitions to the backup ISP if a similar situation occurs in the future.
After confirming the issue was ISP-related, we manually redirected all Internet traffic to our secondary ISP earlier this morning. Since making that change, network performance has remained stable with no additional issues reported.
The ISP has since resolved the underlying issue and provided the following information:
Root Cause: A fiber fault in Albany, Oregon, impacted client services.
Resolution: The damaged fiber was repaired, restoring services to a stable operating state.
We will restore our normal redundant Internet configuration and re-enable automated ISP failover after hours. No disruption to network services is expected during this change.
Jul 1, 15:22 PDT
Monitoring -
We have identified an issue with one of our Internet Service Providers (ISP) which was causing network slowness and intermittent disconnections for the Oregon sites. We have manually switched over to another ISP which has alleviated the issues.
Jul 1, 10:18 PDT
Update -
We are actively troubleshooting network slowness and intermittent connectivity issues affecting our Oregon sites. Our team is working to determine whether the root cause is related to one of our Internet Service Providers or a network hardware failure. Additional updates as more information becomes available.
Jul 1, 10:00 PDT
Investigating -
We are experiencing network issues in Oregon and are actively investigating.
Jul 1, 09:13 PDT