Per LUMEN (vendor): We just received an update from the field team, paper cable issue. Technician working with cable splicers trying to isolate bad spot on this 1-mile cable. Extremely hopeful to repair this tonight as rain may come. They are also looking at a possibility splicing around it. This has been escalated to the field teams as well for visibility.
Posted Oct 08, 2020 - 17:21 PDT
Update
LUMEN technicians are working on repairing the faulty cable and the vendor estimate provided for completion time is two hours (information was provided at ~2:30PM PT). Ticket has been escalated to level four (vendor) and a progress report requested in one hour. We are unaware of the extent of the outage/damage to LUMEN’s infrastructure and have not found any information publicly available. We are asking for LUMEN’s top-level management information so that our IT management team can ask for answers to this outage from WesternU upper/senior management.
Posted Oct 08, 2020 - 16:00 PDT
Update
Late yesterday we were notified that the Lumen on-site technician identified the source of the issue on their end. A partial repair was started last night which required additional equipment the technician did not have on hand. Lumen support stated the repair should be finalized this morning. We will continue to monitor this closely and report back when we have some more information.
Posted Oct 08, 2020 - 08:53 PDT
Monitoring
We received an update from the vendor (LUMEN/CenturyLink) that indicated the problem is impacting 500 customers. The update was received at 3:00 PM and the estimated time of repair was two hours. As of 5:00PM, the issue has not been resolved.
Posted Oct 07, 2020 - 17:24 PDT
Identified
We have determined that the issue is external to our network and appears to be within Lumen’s (formally CenturyLink) network. We have been in constant communication with Lumen support ever since this incident was reported on Monday. We have made sure that the Lumen support team is aware of the impact this is having to not only our Lebanon campus, but also the Oliver Station medical center in Portland. After multiple interactions with the Lumen team, we have since been escalated to their tier 3 support team. However, with this still an issue in day 3 we have also reached out to our account reps to have this escalated to Lumen’s VP/Executive level team for review. Several workarounds are still currently in place as reported in previous updates to allow forwarding, 4-digit dialing and voicemail.
Posted Oct 07, 2020 - 11:57 PDT
Update
WesternU IT is continuing research the root cause of this problem but the vendor (CenturyLink) has had additional complications (which they indicate are wildfire related) that have delayed identification and resolution of this issue. Temporary workarounds are in place as reported in the previous update.
Posted Oct 06, 2020 - 14:29 PDT
Investigating
We are continuing to work internally to troubleshoot this issue while reaching out to our vendor (CenturyLink). As a temporary work around, we have the clinic’s number on forward to a Pomona number that is then internally routed back to the clinic in Portland. There is a possibility that physical hardware has failed and will require replacement. Replacement parts have been shipped from Pomona to Portland and should arrive around mid-day.
Posted Oct 05, 2020 - 17:05 PDT
Update
Internal dialing to Lebanon and Oliver Station is working. Calling 10-digit numbers into our Lebanon pool of direct dial numbers (541-259-XXXX) or Oliver Station Clinic Area Code 503 regular telephone lines is not working.
Posted Oct 05, 2020 - 12:39 PDT
Update
We can confirm this outage is also affecting the ability to make outgoing calls from the Lebanon campus. Actively working with our vendor, CenturyLink to resolve this issue.
Posted Oct 05, 2020 - 11:19 PDT
Identified
Oliver Station Clinic Main line for incoming calls is down. Ticket opened with Centurylink. Tech confirms he is unable to connect to their ONT (Optical Network Terminal box) which is a sign of trouble in their network/switching equipment.
Posted Oct 05, 2020 - 10:40 PDT
This incident affected: University Networks (Oliver Station) and Campus Services (Telecom).