All Systems Operational
University Networks Operational
eduroam (wireless) ? Operational
Pomona Campus Operational
East Valley Clinic Operational
Rancho Mirage Operational
COMP-Northwest ? Operational
Oliver Station Operational
VPN - GlobalProtect Operational
CHS-Northwest ? Operational
Academic Services Operational
Canvas Operational
Elentra ? Operational
EDU Rotations Operational
Mediasite ? Operational
Top Hat Operational
Administrative Services Operational
Banner ? Operational
Cognos Operational
Raiser's Edge Operational
bswift ? Operational
Misc. Operational
ADP Operational
Clinical Systems Operational
EPIC Operational
Axium Operational
ImproMed Operational
Campus Services Operational
Self Service (Banweb) Operational
My.WesternU.edu (previously YOU Portal) ? Operational
Intranet Operational
www.westernu.edu Operational
Telecom ? Operational
Electricity Operational
RightFax Operational
Microsoft 365 Services Operational
Active Directory Federated Services (Single Sign-On) ? Operational
Sharepoint Online Operational
Outlook Online Operational
Teams Operational
Misc. Operational
Zoom Operational
Zoom Meetings Operational
Zoom Video Webinars Operational
Zoom Chat Operational
Zoom Cloud Recording Operational
Third-Party Services Operational
TeamDynamix (support.westernu.edu) Operational
ExamSoft ? Operational
Slate (apply.westernu.edu) ? Operational
Qualtrics Operational
Jamf (Apple Management) ? Operational
Miscellaneous Operational
Miscellaneous Operational
Internal API Testing ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
The December maintenance window is scheduled for the 3rd THURSDAY of the month. For additional information please visit: https://support.westernu.edu/TDClient/1848/Portal/KB/ArticleDet?ID=89703
Posted on Nov 22, 2024 - 11:09 PST
Past Incidents
Dec 8, 2024

No incidents reported today.

Dec 7, 2024

No incidents reported.

Dec 6, 2024
Completed - The scheduled maintenance has been completed.
Dec 6, 21:00 PST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 6, 20:00 PST
Scheduled - An emergency maintenance window has been scheduled to address an urgent security issue. During this time VPN (GlobalProtect) connections may drop or be briefly unavailable. VPN connections that are dropped may have to be re-established.
Dec 6, 10:57 PST
Dec 5, 2024

No incidents reported.

Dec 4, 2024
Resolved - Microsoft has Resolved their Outlook issue with the following statement:

The fix has been deployed, and we've confirmed the databases have returned to a healthy state and are functioning within acceptable thresholds. After monitoring to ensure service health was stable, we've verified impact has been remediated.

Dec 4, 08:44 PST
Monitoring - Below is the latest update from Microsoft:

"We're preparing to deploy our fix and anticipate that this process should be underway by our next scheduled communications update. We'll provide a precise remediation timeline once available."

Dec 2, 16:26 PST
Update - Below is the latest update from Microsoft:

"We've determined the error impacting certain portions of infrastructure is resulting from a table responsible for updating indexes on calls to the database being deleted and metadata not updating to reflect this table being We've determined the error impacting certain portions of infrastructure is resulting from a table responsible for updating indexes on calls to the database being deleted and metadata not updating to reflect this table being deleted. We suspect this table may have been deleted twice in two separate but concurrent transactions, leaving the metadata in a dysfunctional state. We’ve developed a fix to manually update the metadata so the replication mechanism can correct itself and are testing it to determine its effectiveness."

Dec 2, 11:14 PST
Identified - Microsoft has reported the following issues with Outlook:

Some users may be unable to send or receive email messages in Exchange Online using Outlook on the web or desktop client


*UPDATE*
We've identified an error which is occurring in the affected portions of infrastructure, and are developing a fix to address the issue while we investigate the cause of this error.

Dec 2, 08:44 PST
Dec 3, 2024
Completed - The scheduled maintenance has been completed.
Dec 3, 05:30 PST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 3, 05:00 PST
Scheduled - Elentra will be upgrading its infrastructure, which means you can look forward to enhanced performance, security, and a more reliable platform experience overall. As part of this upgrade, we will need to schedule a maintenance window of approximately one hour and 30 minutes. During this time, access to your Elentra environment will be temporarily unavailable.
Nov 20, 12:10 PST
Dec 2, 2024
Dec 1, 2024

No incidents reported.

Nov 30, 2024

No incidents reported.

Nov 29, 2024

No incidents reported.

Nov 28, 2024

No incidents reported.

Nov 27, 2024

No incidents reported.

Nov 26, 2024
Resolved - Below is the final update from Microsoft:

"After a period of extended monitoring, we’ve confirmed through customer reports and service health telemetry that the issue is resolved."

Nov 26, 13:25 PST
Monitoring - Below are the latest updates from Microsoft stating with the oldest to the latest:

11/25/2024 - 6:10PM" Our monitoring indicates that a large portion of affected users and services are seeing recovery following our mitigation efforts. We're working on addressing the lingering regions that are still seeing small impact to fully restore service availability, which we still expect to complete by Monday, November 25, 2024 at 7:00 PM PST"

11/25/2024 - 7:31PM " We’ve successfully restored functionality for all previously impacted services and users with the exception of Outlook on the web, which is showing prolonged impact for a small number of users. We’ll continue carefully monitoring the service health and focus on troubleshooting this persisting impact to fully recover for the remaining affected users. We'll provide a new timeline within the next update."

11/25/2024 - 8:32PM "Impact to core services have been restored with the exception of Outlook on the web, which we’ll continue to monitor and actively troubleshoot until full recovery.

This quick update is designed to give the latest information on this issue."

11/25/2024 - 9:04PM "We're continuing to address lingering impact to the Outlook on the web service that is still affecting some users. In parallel, we're investigating some mail queuing delays that is resulting in mail taking longer than expected to be delivered. Due to the impact of this incident, we will enter a period of extended monitoring prior to declaring this issue resolved."

11/26/2024 - 12:04AM "We’ve isolated the cause of mail queue delays and have restarted the affected infrastructure to drain stalled queues. We’ll remain in an extended monitoring phase until this draining is completed and we can consider the incident fully recovered.

This quick update is designed to give the latest information on this issue."

11/26/2024 - 1:03AM "We're still addressing the remaining impact to the Outlook on the web service that is affecting some users. We’ve applied mitigation actions to reduce the mail queues and we’re continuing the extended period of monitoring to ensure stability continues."

11/26/2024 - 2:02AM "While we continue our period of extended monitoring, the availability of the Outlook on the Web service has reached expected availability levels. We’re continuing to optimize the environment to address the remaining impact.

This quick update is designed to give the latest information on this issue."

11/26/2024 - 2:59AM " From monitoring service telemetry, most users should now experience relief. We’ve completed our optimizations and we're continuing our period of extended monitoring to ensure the availability remains stable."

11/26/2024 - 4:53AM "We’re continuing our period of monitoring service telemetry, which shows the service availability has remained healthy."

11/26/2024 - 7:58AM "We’re conducting additional optimizations to further improve service availability, which remains healthy. We're closing in on a full mitigation and we're closely monitoring telemetry to ensure this trend continues."

Nov 26, 08:14 PST
Update - Below is the latest update from Microsoft:

"The service is continuing to incrementally recover. We're forecasting that impact will be fully remediated by Monday, November 25, 2024 at 7:00 PM PST, though we're closely monitoring our progress and will provide an updated timeline should the estimate change."

Nov 25, 16:13 PST
Update - Below is the latest update from Microsoft:

"We identified a change in the environment that resulted in an influx in request retries routed through affected servers. Our optimizations, which enhanced the infrastructure's processing capabilities, continue to provide incremental relief. We're monitoring the service and continuing our work to perform any follow-up actions or opening additional workstreams needed to fully resolve the problem.

We understand the significant impact of this event to your organization, we're treating this issue with the highest priority, and we're working to provide relief as soon as possible."

Nov 25, 14:29 PST
Update - Below is the latest update from Microsoft:

"Services are showing significant signs of restoration following our infrastructure optimizations to redistribute traffic and restart affected servers, upwards of 90 percent recovery as we continue to increase processing capability. We're continuing to perform appropriate optimizations on the subset of remaining affected machines to resolve the issue for all users."

Nov 25, 13:02 PST
Update - Below is the latest update from Microsoft:

"Our mitigative actions haven't provided relief as expected, and a portion of infrastructure remains in an unhealthy state. We determined that some of the targeted server restarts did not succeed due to processing issues, which are under investigation. We’re currently focused on spreading traffic to healthy infrastructure, and we're seeing some recovery."

Nov 25, 11:22 PST
Update - Below is the latest update from Microsoft:

"We're continuing to reroute traffic to alternate infrastructure and have reinitiated targeted server restarts to ensure the fix takes effect as expected. We're monitoring to confirm the restarts proceed successfully. We don't yet have an estimated time to resolution; however, we'll provide one as soon as it becomes available"

Nov 25, 10:43 PST
Identified - Microsoft is aware of the following issues but is working to resolve them.

Here is the latest update:

Microsoft is rerouting traffic to alternate infrastructure to expedite mitigation. In parallel, they're resolving complications leading to the delays in targeted server restarts.

ISSUES:

Users may be unable to access Exchange Online using the following impacted connection methods:
- Outlook on the web
- Outlook desktop client

Users may be unable to use the following features within Microsoft Teams:
- create or update Virtual Events, including webinars and Town Halls.
- unable to access or modify their calendar in Microsoft Teams. This would include loading calendar, viewing meetings, creating/updating meetings and joining meetings.
- unable to create chat, add users and create or edited meetings.
- unable to create or modify new teams and channels.
- may be unable to update presence.
- may be unable to use the search function.
- may not see updated list of files and links failing to load within the Chat shared tab.

Users may be unable to use the search feature within SharePoint Online.

Users may experience errors running flows that utilize cloud connectors in Power Automate for Desktop.

Users may be unable to access their bookings within Microsoft Bookings.

Nov 25, 09:06 PST
Nov 25, 2024
Nov 24, 2024

No incidents reported.