WesternU Status

All Systems Operational

University Networks Operational

eduroam (wireless) Operational

Pomona Campus Operational

East Valley Clinic Operational

Rancho Mirage Operational

COMP-Northwest Operational

Oliver Station Operational

VPN - GlobalProtect Operational

CHS-Northwest Operational

Academic Services Operational

Canvas Operational

Elentra Operational

EDU Rotations Operational

Mediasite Operational

Top Hat Operational

Administrative Services Operational

Banner Operational

Cognos Operational

Raiser's Edge Operational

bswift Operational

Misc. Operational

ADP Operational

Clinical Systems Operational

EPIC Operational

Axium Operational

ImproMed Operational

Campus Services Operational

Self Service (Banweb) Operational

My.WesternU.edu (previously YOU Portal) Operational

Intranet Operational

www.westernu.edu Operational

Telecom Operational

Electricity Operational

RightFax Operational

Microsoft 365 Services Operational

Active Directory Federated Services (Single Sign-On) Operational

Sharepoint Online Operational

Outlook Online Operational

Teams Operational

Misc. Operational

Zoom Operational

Zoom Meetings Operational

Zoom Video Webinars Operational

Zoom Chat Operational

Zoom Cloud Recording Operational

Third-Party Services Operational

TeamDynamix (support.westernu.edu) Operational

ExamSoft Operational

Slate (apply.westernu.edu) Operational

Qualtrics Operational

Jamf (Apple Management) Operational

Miscellaneous Operational

Internal API Testing Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Scheduled Maintenance

Monthly IT Patch Maintenance - December 2024 Dec 19, 2024 18:00-22:00 PST

The December maintenance window is scheduled for the 3rd THURSDAY of the month. For additional information please visit: https://support.westernu.edu/TDClient/1848/Portal/KB/ArticleDet?ID=89703
Posted on Nov 22, 2024 - 11:09 PST

Past Incidents

Dec 8, 2024

No incidents reported today.

Dec 7, 2024

No incidents reported.

Dec 6, 2024

VPN (GlobalProtect) System Maintenance

Completed - The scheduled maintenance has been completed.
Dec 6, 21:00 PST

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 6, 20:00 PST

Scheduled - An emergency maintenance window has been scheduled to address an urgent security issue. During this time VPN (GlobalProtect) connections may drop or be briefly unavailable. VPN connections that are dropped may have to be re-established.
Dec 6, 10:57 PST

Dec 5, 2024

No incidents reported.

Dec 4, 2024

Outlook Issue

Resolved - Microsoft has Resolved their Outlook issue with the following statement:

The fix has been deployed, and we've confirmed the databases have returned to a healthy state and are functioning within acceptable thresholds. After monitoring to ensure service health was stable, we've verified impact has been remediated.
Dec 4, 08:44 PST

Monitoring - Below is the latest update from Microsoft:

"We're preparing to deploy our fix and anticipate that this process should be underway by our next scheduled communications update. We'll provide a precise remediation timeline once available."
Dec 2, 16:26 PST

Update - Below is the latest update from Microsoft:

"We've determined the error impacting certain portions of infrastructure is resulting from a table responsible for updating indexes on calls to the database being deleted and metadata not updating to reflect this table being We've determined the error impacting certain portions of infrastructure is resulting from a table responsible for updating indexes on calls to the database being deleted and metadata not updating to reflect this table being deleted. We suspect this table may have been deleted twice in two separate but concurrent transactions, leaving the metadata in a dysfunctional state. We’ve developed a fix to manually update the metadata so the replication mechanism can correct itself and are testing it to determine its effectiveness."
Dec 2, 11:14 PST

Identified - Microsoft has reported the following issues with Outlook:

Some users may be unable to send or receive email messages in Exchange Online using Outlook on the web or desktop client

*UPDATE*
We've identified an error which is occurring in the affected portions of infrastructure, and are developing a fix to address the issue while we investigate the cause of this error.
Dec 2, 08:44 PST

Dec 3, 2024

Elentra Infrastructure Upgrade

Completed - The scheduled maintenance has been completed.
Dec 3, 05:30 PST

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 3, 05:00 PST

Scheduled - Elentra will be upgrading its infrastructure, which means you can look forward to enhanced performance, security, and a more reliable platform experience overall. As part of this upgrade, we will need to schedule a maintenance window of approximately one hour and 30 minutes. During this time, access to your Elentra environment will be temporarily unavailable.
Nov 20, 12:10 PST

Dec 2, 2024

Dec 1, 2024

No incidents reported.

Nov 30, 2024

No incidents reported.

Nov 29, 2024

No incidents reported.

Nov 28, 2024

No incidents reported.

Nov 27, 2024

No incidents reported.

Nov 26, 2024

Microsoft 365 services and features Issues

Resolved - Below is the final update from Microsoft:

"After a period of extended monitoring, we’ve confirmed through customer reports and service health telemetry that the issue is resolved."
Nov 26, 13:25 PST

Monitoring - Below are the latest updates from Microsoft stating with the oldest to the latest:

11/25/2024 - 6:10PM" Our monitoring indicates that a large portion of affected users and services are seeing recovery following our mitigation efforts. We're working on addressing the lingering regions that are still seeing small impact to fully restore service availability, which we still expect to complete by Monday, November 25, 2024 at 7:00 PM PST"

11/25/2024 - 7:31PM " We’ve successfully restored functionality for all previously impacted services and users with the exception of Outlook on the web, which is showing prolonged impact for a small number of users. We’ll continue carefully monitoring the service health and focus on troubleshooting this persisting impact to fully recover for the remaining affected users. We'll provide a new timeline within the next update."

11/25/2024 - 8:32PM "Impact to core services have been restored with the exception of Outlook on the web, which we’ll continue to monitor and actively troubleshoot until full recovery.

This quick update is designed to give the latest information on this issue."

11/25/2024 - 9:04PM "We're continuing to address lingering impact to the Outlook on the web service that is still affecting some users. In parallel, we're investigating some mail queuing delays that is resulting in mail taking longer than expected to be delivered. Due to the impact of this incident, we will enter a period of extended monitoring prior to declaring this issue resolved."

11/26/2024 - 12:04AM "We’ve isolated the cause of mail queue delays and have restarted the affected infrastructure to drain stalled queues. We’ll remain in an extended monitoring phase until this draining is completed and we can consider the incident fully recovered.

This quick update is designed to give the latest information on this issue."

11/26/2024 - 1:03AM "We're still addressing the remaining impact to the Outlook on the web service that is affecting some users. We’ve applied mitigation actions to reduce the mail queues and we’re continuing the extended period of monitoring to ensure stability continues."

11/26/2024 - 2:02AM "While we continue our period of extended monitoring, the availability of the Outlook on the Web service has reached expected availability levels. We’re continuing to optimize the environment to address the remaining impact.

This quick update is designed to give the latest information on this issue."

11/26/2024 - 2:59AM " From monitoring service telemetry, most users should now experience relief. We’ve completed our optimizations and we're continuing our period of extended monitoring to ensure the availability remains stable."

11/26/2024 - 4:53AM "We’re continuing our period of monitoring service telemetry, which shows the service availability has remained healthy."

11/26/2024 - 7:58AM "We’re conducting additional optimizations to further improve service availability, which remains healthy. We're closing in on a full mitigation and we're closely monitoring telemetry to ensure this trend continues."
Nov 26, 08:14 PST

Update - Below is the latest update from Microsoft:

"The service is continuing to incrementally recover. We're forecasting that impact will be fully remediated by Monday, November 25, 2024 at 7:00 PM PST, though we're closely monitoring our progress and will provide an updated timeline should the estimate change."
Nov 25, 16:13 PST

Update - Below is the latest update from Microsoft:

"We identified a change in the environment that resulted in an influx in request retries routed through affected servers. Our optimizations, which enhanced the infrastructure's processing capabilities, continue to provide incremental relief. We're monitoring the service and continuing our work to perform any follow-up actions or opening additional workstreams needed to fully resolve the problem.

We understand the significant impact of this event to your organization, we're treating this issue with the highest priority, and we're working to provide relief as soon as possible."
Nov 25, 14:29 PST

Update - Below is the latest update from Microsoft:

"Services are showing significant signs of restoration following our infrastructure optimizations to redistribute traffic and restart affected servers, upwards of 90 percent recovery as we continue to increase processing capability. We're continuing to perform appropriate optimizations on the subset of remaining affected machines to resolve the issue for all users."
Nov 25, 13:02 PST

Update - Below is the latest update from Microsoft:

"Our mitigative actions haven't provided relief as expected, and a portion of infrastructure remains in an unhealthy state. We determined that some of the targeted server restarts did not succeed due to processing issues, which are under investigation. We’re currently focused on spreading traffic to healthy infrastructure, and we're seeing some recovery."
Nov 25, 11:22 PST

Update - Below is the latest update from Microsoft:

"We're continuing to reroute traffic to alternate infrastructure and have reinitiated targeted server restarts to ensure the fix takes effect as expected. We're monitoring to confirm the restarts proceed successfully. We don't yet have an estimated time to resolution; however, we'll provide one as soon as it becomes available"
Nov 25, 10:43 PST

Identified - Microsoft is aware of the following issues but is working to resolve them.

Here is the latest update:

Microsoft is rerouting traffic to alternate infrastructure to expedite mitigation. In parallel, they're resolving complications leading to the delays in targeted server restarts.

ISSUES:

Users may be unable to access Exchange Online using the following impacted connection methods:
- Outlook on the web
- Outlook desktop client

Users may be unable to use the following features within Microsoft Teams:
- create or update Virtual Events, including webinars and Town Halls.
- unable to access or modify their calendar in Microsoft Teams. This would include loading calendar, viewing meetings, creating/updating meetings and joining meetings.
- unable to create chat, add users and create or edited meetings.
- unable to create or modify new teams and channels.
- may be unable to update presence.
- may be unable to use the search function.
- may not see updated list of files and links failing to load within the Chat shared tab.

Users may be unable to use the search feature within SharePoint Online.

Users may experience errors running flows that utilize cloud connectors in Power Automate for Desktop.

Users may be unable to access their bookings within Microsoft Bookings.
Nov 25, 09:06 PST

Nov 25, 2024

Nov 24, 2024

No incidents reported.