WesternU Status

Update - We are still monitoring this issue and still believe this is something on Microsoft's side, but we have internal reports of speeds being much better today. We will continue to monitor and collect data should this not be fully resolved.
Aug 5, 12:38 PDT

Monitoring - We are aware of reports of customers experiencing slowness and 5-10 second lockups in the full/desktop version of Outlook. Microsoft experienced a major service outage on July 20-21 and the internal belief is that these are lingering issues. At this time, we do not believe this is something we have the ability to resolve or troubleshoot as this is a hosted system but we will continue to monitor reports.
Jul 27, 11:37 PDT

University Networks Operational

eduroam Operational

Pomona Campus Operational

East Valley Clinic Operational

Rancho Mirage Operational

COMP-Northwest Operational

Oliver Station Operational

VPN - GlobalProtect Operational

CHS-Northwest Operational

Academic Services Operational

Canvas Operational

Elentra Operational

EDU Rotations Operational

Mediasite Operational

Top Hat Operational

Administrative Services Operational

Banner Operational

Cognos Operational

Raiser's Edge Operational

bswift Operational

Misc. Operational

ADP Operational

Clinical Systems Operational

EPIC Operational

Axium Operational

ImproMed Operational

Campus Services Operational

Self Service (Banweb) Operational

YOU Portal Operational

Intranet Operational

www.westernu.edu Operational

Telecom Operational

Electricity Operational

RightFax Operational

Microsoft 365 Services Degraded Performance

Active Directory Federated Services (Single Sign-On) Operational

Sharepoint Online Operational

Outlook Online Operational

Teams Operational

Misc. Degraded Performance

Zoom Operational

Zoom Meetings Operational

Zoom Video Webinars Operational

Zoom Chat Operational

Zoom Cloud Recording Operational

Third-Party Services Operational

TeamDynamix (support.westernu.edu) Operational

ExamSoft Operational

Slate (apply.westernu.edu) Operational

Qualtrics Operational

Miscellaneous Operational

Internal API Testing Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Scheduled Maintenance

Epic Scheduled Maintenance - August 2022 Aug 19, 00:00-03:30 PDT

On Friday morning, August 19, 2022, OCHIN will be performing maintenance on Epic Production and ancillaries. Downtime is scheduled for 12:10 am to 3:30 am PT so we can complete monthly maintenance tasks and apply the Epic August Service Pack.

Please review the ‘Who/What is affected?’ section below, which calls out the services that will be impacted.

Who/What is affected?
All users who access Epic Production and ancillary tools and applications listed below:
• Epic Production (PRD) will be offline
• Epic Shadow (SHD) will be online
• Electronic Prescribing – Superscripts will be offline
• Epic Link will be offline
• Haiku/Canto will be offline
• My Chart will be offline
• Care Everywhere will be offline
• Mirth Interface servers will be restarted and intermittently offline affecting Health Information Exchanges (HIE)
o All ADT, ORM, ORU, DFT, MDM, Immunizations, file transfers interfaces will be affected
o All queued messages will be sent once servers are back online
• OnBase server will be offline, and the following will be unavailable:
o Scanning and indexing
o Scanning and viewing documents in Front Desk Scanning, ROI Module Scanning, and Referral Module Scanning
o Viewing documents in patient charts
o Accessing OnBase Report Services
• Hillrom (formerly Welch Allyn) database server will be offline
• Viewing and saving dental images via MiPACs/Wisdom will be unavailable
• Clarity database ETL Processing Delays
o As a result of PRD maintenance, service pack and quarterly release Epic upgrade cycles, there will likely be delays in our Clarity ETL process as the process will begin later than usual, causing the delay in completion time. This affects all Clarity database reporting including our Clarity replication members, Business Objects reports, custom Clarity reports, Tableau data sources, and any Epic-Crystal reports or Clarity-based objects in Hyperspace.

Why this downtime?
This is routine scheduled maintenance
Posted on Aug 16, 16:53 PDT

Past Incidents

Aug 16, 2022

Banner 9 Not Loading

Resolved - This incident has been resolved.
Aug 16, 11:40 PDT

Investigating - We have some reports of customers unable to login to Banner 9 and/or it loading very slowly, We are currently investigating this issue.
Aug 16, 11:37 PDT

Aug 15, 2022

TeamDynamix (support.westernu.edu) Outage

Resolved - No reports of issues since TDX announced this was resolved early this afternoon. We are going to close out this incident.
Aug 15, 16:35 PDT

Monitoring - Teamdynamix states that services have been restored and they will continue to monitor; we will do the same internally.
Aug 15, 13:17 PDT

Identified - Teamdynamix confirmed there is an issue affecting their services. Some webpages and services are coming back up but may not be fully functional yet. We have no external reports of anything being fixed, but internally it looks like things are loading again, proceed with caution.
Aug 15, 13:11 PDT

Investigating - We are currently investigating this issue.
Aug 15, 12:43 PDT

Mediasite Search Outage

Resolved - Mediasite search was unavailable from about 9:50AM 10:40AM. The issue has now but fixed, if you are still having problems please log out of Mediasite, then log back and try again. If that does not work please contact TechSupport.
Aug 15, 10:00 PDT

Aug 14, 2022

No incidents reported.

Aug 13, 2022

Elentra Scheduled Maintenance

Completed - The scheduled maintenance has been completed.
Aug 13, 09:00 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 13, 03:00 PDT

Scheduled - Elentra has a scheduled maintenance during this time. The outage could be as little as 5 minutes or as long as 6 hours.
Aug 11, 14:45 PDT

Aug 12, 2022

No incidents reported.

Aug 11, 2022

No incidents reported.

Aug 10, 2022

Network & Wireless Maintenance (HEC & PCC)

Completed - The scheduled maintenance has been completed.
Aug 10, 21:01 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 10, 20:00 PDT

Scheduled - The network (wired & wireless) in the HEC and PCC buildings may be offline for ~15 minutes between 8:00-9:00PM tonight to patch our software on our infrastructure equipment. This will affect those in the building as well as those attempting to remote into devices located in these buildings.
Aug 10, 14:58 PDT

Elentra Down

Resolved - Elentra has resolved the issue. Access to Elentra is now available.
Aug 10, 15:15 PDT

Monitoring - A fix has been implemented by Elentra and access should be possible. Will continue to monitor and update when fully resolved.
Aug 10, 09:27 PDT

Investigating - Elentra is aware of the issue and are currently working on resolving the issue.
Aug 10, 09:23 PDT

Aug 9, 2022

No incidents reported.

Aug 8, 2022

No incidents reported.

Aug 7, 2022

No incidents reported.

Aug 6, 2022

Completed - The scheduled maintenance has been completed.
Aug 6, 08:00 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 5, 22:00 PDT

Scheduled - Banner and all related systems, including Banner 8 & 9, Self Service Banner, YOU Portal, Banner Document Manager (BDM), eProcurement, CashNet, etc., will be unavailable.
Aug 4, 16:00 PDT

TeamDynamix (support.westernu.edu) Scheduled Maintenance

Completed - The scheduled maintenance has been completed.
Aug 6, 01:00 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 5, 21:00 PDT

Scheduled - TeamDynamix will be undergoing maintenance for the 11.5 release scheduled for August 6, 2022.
Jul 28, 13:18 PDT

Aug 5, 2022

Teamdynamix Microservices Issue

Resolved - Per TDX as of early this AM: Microservices were redeployed to allow webhook processing to operate as normally expected.
Aug 5, 12:36 PDT

Identified - Teamdynamix is investigating an issue where some microservices are not working. This will not affect ticket creation but may affect some internal microservices that allow for ticket routing and email notifications.
Aug 2, 10:38 PDT

Aug 4, 2022

Banner 9 Outage

Resolved - This incident has been resolved.
Aug 4, 12:56 PDT

Identified - Outage Banner 9 is down. ETA to fix is about 30 minutes.
Aug 4, 11:57 PDT

Aug 3, 2022

Wireless Network Update / Maintenance

Completed - The scheduled maintenance has been completed.
Aug 3, 23:00 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 3, 21:00 PDT

Scheduled - Eduroam wireless network will be be getting an update to improve connection when roaming between wireless access points across campus.
Aug 3, 14:56 PDT

Aug 2, 2022

Mediasite Scheduled Maintenance

Completed - The scheduled maintenance has been completed.
Aug 2, 19:00 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 2, 17:15 PDT

Scheduled - We will be undergoing scheduled maintenance during this time.
Aug 2, 17:13 PDT

ADFS / MFA / SSO Login Issues

Resolved - This issue is technically not resolved, but the problematic server has been removed from production. Two new servers have been brought up to test and the issue has re-presented itself each time. For now, the problem is resolved in the sense that no customers are affected, but further follow-up is required with Microsoft (actively working with them) to get this resolved long term. Should this issue begin affecting customers again we will open a new incident.
Aug 2, 16:29 PDT

Update - No reports of new issues today after disabling the affected server. This problem has not been solved, but it has been temporarily removed. Services were operational for our community today. We will provide another update when available.
Jul 26, 17:20 PDT

Update - Troubleshooting continues, but root cause has not yet been determined. We have opened a case with Microsoft. For now, we have completely disabled the server causing problems. It may be necessary to briefly re-enable the server to help identify the problem.
Jul 25, 22:36 PDT

Update - We are continuing to see issues with applications and services tied to Single Sign-On (SSO), Active Directory Federated Services (ADFS), and Multi-factor Authentication (MFA). We are actively troubleshooting these issues.
Jul 25, 16:15 PDT

Update - We are still trying to determine the root cause of these issues, but we believe we have identified a single ADFS server causing the problems. This single server has been removed from production and the problems have subsided for now, however, this leaves us with no redundancy should something else go down. We are continuing to troubleshoot the issue, but you *SHOULD* NOT have login problems while this server is offline.
Jul 25, 14:53 PDT

Investigating - We are expanding this issue to include other services that use ADFS and MFA as we believe this may be a larger issue. We are seeing customers having issues with logging into a variety of applications that require MFA.
Jul 25, 12:02 PDT

Monitoring - Internal testing and customer feedback indicate that the web version is working again. No changes made our end, so this is assumed an issue on ADP's side. We will continue to monitor the situation and close out later today if no additional problems.
Jul 25, 09:29 PDT

Investigating - Customers reporting issues logging into ADP via the web. Recommend using the mobile app until resolved.
Jul 25, 07:54 PDT