Update - We are still monitoring this issue and still believe this is something on Microsoft's side, but we have internal reports of speeds being much better today. We will continue to monitor and collect data should this not be fully resolved.
Aug 5, 12:38 PDT
Monitoring - We are aware of reports of customers experiencing slowness and 5-10 second lockups in the full/desktop version of Outlook. Microsoft experienced a major service outage on July 20-21 and the internal belief is that these are lingering issues. At this time, we do not believe this is something we have the ability to resolve or troubleshoot as this is a hosted system but we will continue to monitor reports.
Jul 27, 11:37 PDT
University Networks Operational
eduroam ? Operational
Pomona Campus Operational
East Valley Clinic Operational
Rancho Mirage Operational
COMP-Northwest ? Operational
Oliver Station Operational
VPN - GlobalProtect Operational
CHS-Northwest ? Operational
Academic Services Operational
Canvas Operational
Elentra ? Operational
EDU Rotations Operational
Mediasite ? Operational
Top Hat Operational
Administrative Services Operational
Banner ? Operational
Cognos Operational
Raiser's Edge Operational
bswift ? Operational
Misc. Operational
ADP Operational
Clinical Systems Operational
EPIC Operational
Axium Operational
ImproMed Operational
Campus Services Operational
Self Service (Banweb) Operational
YOU Portal Operational
Intranet Operational
www.westernu.edu Operational
Telecom ? Operational
Electricity Operational
RightFax Operational
Microsoft 365 Services Degraded Performance
Active Directory Federated Services (Single Sign-On) ? Operational
Sharepoint Online Operational
Outlook Online Operational
Teams Operational
Misc. Degraded Performance
Zoom Operational
Zoom Meetings Operational
Zoom Video Webinars Operational
Zoom Chat Operational
Zoom Cloud Recording Operational
Third-Party Services Operational
TeamDynamix (support.westernu.edu) Operational
ExamSoft Operational
Slate (apply.westernu.edu) ? Operational
Qualtrics Operational
Miscellaneous Operational
Miscellaneous Operational
Internal API Testing ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
On Friday morning, August 19, 2022, OCHIN will be performing maintenance on Epic Production and ancillaries. Downtime is scheduled for 12:10 am to 3:30 am PT so we can complete monthly maintenance tasks and apply the Epic August Service Pack.

Please review the ‘Who/What is affected?’ section below, which calls out the services that will be impacted.

Who/What is affected?
All users who access Epic Production and ancillary tools and applications listed below:
• Epic Production (PRD) will be offline
• Epic Shadow (SHD) will be online
• Electronic Prescribing – Superscripts will be offline
• Epic Link will be offline
• Haiku/Canto will be offline
• My Chart will be offline
• Care Everywhere will be offline
• Mirth Interface servers will be restarted and intermittently offline affecting Health Information Exchanges (HIE)
o All ADT, ORM, ORU, DFT, MDM, Immunizations, file transfers interfaces will be affected
o All queued messages will be sent once servers are back online
• OnBase server will be offline, and the following will be unavailable:
o Scanning and indexing
o Scanning and viewing documents in Front Desk Scanning, ROI Module Scanning, and Referral Module Scanning
o Viewing documents in patient charts
o Accessing OnBase Report Services
• Hillrom (formerly Welch Allyn) database server will be offline
• Viewing and saving dental images via MiPACs/Wisdom will be unavailable
• Clarity database ETL Processing Delays
o As a result of PRD maintenance, service pack and quarterly release Epic upgrade cycles, there will likely be delays in our Clarity ETL process as the process will begin later than usual, causing the delay in completion time. This affects all Clarity database reporting including our Clarity replication members, Business Objects reports, custom Clarity reports, Tableau data sources, and any Epic-Crystal reports or Clarity-based objects in Hyperspace.

Why this downtime?
This is routine scheduled maintenance

Posted on Aug 16, 16:53 PDT
Past Incidents
Aug 16, 2022
Resolved - This incident has been resolved.
Aug 16, 11:40 PDT
Investigating - We have some reports of customers unable to login to Banner 9 and/or it loading very slowly, We are currently investigating this issue.
Aug 16, 11:37 PDT
Aug 15, 2022
Resolved - No reports of issues since TDX announced this was resolved early this afternoon. We are going to close out this incident.
Aug 15, 16:35 PDT
Monitoring - Teamdynamix states that services have been restored and they will continue to monitor; we will do the same internally.
Aug 15, 13:17 PDT
Identified - Teamdynamix confirmed there is an issue affecting their services. Some webpages and services are coming back up but may not be fully functional yet. We have no external reports of anything being fixed, but internally it looks like things are loading again, proceed with caution.
Aug 15, 13:11 PDT
Investigating - We are currently investigating this issue.
Aug 15, 12:43 PDT
Resolved - Mediasite search was unavailable from about 9:50AM 10:40AM. The issue has now but fixed, if you are still having problems please log out of Mediasite, then log back and try again. If that does not work please contact TechSupport.
Aug 15, 10:00 PDT
Aug 14, 2022

No incidents reported.

Aug 13, 2022
Completed - The scheduled maintenance has been completed.
Aug 13, 09:00 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 13, 03:00 PDT
Scheduled - Elentra has a scheduled maintenance during this time. The outage could be as little as 5 minutes or as long as 6 hours.
Aug 11, 14:45 PDT
Aug 12, 2022

No incidents reported.

Aug 11, 2022

No incidents reported.

Aug 10, 2022
Completed - The scheduled maintenance has been completed.
Aug 10, 21:01 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 10, 20:00 PDT
Scheduled - The network (wired & wireless) in the HEC and PCC buildings may be offline for ~15 minutes between 8:00-9:00PM tonight to patch our software on our infrastructure equipment. This will affect those in the building as well as those attempting to remote into devices located in these buildings.
Aug 10, 14:58 PDT
Resolved - Elentra has resolved the issue. Access to Elentra is now available.
Aug 10, 15:15 PDT
Monitoring - A fix has been implemented by Elentra and access should be possible. Will continue to monitor and update when fully resolved.
Aug 10, 09:27 PDT
Investigating - Elentra is aware of the issue and are currently working on resolving the issue.
Aug 10, 09:23 PDT
Aug 9, 2022

No incidents reported.

Aug 8, 2022

No incidents reported.

Aug 7, 2022

No incidents reported.

Aug 6, 2022
Completed - The scheduled maintenance has been completed.
Aug 6, 08:00 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 5, 22:00 PDT
Scheduled - Banner and all related systems, including Banner 8 & 9, Self Service Banner, YOU Portal, Banner Document Manager (BDM), eProcurement, CashNet, etc., will be unavailable.
Aug 4, 16:00 PDT
Completed - The scheduled maintenance has been completed.
Aug 6, 01:00 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 5, 21:00 PDT
Scheduled - TeamDynamix will be undergoing maintenance for the 11.5 release scheduled for August 6, 2022.
Jul 28, 13:18 PDT
Aug 5, 2022
Resolved - Per TDX as of early this AM: Microservices were redeployed to allow webhook processing to operate as normally expected.
Aug 5, 12:36 PDT
Identified - Teamdynamix is investigating an issue where some microservices are not working. This will not affect ticket creation but may affect some internal microservices that allow for ticket routing and email notifications.
Aug 2, 10:38 PDT
Aug 4, 2022
Resolved - This incident has been resolved.
Aug 4, 12:56 PDT
Identified - Outage Banner 9 is down. ETA to fix is about 30 minutes.
Aug 4, 11:57 PDT
Aug 3, 2022
Completed - The scheduled maintenance has been completed.
Aug 3, 23:00 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 3, 21:00 PDT
Scheduled - Eduroam wireless network will be be getting an update to improve connection when roaming between wireless access points across campus.
Aug 3, 14:56 PDT
Aug 2, 2022
Completed - The scheduled maintenance has been completed.
Aug 2, 19:00 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 2, 17:15 PDT
Scheduled - We will be undergoing scheduled maintenance during this time.
Aug 2, 17:13 PDT
Resolved - This issue is technically not resolved, but the problematic server has been removed from production. Two new servers have been brought up to test and the issue has re-presented itself each time. For now, the problem is resolved in the sense that no customers are affected, but further follow-up is required with Microsoft (actively working with them) to get this resolved long term. Should this issue begin affecting customers again we will open a new incident.
Aug 2, 16:29 PDT
Update - No reports of new issues today after disabling the affected server. This problem has not been solved, but it has been temporarily removed. Services were operational for our community today. We will provide another update when available.
Jul 26, 17:20 PDT
Update - Troubleshooting continues, but root cause has not yet been determined. We have opened a case with Microsoft. For now, we have completely disabled the server causing problems. It may be necessary to briefly re-enable the server to help identify the problem.
Jul 25, 22:36 PDT
Update - We are continuing to see issues with applications and services tied to Single Sign-On (SSO), Active Directory Federated Services (ADFS), and Multi-factor Authentication (MFA). We are actively troubleshooting these issues.
Jul 25, 16:15 PDT
Update - We are still trying to determine the root cause of these issues, but we believe we have identified a single ADFS server causing the problems. This single server has been removed from production and the problems have subsided for now, however, this leaves us with no redundancy should something else go down. We are continuing to troubleshoot the issue, but you *SHOULD* NOT have login problems while this server is offline.
Jul 25, 14:53 PDT
Investigating - We are expanding this issue to include other services that use ADFS and MFA as we believe this may be a larger issue. We are seeing customers having issues with logging into a variety of applications that require MFA.
Jul 25, 12:02 PDT
Monitoring - Internal testing and customer feedback indicate that the web version is working again. No changes made our end, so this is assumed an issue on ADP's side. We will continue to monitor the situation and close out later today if no additional problems.
Jul 25, 09:29 PDT
Investigating - Customers reporting issues logging into ADP via the web. Recommend using the mobile app until resolved.
Jul 25, 07:54 PDT