tag:status.westernu.edu,2005:/historyWesternU Status - Incident History2024-03-29T03:00:13-07:00WesternUtag:status.westernu.edu,2005:Incident/201798562024-03-16T09:00:56-07:002024-03-16T09:00:56-07:00Elentra Scheduled Maintenance<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>05:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>14:13</var> PST</small><br><strong>Scheduled</strong> - From Elentra:<br /><br />Reason for Maintenance: During this time, we will be performing essential updates to our database software as mandated by Google. These updates are crucial to continue to deliver secure and reliable services.<br /> <br />Impact on Service: As a result of this maintenance activity, Elentra will be temporarily unavailable. We understand the inconvenience this may cause and assure you that we will work diligently to minimize the downtime.<br />Action Required: We kindly ask you to plan your work accordingly and refrain from accessing the platform during the specified maintenance window.<br /> <br />We Are Here to Help: If you have any questions or concerns regarding this scheduled maintenance, please do not hesitate to reach out to our support team at support@elentra.com. We appreciate your understanding and patience as we strive to improve our services.</p>tag:status.westernu.edu,2005:Incident/202378402024-03-16T00:00:57-07:002024-03-16T00:00:58-07:00Banner & Related Systems Extended Maintenance - March 2024<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>20:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>14:54</var> PDT</small><br><strong>Scheduled</strong> - Banner and all related systems, including Banner 8 & 9, Self Service Banner, YOU Portal, Banner Document Manager (BDM), eProcurement, CashNet, etc., will be unavailable.</p>tag:status.westernu.edu,2005:Incident/202378212024-03-15T20:01:04-07:002024-03-15T20:01:05-07:00Monthly IT Patch Maintenance - March 2024<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>20:01</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>18:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>14:51</var> PDT</small><br><strong>Scheduled</strong> - The 3rd Friday of each month is the designated IT maintenance window. For additional information please visit: https://support.westernu.edu/TDClient/1848/Portal/KB/ArticleDet?ID=89703</p>tag:status.westernu.edu,2005:Incident/201976322024-03-15T03:30:56-07:002024-03-15T03:30:56-07:00Epic Scheduled Maintenance March 2024<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>03:30</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>15:49</var> PST</small><br><strong>Scheduled</strong> - On Friday morning, March 15, 2024, OCHIN will be performing maintenance on Epic Production and ancillaries. Downtime is scheduled for 12:10 am to 3:30 am PT so we can complete monthly maintenance tasks and apply March EP/SP Pack items. <br />Please review the ‘Who/What is affected?’ section below, which calls out the services that will be impacted. <br /> <br />Who/What is affected?<br />All users who access Epic Production and ancillary tools and applications listed below: <br />• Epic Production (PRD) will be offline <br />• Epic Shadow (SHD) will be online <br />• Electronic Prescribing – Superscripts will be offline <br />• Epic Link will be offline <br />• Haiku/Canto will be offline <br />• My Chart will be offline <br />• Care Everywhere will be offline <br />• Mirth Interface servers will be restarted and intermittently offline affecting Health Information Exchanges (HIE) <br />• All ADT, ORM, ORU, DFT, MDM, Immunizations, file transfers interfaces will be affected <br />• All queued messages will be sent once servers are back online <br />• OnBase server will be offline and the following will be unavailable: <br />• Scanning and indexing <br />• Scanning and viewing documents in Front Desk Scanning, ROI Module Scanning, and Referral Module Scanning <br />• Viewing documents in patient charts <br />• Accessing OnBase Report Services <br />• Hillrom (formerly Welch Allyn) database server will be offline <br />• Viewing and saving dental images via MiPACs/Wisdom will be unavailable <br />• Clarity database ETL Processing Delays <br />• As a result of PRD maintenance, service pack and quarterly release Epic upgrade cycles, there will likely be delays in our Clarity ETL process as the process will begin later than usual, causing the delay in completion time. This affects all Clarity database reporting including our Clarity replication members, Business Objects reports, custom Clarity reports, Tableau data sources, and any Epic-Crystal reports or Clarity-based objects in Hyperspace. <br /><br />Why this downtime?<br />This is routine scheduled maintenance.</p>tag:status.westernu.edu,2005:Incident/202171362024-03-12T09:37:37-07:002024-03-12T09:37:37-07:00External Phone Calls Issue - Rancho Mirage<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:37</var> PDT</small><br><strong>Resolved</strong> - This issue has now been resolved. Last weekend we experienced a (near) internet outage with the circuit between our Pomona and Lebanon campuses. This outage happened after hours, we received notice that our primary circuit went down and we manually cut over to our backup internet circuit. No customers were impacted. However, this back circuit caused a phone issue with external calling as posted in the incident (link below). We were able to fix the issue in the incident below for the main campus, but issues lingered at Rancho Mirage. Once we determined the primary internet circuit was stable again, we cut back over (last night). Returning to the primary circuit resolved the Rancho Mirage phone issues. We have opened a case with Cisco for them to review and help us determine why this change caused in hopes that we can prevent next time.<br /><br />https://status.westernu.edu/incidents/tvf05mnt0y68</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>10:07</var> PDT</small><br><strong>Investigating</strong> - We are actively investigating an issue with the phone system at the clinic in Rancho Mirage clinic. If customers call into the clinic, WesternU employees are unable hear them speaking, but external callers are able to hear WesternU employees. This is similar to the issue experienced last week ( https://status.westernu.edu/incidents/tvf05mnt0y68 ), but limited to Rancho Mirage.</p>tag:status.westernu.edu,2005:Incident/202178162024-03-11T14:57:59-07:002024-03-11T14:57:59-07:00Possible Network Outage in Anderson Tower<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>14:57</var> PDT</small><br><strong>Resolved</strong> - We have identified the issue and will take steps to prevent future outages with the knowledge with gathered.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:42</var> PDT</small><br><strong>Update</strong> - We believe network is back up, but we are still investigating the cause.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:36</var> PDT</small><br><strong>Investigating</strong> - There may be a network outage currently affecting Anderson Tower. We have several confirmed reports of no network.</p>tag:status.westernu.edu,2005:Incident/201870432024-03-07T16:31:52-08:002024-03-07T16:31:52-08:00External Phone Call Issue<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>16:31</var> PST</small><br><strong>Resolved</strong> - Confirmed that external phone calls are working normally.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>11:33</var> PST</small><br><strong>Monitoring</strong> - A fix has been applied to address the external phone call issue. <br /><br />The issue will be monitored for the remainder of the day. <br /><br />If no issues arise, the issue will be Resolved.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>10:05</var> PST</small><br><strong>Investigating</strong> - WesternU is currently experiencing issues with external phone calls with no audio from the external individual. This is affecting both outgoing and incoming external phone calls. <br /><br />Internal phone calls are working normally.</p>tag:status.westernu.edu,2005:Incident/201675312024-03-20T17:28:40-07:002024-03-20T17:28:40-07:00Faxing with EPIC<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>17:28</var> PDT</small><br><strong>Update</strong> - Earlier this week WesternU IT met with Epic (OCHIN) and RightFax (OpenText) technicians. All parties are aware of the problem that is happening (not showing print jobs in the RightFax directory), but we are unable to produce any error messages or find anything in the event viewer that would point to the cause of this problem. We also cannot be certain that the configuration script that was provided is working correctly either. A technician that is familiar with Epic and RightFax integration has been identified. We are trying to loop them into these ongoing meetings to help find a resolution.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>15:49</var> PDT</small><br><strong>Update</strong> - We activated the trial license from the vendor and unfortunately this did not resolve the faxing with Epic issue. In fact, it caused other issues outside of Epic with RightFax in other locations on campus. We have since reverted back to the original RightFax license and are working with the RightFax vendor to come up with next steps.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:45</var> PDT</small><br><strong>Update</strong> - We have received a trial license for faxing module from RightFax. We are now working with the vendor for Epic, OCHIN, to get the module configured.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>13:10</var> PST</small><br><strong>Update</strong> - This issue stems from a migration from our legacy phone system (Avaya) to our new phone system (Cisco). After we retired the last of the legacy equipment, faxing though Epic stopped working. The vendor believes this is due to a licensing issue, however, a license was not required on the legacy system. We powered back on the legacy equipment and unfortunately faxing through Epic is still not working and would require extensive work to revert. We are working with the vendor to get a trial license for this feature and to test if this will solve the issue.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>07:36</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.westernu.edu,2005:Incident/201390822024-03-02T07:00:56-08:002024-03-02T07:00:56-08:00Banner & Related Systems Emergency Maintenance<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>07:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>22:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>16:17</var> PST</small><br><strong>Scheduled</strong> - Banner and all related systems, including Banner 8 & 9, Self Service Banner, MY Portal, Banner Document Manager (BDM), eProcurement, CashNet, etc., will be unavailable.</p>tag:status.westernu.edu,2005:Incident/201383992024-03-01T16:23:44-08:002024-03-01T16:23:44-08:00Banner Login Issue<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>16:23</var> PST</small><br><strong>Resolved</strong> - An emergency Banner Maintenance has been scheduled to permanently resolve the Banner issues.<br /><br />This Incident will be Resolved but the Banner Maintenance will occur starting tonight, 3/1 @10PM through Saturday 3/2 @7AM.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>14:57</var> PST</small><br><strong>Monitoring</strong> - A temporary fix has been applied but we have discovered an issue that likely will need to be addressed after hours during an emergency maintenance, more details to come.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>14:46</var> PST</small><br><strong>Update</strong> - This error is also affecting Banner 9, Self-Service 9 and Self-Service 8.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>14:29</var> PST</small><br><strong>Investigating</strong> - Some employees are reporting issues signing into Banner if not already SSO'd in. The error message provided says, "An error occurred. Contact your administrator for more information."<br /><br />We are actively investigating this issue.</p>tag:status.westernu.edu,2005:Incident/200827582024-02-29T08:49:44-08:002024-02-29T08:49:44-08:00My.WesternU.edu<p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>08:49</var> PST</small><br><strong>Resolved</strong> - My.WesternU.edu is not fully functional. All back-end issues resolved.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:24</var> PST</small><br><strong>Update</strong> - Currently working with the Vendor to resolve the last remaining issue for My.WesternU.edu.<br /><br />Some students may experience issues with accessing their Student Course Card at My.WesternU.edu.<br /><br />WesternU IT does not have an estimated time of resolution but will update as soon as possible.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>16:49</var> PST</small><br><strong>Monitoring</strong> - My.WesternU.edu is still experiencing back-end issues that are currently being investigated. <br /><br />This issue has been moved to Monitoring until the back-end issues are resolved.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>14:29</var> PST</small><br><strong>Identified</strong> - Some users may receive the following error when trying to access My.WesternU.edu "a temporary server problem. Some cards and information may not be visible at this time."<br /><br />IT is currently on resolving the issue but no ETA of resolution.</p>tag:status.westernu.edu,2005:Incident/200805232024-02-26T16:37:35-08:002024-02-26T16:37:35-08:00CASHnet and Elixir<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>16:37</var> PST</small><br><strong>Resolved</strong> - CASHNet and Elixir services confirmed restored with no additional issues.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>10:17</var> PST</small><br><strong>Monitoring</strong> - CASHNet and Elixir services have been restored but IT will continue to monitor the issue throughout the day.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>09:13</var> PST</small><br><strong>Update</strong> - Elixir is currently down and the issue is being investigated. No current ETA for resolution. <br /><br />CASHNet should be in working order shortly.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>09:06</var> PST</small><br><strong>Identified</strong> - CASHNet is currently down but a fix has been implemented to and services should be up shortly.</p>tag:status.westernu.edu,2005:Incident/200668932024-02-24T13:42:56-08:002024-02-24T15:40:33-08:00Misc. Outage<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>13:42</var> PST</small><br><strong>Resolved</strong> - Successfully recovered to redundant hardware, there should have been no impact to customers (on Saturday). There WAS an outage on Friday evening related to this same hardware failure. There should be no customer impact at this time.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>12:37</var> PST</small><br><strong>Investigating</strong> - We have experienced hardware failure on some of our IT Infrastructure and some services and applications may be down currently. More info will be provided when available.</p>tag:status.westernu.edu,2005:Incident/200325752024-02-24T02:00:56-08:002024-02-24T02:00:56-08:00Mediasite Video Cloud Update - February 2024<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>02:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>22:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>13:00</var> PST</small><br><strong>Scheduled</strong> - Mediasite will be updating Mediasite Video Cloud to the latest release during their scheduled maintenance window on starting on February 23, at 10:01 PM and completing on February 24, at 2:00 AM PST. Extended downtime is not expected. However, WesternU's Mediasite may be unavailable for brief periods during this time.</p>tag:status.westernu.edu,2005:Incident/200592222024-02-23T13:49:18-08:002024-02-23T13:49:18-08:00eTranscript<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>13:49</var> PST</small><br><strong>Resolved</strong> - eTranscript services have been restored.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>09:38</var> PST</small><br><strong>Investigating</strong> - An issue was reported with eTransript, which is a service used to provide transcripts to students/alumni and used in conjunction with the Office of the Registrar. The issue seems to be software related and we will be working with the vendor to find a resolution.</p>tag:status.westernu.edu,2005:Incident/200395682024-02-21T16:39:17-08:002024-02-21T16:39:17-08:00my.westernu.edu Issue<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>16:39</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>08:44</var> PST</small><br><strong>Monitoring</strong> - This was a vendor issue that has now been resolved. We will continue to monitor the situation, but access should be restored now.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>08:12</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.westernu.edu,2005:Incident/200008952024-02-16T20:00:57-08:002024-02-16T20:00:57-08:00Monthly IT Patch Maintenance - February 2024<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>20:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>18:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>08:00</var> PST</small><br><strong>Scheduled</strong> - The 3rd Friday of each month is the designated IT maintenance window. For additional information please visit: https://support.westernu.edu/TDClient/1848/Portal/KB/ArticleDet?ID=89703</p>tag:status.westernu.edu,2005:Incident/199699182024-02-16T03:30:57-08:002024-02-16T03:30:57-08:00Epic Scheduled Maintenance - February 2024<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>03:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>00:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>13:27</var> PST</small><br><strong>Scheduled</strong> - On Friday morning, February 16, 2024, OCHIN will be performing maintenance on Epic Production and ancillaries. Downtime is scheduled for 12:10 am to 3:30 am PT so we can complete monthly maintenance tasks and apply February Service Pack items. <br /><br />Please review the ‘Who/What is affected?’ section below, which calls out the services that will be impacted. <br /> <br />Who/What is affected?<br />All users who access Epic Production and ancillary tools and applications listed below: <br />• Epic Production (PRD) will be offline <br />• Epic Shadow (SHD) will be online <br />• Electronic Prescribing – Superscripts will be offline <br />• Epic Link will be offline <br />• Haiku/Canto will be offline <br />• My Chart will be offline <br />• Care Everywhere will be offline <br />• Mirth Interface servers will be restarted and intermittently offline affecting Health Information Exchanges (HIE) <br />o All ADT, ORM, ORU, DFT, MDM, Immunizations, file transfers interfaces will be affected <br />o All queued messages will be sent once servers are back online <br />• OnBase server will be offline and the following will be unavailable: <br />o Scanning and indexing <br />o Scanning and viewing documents in Front Desk Scanning, ROI Module Scanning, and Referral Module Scanning <br />o Viewing documents in patient charts <br />o Accessing OnBase Report Services <br />• Hillrom (formerly Welch Allyn) database server will be offline <br />• Viewing and saving dental images via MiPACs/Wisdom will be unavailable <br />• Clarity database ETL Processing Delays <br />o As a result of PRD maintenance, service pack and quarterly release Epic upgrade cycles, there will likely be delays in our Clarity ETL process as the process will begin later than usual, causing the delay in completion time. This affects all Clarity database reporting including our Clarity replication members, Business Objects reports, custom Clarity reports, Tableau data sources, and any Epic-Crystal reports or Clarity-based objects in Hyperspace. <br /><br />Why this downtime?<br />This is routine scheduled maintenance.</p>tag:status.westernu.edu,2005:Incident/199231612024-02-06T17:16:44-08:002024-02-06T17:16:44-08:00Banner Self-Service Down<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>17:16</var> PST</small><br><strong>Resolved</strong> - Confirmed access to Banner Self-Service (Banweb.WesternU.edu) and Banner links within My.WesternU.edu are stable and working.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>10:28</var> PST</small><br><strong>Monitoring</strong> - Banner Self-Service and Banner links at My.WesternU.edu are now working and accessible.<br /><br />We are monitoring the situation.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>08:40</var> PST</small><br><strong>Investigating</strong> - Banner Self-Service (Banweb.WesternU.edu) and Banner links within My.WesternU.edu are not working.<br /><br />We are currently investigating the issue.</p>tag:status.westernu.edu,2005:Incident/198947822024-02-02T22:30:42-08:002024-02-02T22:30:42-08:00Banner & Related Systems Emergency Maintenance<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>22:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>22:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>08:58</var> PST</small><br><strong>Scheduled</strong> - Banner and all related systems, including Banner 8 & 9, Self Service Banner, YOU Portal, Banner Document Manager (BDM), eProcurement, CashNet, etc., will be unavailable.</p>tag:status.westernu.edu,2005:Incident/198875062024-02-02T17:15:02-08:002024-02-02T17:15:02-08:00Goodnotes 6 Licensing Issue<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>17:15</var> PST</small><br><strong>Resolved</strong> - The Goodnotes version 6 license has been applied to COMP / CPM iPads.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>10:00</var> PST</small><br><strong>Monitoring</strong> - IT has applied the license key to Goodnotes version 6 and that will be pushed to the iPad apps.<br /><br />With the application of the license, the iPads will require that the App store be signed into with an Apple ID. This should be a personal Apple ID and not associated with your WesternU email.<br /><br />The Goodnotes app may need to be restarted to apply the license.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>11:56</var> PST</small><br><strong>Identified</strong> - Goodnotes has updated their application to version 6, which requires a license key. We are currently in communication with Goodnotes to acquire the key and distribute it to the COMP/CPM iPads.</p>tag:status.westernu.edu,2005:Incident/198596742024-01-29T10:08:06-08:002024-01-29T10:08:28-08:00EPIC - Printing Issue<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>10:08</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>09:13</var> PST</small><br><strong>Investigating</strong> - We are currently aware of an issue preventing printing on EPIC and are currently investigating.</p>tag:status.westernu.edu,2005:Incident/198338022024-01-26T22:00:56-08:002024-01-26T22:00:56-08:00Print Server Upgrade and Migration - Pomona Campus<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>22:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>21:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>16:02</var> PST</small><br><strong>Scheduled</strong> - Starting at 9:00 pm, printing on the WesternU Pomona campus will be unavailable while the print server is updated and migrated. The upgrade and migration will be completed by 10:00 pm and printing will be available.</p>tag:status.westernu.edu,2005:Incident/198397172024-01-26T20:07:47-08:002024-01-26T20:07:47-08:00Microsoft Teams - Various Issues<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>20:07</var> PST</small><br><strong>Resolved</strong> - All issues have been resolved. See details below. <br /><br />Per Microsoft: <br /><br />Everything is up and running.<br /><br />Final status: After extended monitoring and various optimizations and mitigation efforts, we’ve confirmed that our Microsoft Teams service and features have been restored or returned to optimal health.<br /><br />Start time: Friday, January 26, 2024, at 2:55 PM UTC<br /><br />End time: Saturday, January 27, 2024, at 1:30 AM UTC<br /><br />This is the final update for the event.<br /><br />Details via https://portal.office.com/servicestatus</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>13:14</var> PST</small><br><strong>Update</strong> - From Microsoft...<br /><br />Current status: Our failover operation did not provide the immediate relief intended for end users in North and South America regions. However, we’re monitoring telemetry closely as we continue to optimize traffic patterns and apply configuration changes intended to reduce customer impact as quickly as possible. We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>10:00</var> PST</small><br><strong>Identified</strong> - Microsoft is reporting various issues with Microsoft Teams. Internal testing shows the problems are extremely noticeable on New Teams, but is affecting classic Teams too. <br /><br />From Microsoft:<br /><br />Title: Some users may experience multiple issues with their Microsoft Teams<br /><br />User impact: Users may experience multiple issues with their Microsoft Teams.<br /><br />More info: Affected scenarios include, but aren't limited to:<br />- Users performing a cold boot may not able to log into teams and will see an "oops" page<br />- Users logging in or unlocking their devices after some time may see missing messages<br />- Users may fail to load messages in channels and chats<br />- Users are unable to view or download their media (images, videos, audio, call recordings, code snippets)<br />- Some messages may experience delays being sent<br />- Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online<br />- Users may be unable to load previous Copilot history, or new history is not written<br />- Bots may be unable to download attachments<br />- Sending and receiving read receipt notifications may be delayed<br /><br />Current status: We’ve identified that a portion of the Microsoft Teams service is experiencing a networking issue. We’re performing a failover to an alternate infrastructure to remediate impact and estimate this should take approximately an hour to complete.<br /><br />Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.<br /><br />Start time: Friday, January 26, 2024 at 6:55 AM PST<br /><br />Next update by: Friday, January 26, 2024 at 11:30 AM PST</p>tag:status.westernu.edu,2005:Incident/198333972024-01-26T03:31:07-08:002024-01-26T03:31:07-08:00Epic Unplanned Maintenance - Friday, 1/26<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>03:31</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>00:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>14:44</var> PST</small><br><strong>Scheduled</strong> - On Friday morning, January 26, 2024, OCHIN will be performing maintenance on the Epic Production environment and ancillaries. Downtime is scheduled for 12:10 am to 3:30 am PT so they can update their Operational Database (ODB) hardware to bolster the resilience and reliability of their system to support network growth. <br /><br />Who/What is affected?<br />All users who access Epic Production and the ancillary tools and applications listed below: <br />• Epic Production (PRD) will be offline <br />• Epic Shadow (SHD) will be online <br />• Electronic Prescribing – Superscripts will be offline <br />• Epic Link will be offline <br />• Haiku/Canto will be offline <br />• My Chart will be offline <br />• Care Everywhere will be offline <br />• Mirth Interface servers will be restarted and intermittently offline affecting Health Information Exchanges (HIE) <br />o All ADT, ORM, ORU, DFT, MDM, Immunizations, file transfers interfaces will be affected <br />o All queued messages will be sent once servers are back online <br />• OnBase server will be offline and the following will be unavailable: <br />o Scanning and indexing <br />o Scanning and viewing documents in Front Desk Scanning, ROI Module Scanning, and Referral Module Scanning <br />o Viewing documents in patient charts <br />o Accessing OnBase Report Services <br />• Hillrom (formerly Welch Allyn) database server will be offline <br />• Viewing and saving dental images via MiPACs/Wisdom will be unavailable <br />• Clarity database ETL Processing Delays <br />o As a result of PRD maintenance, service pack and quarterly release Epic upgrade cycles, there will likely be delays in our Clarity ETL process as the process will begin later than usual, causing the delay in completion time. This affects all Clarity database reporting including our Clarity replication members, Business Objects reports, custom Clarity reports, Tableau data sources, and any Epic-Crystal reports or Clarity-based objects in Hyperspace. <br /><br />Why this downtime?<br />The ODB hardware upgrade is needed to expand the processing capabilities as OCHIN grows, and to replace hardware that is coming to end of life.</p>