Axium Issue

Incident Report for WesternU

Resolved

We are pleased to report that after extensive monitoring we can confidently say that this issue has been resolved. We believe all workstations have been updated to reach the new Axium server, but if you notice any that may have been missed, please that information to CDM IT and it will be addressed. We are now closing out this incident. Thank you.
Posted Sep 14, 2023 - 18:02 PDT

Monitoring

Maintenance has been completed and believe the issue major issue to be resolved. There is likely to be some odds and ends at the client/computer level tomorrow. Please report these issues to CDM IT for review and resolution. We will continue to monitor through the day until we deem the issue completely resolved.
Posted Sep 12, 2023 - 21:49 PDT

Identified

We have successfully load tested the Axium database on a new server. A maintenance has been scheduled from 5PM tonight until 7AM to move the production database from the old server to the new server and re-setup everything. See maintenance information in the link below.

https://status.westernu.edu/incidents/bjm47fs5kl1j
Posted Sep 12, 2023 - 14:17 PDT

Update

We have developed a game plan and put it in action yesterday. A new Axium server was created. Early this afternoon, a stress test of the new server will take place to see if this is a viable solution. If this works, we will look to move the production database over to this new server. Continuous process is being made and we will update as things progress.
Posted Sep 12, 2023 - 10:19 PDT

Investigating

We are still experiencing significant issues with Axium this morning. Right now, we are actively reviewing and discussing more drastic options, including rebuilding the server/operating system.
Posted Sep 11, 2023 - 09:07 PDT

Monitoring

The tuning that was performed on Wednesday night has resulted in significant improvements with reduced downtime and interruptions to services. The only reports of interruptions came Thursday morning around 9:30AM, but we did not receive reports of any other issues after that.

Last night our Database Administrator added additional tuning and space to aim towards complete service restoration. We are confident that we have implemented the correct solution. We will continue to monitor throughout the day.
Posted Sep 08, 2023 - 08:07 PDT

Investigating

Unfortunately, as soon as we posted that things were going in the right direction, the database started having issues again. We are investigating.
Posted Sep 07, 2023 - 09:27 PDT

Update

Changes were made to the database late last night/early this morning. Thus far things have been stable and we are cautiously optimistic that this could be a resolution. We will continue to monitor Axium throughout the day. Please report any issues to CDM IT.
Posted Sep 07, 2023 - 09:23 PDT

Update

Steady work continues on this issue. Late today our Database Administrator reports that with the assistance of Oracle, a root cause has been identified. To come up with a solution additional diagnostics and queries must be completed and sent to Oracle. A group comprised of IT leadership, our DBA, and the Director of Information Systems (CDM) will meet tomorrow morning to discuss next steps forward towards a resolution.
Posted Sep 06, 2023 - 17:07 PDT

Monitoring

Additional troubleshooting and fine tuning of the Axium database took place over the 3-day weekend. We will not know if these changes are fully successful until customers start using the system on Tuesday morning. We will closely monitor the situation during the workday.
Posted Sep 04, 2023 - 20:39 PDT

Identified

We are still experiencing issues, currently investigating.
Posted Sep 01, 2023 - 08:12 PDT

Update

Work successfully completed last night, we will continue to monitor this morning to ensure stability.
Posted Sep 01, 2023 - 07:31 PDT

Monitoring

Maintenace has been scheduled for tonight from 5-9PM to fully resolve the issue. Please see https://status.westernu.edu/incidents/x88qv2zlctkj for more details.
Posted Aug 31, 2023 - 15:34 PDT

Identified

The issue has been identified, and a resolution is in the works. Axium is currently operational after a temporary fix, a more permanent fix will be applied outside of business hours.
Posted Aug 31, 2023 - 15:15 PDT

Investigating

We are aware of a problem affecting Axium causing freezing and slowness and are actively investigating the issue
Posted Aug 31, 2023 - 14:13 PDT
This incident affected: Clinical Systems (Axium).