Service Status History

Service Status History2020-03-12T17:50:24+00:00
5/01/2020

Lab on Demand

Summary of impact: From 5:30 EST to 6:00 EST, some users may have received errors resuming, launching or connecting to labs in EU North.

Root Cause: A storage service experienced degraded performance.

Resolution: The affected service was transferred to redundant hardware. Service was restored by 6:00 EST.

4/24/2020

Training Management System

Summary of impact: From 00:05 EST to 00:25 EST some users may have received errors accessing the Training Management System.

Root Cause: A cloud service was temporarily unresponsive due to an underlying resource issue.

Resolution: The affected service was recycled and performance normalized by 00:25 EST.

4/21/2020

Lab on Demand

Summary of impact: From 13:13 EST to 13:46 EST, some users experienced lab launch failures in Lab on Demand. Users already in sessions were able to resume their lab activities, but new launches may have received a timeout error.

Root Cause: A cloud service was temporarily unresponsive due to an interruption in communication. Note: A national internet service provider experienced a fiber cut which caused widespread service disruption and may have caused routing issues between major cities. (Learn more)

Resolution: The issue was resolved on April 21 at 13:46 EST. We continue to monitor the labs closely.

4/14/2020

Lab on Demand

Summary of impact: From 8:05 EST to 9:00 EST, some users experienced lab launch failures or longer than normal launch times in Lab on Demand due to timeouts.

Root Cause: An error in the deployment of a feature update caused a database query to underperform.  The query was caused by a specific dataset on a webpage used only by our technical support team.

Resolution: A hot fix was deployed at 8:25 EST and performance normalized by 9:00 EST.

4/13/2020

Lab on Demand

Summary of impact: From 10:52 EST to 11:22 EST, users experienced lab launch failures in Lab on Demand due to timeouts.

Root Cause: A temporary period of slow performance in one of our cloud databases.

Resolution: Resolved on cloud provider, root cause under investigation.

1/16/2020

Training Management System

Summary of impact: At 12:53 EST for a duration of 10 minutes our TMS experienced a major service disruption.

Root Cause: Upstream network issue, caused externally

Resolution: Upstream network partners were engaged and resolved the issue.