Downtime Log

2024-10-03 High packet loss across the network. This affects everything. Packet loss is not complete failure and things keep working but very slowly. At 4am Ian powered off the misbehaving router allowing the redundant router to fail up.

2024-10-01 The community0p server rebooted for unknown reasons taking the VMs hosted there offline for 3.5 hours until Ian unlocked the VM storage devices and restarted the VMs again.

2024-09-30 The rack of servers was powered down overnight. Ian went into the datacenter the two PDUs were found with the circuit breaker switches tripped. Post mortem discussion indicated that the new servers installed recently were not load balanced on the PDUs and were overloading the power capacity. Ian plans to upgrade the PDUs ASAP.

2024-09-29 High packet loss occuring among the network. Systems were very slow for several hours. Ian powered down socket 19 powering down one of the two redundant network switches and the packet loss went away. Noting that at this time 18 network ports are being used.