Downtime Log

2024-10-15 The corrupted config.pck files were restored from backup. Unfortunately the backup was "a few months old". This returns those mailing lists back into service.

2024-10-14 The lists.gnu.org Mailman web UI was found broken. Apparently broken since the 11th. The Mailman processes were passing email okay. But the web interface was crashed with Mailman throwing exceptions. Investigation found 4 Mailman list config.pck files were zero sized preventing Mailman from running. Those four lists were disabled. That brought the Mailman web UI back online.

A request to recover those four config.pck files from backup was submitted. Those files hold the subscriber lists. bug-wget, gnokii-users, gnowsys-bug, and listhelper-moderate were the lists affected. An old copy of the file for listhelper-moderate from 2019 happened to be there and was used to restore that list to working condition.

2024-10-11 lists.gnu.org VM was down around 7:00am with a random crash. It was noticed at 10:30am and reported as down. At 11:00am sysadmin booted it back online.

2024-10-03 High packet loss across the network. This affects everything. Packet loss is not complete failure and things keep working but very slowly. At 4am Ian powered off the misbehaving router allowing the redundant router to fail up.

2024-10-01 The community0p server rebooted for unknown reasons taking the VMs hosted there offline for 3.5 hours until Ian unlocked the VM storage devices and restarted the VMs again.

2024-09-30 The rack of servers was powered down overnight. Ian went into the datacenter the two PDUs were found with the circuit breaker switches tripped. Post mortem discussion indicated that the new servers installed recently were not load balanced on the PDUs and were overloading the power capacity. Ian plans to upgrade the PDUs ASAP.

2024-09-29 High packet loss occuring among the network. Systems were very slow for several hours. Ian powered down socket 19 powering down one of the two redundant network switches and the packet loss went away. Noting that at this time 18 network ports are being used.