Server upgrades, remotely fail and what to do
by Mountain Computers Inc., Publication Date: Monday, July 12, 2021
View Count: 180, Keywords: Servers, Updates, Upgrades, Failure, Recovery, Hashtags: #Servers #Updates #Upgrades #Failure #Recovery
Well, we had a few remote server update upgrades over the weekend, and 5 of them went well, and 1 did not. All of them are over 25 miles away, so the one that is not responding is 50 miles away (actually 33 miles away per the customer since he read our blog and asked, hey, is that me?) I replied yes, and I have two more business clients within 10 minutes around you that this update worked fine, alas.
What happened next was hell for 3-4 hours. I helped him recover everything and get back to business yet it was at best, like not wanting to go to your dentist for a broken crown. I have to admit, backups saved me, him and the company. He had backups, and I had backups, and he was pissed. I did not do anything. All I did was what was necessary. In the end, he and I pulled together, we reconfigured the network, server, and all the other resources as quickly as possible. I was like, shit, this should not have happened, yet it did, and he was like, what else can I do, just fix it, free of charge, per se under warranty and move forward. Well, it ended up being 4 hours of work to reconfigure the network, restore the data, reconfigure a workstation to be a server, and move forward. Alas, I did not want to be paid, because in a way, it was a fault of the update, and a way a fault of the Microsoft emergency software update. To the end, my client paid me though I would have gladly walked away with it fixed and only come back when requested. It was a tough situation for me because I don't create work and I love my clients to be running seamlessly forever.
On the other hand, and forethought - this post was added below with consider of why this might have happened, and it was actually due to a number of waterfall issues. So, remember, even with your best efforts, notes, and documentation, there will always be something you might miss.
Our onsite technical assistance is being deployed and they said it looks like a weak CMOS battery and that caused the reboot failure and now the system does not know its configuration or name. That requires a little nudge from the local support to reset and get it back online. It will wake back up. We do have backups in place.
The best solutions and salvation was a series of good backups we used.
more to come...
if you found this article helpful, consider contributing $10, 20 an Andrew Jackson or so..to the author. more authors coming soon
FYI we use paypal or patreon, patreon has 3x the transaction fees, so we don't, not yet.
© 2022 myBlog™ v1.1 All rights reserved. We count views as reads, so let's not over think it.