I use two Virtual Private Servers for the software that I'm working on -- one in Atlanta and one in Texas. They are hosted by VaServ, a UK-based company.
Sunday morning, my monitoring service alerted me that my site went down. I attempted to ssh in, and found both servers unreachable. I began to realize the severity of the outage when my emails to VaServ bounced, and even their home page was down.
A moment later, VaServ sent an email saying that they've been hacked via a vulnerability in HyperVM, a piece of software that they license. I went on the WebHostingTalk forum to get the scoop from other customers. It is revealed that there may be data loss. The forum is divided between extremely anxious people:
I have no clue what server I'm on, don't know if I have data loss or not, and certainly have no idea when I'll be back online.. I can't do this anymore, and to retain what sanity I still have I'm getting drunk. Let's hope I pass out and wake to at least one of my servers up
Can ping the node, but not my VPS. Been down the entire time and although I understand they're working very hard, I can't help but pass my customer's frustrations on to them. I've got pages and pages of Paypal disputes.. Down for 2 days and I may be ruined here
and others rebuking those people:
If one company going down can ruin your business and lose you customers, that's just not sound DR planning. Disasters *will* happen. Plan accordingly! Especially if you have paying customers.
The next morning, I am surprised by this Register article about the founder of LxLabs, which owned the flawed software HyperVM that Vaserv licensed:
The boss of Indian software firm LxLabs was found dead in a suspected suicide on Monday.
Reports of the death of K T Ligesh, 32, come in the wake of the exploitation of a critical vulnerability in HyperVM, a virtualization application made by LXLabs, to wipe out data on 100,000 sites hosted by the UK web hosting firm VAserv.
On the forums, most people write their condolences, and how it was not worth giving up his life over his job, no matter how dire it seems at the time.
There are also a significant fraction of the posts asking "What control panel software will I use now??" followed by lengthy discussions on the merits of alternatives to HyperVM.
A few hours later, I am shocked to see a post from someone purporting to be the hacker, addressed to the CEO of the company, Rus:
...We were in ur networks sniffing ur passwds for the past two months... Telling you this cuz we got bored of this ****, it's just too easy and monotonous so patch ur crap, if your too dumb to secure a simple web server my rate is $100/hour or one night with ur sister hauhaiahiaha.
...
[logs to show how he hacked the system, and passwords he stole]
The next few forum posts were all along the lines of "OMG." One said, "This is better than watching a tv show."
By this point, the VaServ people had worked nonstop for 48 hours to restore servers, while customers screamed unabated. Two hours later, I received an email that VaServ was just sold to a larger web hosting company, Blue Square.
On the forum, Rus wrote:
I've personally reached the end of my physical and emotitional tether. With customer attrition etc I'm not sure we could deliver the level of service that we do aim to deliver. We have worked pertty much continsouly for the last few days firefighting. Taking I'm also debating if I'm responsible for someones death I'm not in a good place currently. As such I was left with two fairly stark choices
1) Run away and hide and just say to everyone "good bye"
2) Do what is best for the customer "base" as it stands and get some big boys in behind to help get things back up and running and give people a chance.
This broke my heart a little, to think of this hardworking guy building up his company over five years. They weren't the biggest operation, but they were always responsive and competent. Gradually they were expanding to the US and working on different brands for market segments. It took only one night to undo it all.
The next morning, it's finally confirmed that while my Atlanta server is intact, my Texas server has lost all data. I had backups of data and code, but not of the server configuration. As I began the process of moving onto Amazon EC2 and re-creating the configuration:
sha-mayn: did you find out what happened?
me: one server is ok, the other has 100% data loss.
sha-mayn: oh no!
me: doesn't seem quite as bad when you consider that someone lost his company last night, and someone else lost his life
sha-mayn: that's true. we should all have more perspective about work
It's too bad you didn't lose all your data.
ReplyDeleteBecause then you could go through a poignant 21st century computer scientist version of F. Scott Fitzgerald's experience, who lost the entire manuscript to his novel This Side of Paradise, and then re-wrote the entire thing from memory, which ended up making it way better.
I am sure that you would go through a requisite period of panic, worry, loss, perspective about that guy who died, and then you would set to work re-creating all your code, and it would be better and more elegant than it was the first time, and it would be the first of many masterpieces.
Getting to live out that romantic vignette is what you have missed out on by wisely having two servers.
Hi Niniana. I have a question: aren't you working at Google anymore?
ReplyDeletehttp://forum.lxlabs.com/index.php?t=msg&goto=67882&
ReplyDeleteI received an email that VaServ was just sold to a larger web hosting company, Blue Square. thanx
ReplyDelete