Post subject: What Happened?
PostPosted: Wed May 13, 2009 8:19 am
     
I want to provide everyone with a non-technical view of what took place yesterday and last night so that some of the rumors that are already starting to spread are corrected.

During the early evening yesterday, we noticed that some alarming things were starting to happen on our servers. The first sign of an issue was that some functionality disappeared on our web server. That progressed to not being able to access the web server via FTP or SSH. Finally, the web/forum server tripped offline. Next the library/email server started to exhibit issues. We went in to reboot it, and that is when it failed entirely. The partitions on both of the servers had been removed. We shut down both servers until we can get a technical member up to our Network Operations Center (NOC), and do a comprehensive inspection of the damage.

I have seen some rumors and speculation already on the various forums in the community, and I would like to make sure that everyone knows the facts. Here are some:

Early in the evening, I sent out a bulk email to all forum members letting them know that we anticipate shutting down the Web/Forum server for a period of time. About a third of the way through the process of sending that email to all 60,000 forum members, the hacker succeeded in bringing the server down. The email was did not get sent to everyone. The reason I sent the email, based on what we knew at that time, was because that a minimum, we would need to shut the web/forum server down to inspect the unit and do repairs. At the time the email was sent, we were unaware that a hack was in progress and that our system would be taken offline entirely. There were no hoaxes involved, as some have asserted.

Tom Allensworth
AVSIM Online

Post subject: Backups
Post Posted: Wed May 13, 2009 8:42 am

     
Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers BETWEEN servers. That is, GREEN, our library server, would be backed up to PURPLE, our WEB/Forum server. That way, if one or the other failed, we would have a back up on the remaining active server. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.


Tom Allensworth
AVSIM Online

vgbaron wrote on May 13^th, 2009 at 12:21pm:


Problem at the mo', the temporary Site doesn't seem to accept our usual log-in details!

...I suppose they have been lost, as well!

Paul.