Thanks!
[MENTION=52905]darjr[/MENTION] deserves most of the thanks.
I'm not prone to depression, but this week I saw my livelihood on the verge of dying, and stumbled (again) across the sheer petty nastiness of those who enjoyed and wanted that. It was an unpleasant experience. But we're back!
About £10,000 of code has been lost, and it's going to take a lot of work and money to replace all that; months, if not years.
I did nearly give up. But stubbornness prevented me! So here we are... again. Still here!
I was going to ask, if some kind of security audit could be performed. Basically, get a third party to assess firewalls, etc. Maybe not a fancy shop, but a friend of EN World who is versant in such.
I'm not volunteering anything, just that from my own experience of running IT, the original guy who set up our firewalls may have been certified in the firewall product he chose, but that doesn't mean he did everything the best way, or didn't have some accidental holes, or even chose what the industry considers a decent product.
Hypothetically, you guys need to make sure your firewall is locked down tight. If you don't have control of that at the network infrastructure level, then at the server level.
If it took days to restore the server, recovery plans may need to be tightened up. Nowadays, everybody is running on Virtual machines. That makes it possible to create snapshots, or at least backup the 2 files that represent the server instance. Retaining copies of those once a week would mean you can QUICKLY pop those files back into place, to get the server up and running.
these things won't guarantee safety, but they help make things easier. There's no reason* you shouldn't have been able to snap an older copy of the server image back into place. We'd have lost a few posts, but you'd be back in position, perhaps taking time to secure whatever brought the site down the first time, before bringing it online.
*well, except for not being configured and ready for it
I have no clue what actually happened. I've been in the position of having the IP for my mail server spoofed by an external agency and being black listed by SpamHaus. That means, the bad guy was outside my network and impossible for us to repair because there was nothing infected or invaded in my network. The best we could do was change IP addresses.
My sympathies, and congratulations on bringing the system back up.
While the wound is still fresh, review what happened and check and improve your security and recovery process. it will help reduce the damage that you suffered, not just in downtime.