Recent Downtime and Loss of Content

[Patrick]

Hello,

Thank you for visiting phpBBHacks.com.

It's been a crazy and frustrating 51 hours. 51 hours, in the grand scheme of things, isn't that long. But, as I was working to return this website back to normal, it seemed like an eternity. As you may know, during this time, phpBBHacks.com went through a series of issues, including the site being inaccessible due to a "suspended" message and then being back, but with lost content and read only access - meaning that you couldn't post or edit anything.

I apologize for these issues. I've been running phpBBHacks.com for nearly 10 and a half years and when something like this happens, no one is more frustrated than I am. Downtime happens, but what really bothers me is when content is lost. When people contribute something to this community and that contribution is lost due to technical issues, that eats me up.

And that is what happened. We lost approximately 5 days of contributions which, I can't say this enough, bothers me. I have a regular backup system in place that helps to limit any loss of content due to a catastrophic event like this one - and it worked. If not for our own backup system, we would have lost 10 days worth of contributions, which would have been even worse.

Due to an error by the company that previously hosted this website, our live databases were terminated while I was on the road, visiting my brother at college. Not only that, but they had no backup more recent than 10 days ago. This is inexcusable. I spent yesterday basically trying to find out whether or not that was true, holding out hope that a more recent copy existed, and failing to receive any response for many hours, which added insult to injury.

We were able to get the 10 day old website back online, but knowing that I had a more recent copy of the database backed up, I made it read only so that no further contributions would be lost when I overwrote the 10 day old copy with the 5 day old copy. Not only that, but I decided, to be fully confident moving forward, I wanted to execute an emergency server move before accepting further contributions and opening back up. This added to the delay and that is what I have spent today working on.

If you are a hack, style or download author who submitted something over the past 7 days, I would greatly appreciate if you would submit it again. If you are a member of our forums who posted within the last 7 days, and you recall what your post was about, I would really appreciate it if you would make your post again.

I apologize for the loss of content. I value all solid contributions to our community and to lose any is supremely disappointing to me.

Thank you for your time, patience and understanding.

Sincerely,

Patrick

[femu]

Hi Patrick,

these things may happen. Bad, but no guarantee, that you always have everything backed up. I use MySQLDumper for my backups, which runs via a cronjob (externaly hosted) on a dayly basis in the background. This way I have at least a nearly recent backup of the database available. So worst case 1 or 2 days lost.

Could be something you might implement too, to be a little bit more secure.

[Patrick]

Thanks Felix. I appreciate that. These things do happen, especially with very limited resources. Thankfully, losing content like this is very rare.

We have a pretty good backup system in place given those limited resources (both money and bandwidth). But, your message has me thinking. I may add an additional level to this and use something like MySQLDumper to email backups to a Google Apps account. I need to set this up a certain way, since my network has over 1.14 GB (and growing) in live databases.

Thanks again,

Patrick

[Ptirhiik]

Happy to see you back on (and thanks for your time btw).

If it may bring something to your thinkings : what we do at office to backup sensible customer servers is to have a second server on the same farm, and copy the backups one to the other on a day basis (very quick), then move to a local NAS the backup of the two servers each day (slower, but asynchronous) : this way, we are certain if a hardware failure occurs on one server, we can start back on the second very quickly, and if a problem occurs on the host, we can move to another one with the NAS backup within half a day (duration due rather to the DNS setups than the restart itself). This is something we started when I lost my former domain

[Patrick]

Thank you, Pierre.

Unfortunately, we don't have the resources to have one fully managed dedicated server, let alone two. But, that definitely sounds like a good way to go and I appreciate the thoughts.

We have a pretty good backup system for our limited resources. There is always at least 4 copies of the database on 3 different machines in 2 different physical locations. It is all set to automatically run and, generally, 2 copies can be as old as 4 days, 1 copy can be as old as a day and then 1 is the live copy. This excludes host backups.

2 of those copies require my computer to be on to update them every 2 days. I was traveling while this was happening, so they didn't execute. This may have actually been a blessing in disguise because if they would have run, they might have overwritten the backups that saved us 5 days of content, with the databases that were 10 days old.

But, as I mentioned in reply to Felix, I believe I am going to add an additional layer to it and have a copy of each database emailed to a private Google Apps email account to be stored for a short period.

Thanks again,

Patrick

[Dogs and Things]

Hi Patrick,

It's a pity you had to find out the hard way your former hosting company is bad. No matter what, they should have been more respectful with you, your data and your visitors.

You did a pretty good job, five days of data-loss isn't that bad.

Although I understand your frustration, losing one post makes me feel bad, I don't want to see me ever in your situation.

I second Felix suggestion, I use Mysqldumper too, on a cronjob.
It sends the backups it makes via ftp to an external cloudhosting. That cloudhosting is very cheap and sending backups via ftp is very quick and I feel it's more reliable than email. But that's a personal feeling.

Fingers crossed for the reliability of your new hosting!

Thanks for the hard work.

[Patrick]

Thanks Dogs and Things. I appreciate that. It's not that they are bad, but this situation was unfortunate. They have a long standing history of being good. I just needed to take steps to get the site back online in a way I was comfortable with.

I am looking into adding an additional layer that does not add any additional expense. Unfortunately, I am already pretty strapped resources wise. I have decided that I won't be doing the email thing (that was mainly be just thinking out loud ), because it appears that isn't a very secure way to do it.

Thanks again,

Patrick

[femu]

Hi Patrick,

in the Dumper you can decide, if want to email the backup or not. If not, you simply have the backups on your server and only get a mail, that it's done. I don't email mine too, as the backup is around 15 meg

[Jim_UK]

Not sure about the safety in having the backups on the server. Potentially you lose those backups if the server develops a drive problem.

However having to ftp a database of say 1/2 Gig back to the server does leave me with a feeling of dread but is what I opt for as safest.

Jim

[Dogs and Things]

Mysqldumper allows for ftp-ing the backup automatically, to another server for example. Mine is ftp-ed to a cloudhosting which costs me less than two dollars a month.