Database Logs

[BSF]

Well, it would be even better to combine logging with more frequent backups, yes. In fact, more frequent backups are just a good idea in general, or at least checkpoint snapshots so that the log only needs to continue from the checkpoint.

Well sure it would. It would also be great to backup transactional logs throughout the day. Say, every 15 minutes to tape. That way, if you lose a drive, you can still restore from tape. It would be even better to ship those logs to your offsite continuity server. That way, if you lose a system, you can bring your standby system online. And if we wanted to be hardcore, we would also have a clustered system with an active/passive host, along with the offsite standby. All of these would have updates on the transactional logs and those logs would be backed up to media constantly throughout the day.

But to keep it in perspective, this is still just a web site for a niche hobby. Mind you, I love our hobby and I would love to see a highly redundant setup for the site. But is it reasonable? What are the costs associated with all that? Both in terms of hardware and software? What about people costs? Even with a volunteer effort, it still takes a lot of time and energy to bulldog all of that.

This stuff isn't easy to setup and manage without a good budget. And for all the fundraisers EN World has done, and all the money that has been generated, it still wouldn't come close to paying for an optimal setup.

EN World does have a pretty decent setup for a shoestring budget. If we want more backups in a timely manner, that is also possible. But the best way to do that is to run a solid offline backup on a regular basis. They keywords being "offline" and "regular". That does mean a regular time where the boards are down. As well, the way to keep those backups useful is to download them to a machine, kind of like Michael Morris' archive that he uploaded to get us back to this point. How long did it take for him to upload those? Something like 16 hours? So say EN World does a weekly offline backup. Morrus and company would just need to decide where that 16 hour window should fall.

The bottleneck will really be the network connection between EN World and the person executing the backup. The payload is the database for EN World. More bandwidth and/or a smaller database would reduce those times.

The whole picture is not easy to manage and I do not envy Morrus for the decisions he has to make. EN World is a global community and part of the value is the vast archive of information posted here. Downtime and data trimming (mass thread deletion) will have a negative impact on the community. But will it have more impact than another database corruption/massive downtime/massive loss of data?

Good question! We haven't been invited to help with that discussion. While I am happy to participate, I will defer to the Admins as to whether they would like me to be involved.

 

[Thanee]

Don't really think that this would be very useful... only if it was known exactly, what transaction(s) caused the corruption (if it wasn't something else). But that is probably not so easy to determine on such a huge database.

Bye
Thanee

 

[Rystil Arden]

Yay! There are logs after all! This makes me feel better about all the money I have to pay to take these courses, since before I started, I knew the better part of nothing about databases, and now I know enough to say something that might work.

 

[Elephant]

From what I've heard, the problem was something along these lines: The logs were not being flushed, and they kept growing in size until the disk holding the logs was full. At that point, corruption occurred.

Coupled with inconsistent backup schedules, you get the BOOM! that we all agonized over.