Technical thread for Server Discussion (merged)

[BSF]

I am sure Adlon is looking at a multitude of options and is trying to compute the best that EN World can handle for a 2-3 year plan.

I had wondered about multiple servers as well. Dedicating one box to DB and another to everything else might be a significant performance upgrade. I suppose the question in that case would be one of server requirements when you split the load. Would it be possible to get two slightly lower-end servers for the same cost as one higher end?

As I said, I am sure everyone involed is doing a good job. I am not accustomed to building servers specifically to provide performance toa DB driven website. My experience with databases is usually on the back end of the enterprise. So I am accustomed to processors with large cache and plenty of disk spindles in RAID. Because of that, it seems a little odd to look at a dual disk system, though I am very happy that you are going to use mirroring to avoid a critical crash.

 

[Jupp]

Heh. By downgrading the CPU (even a single opteron will do) you can afford a SCSI-controller, say Tekram. And the 36GB SCSI-drives aren't that expansive. SCSI-RAID, on the other hand...

It's not that the FX53 is a bad decision, it is after all the fastest processor right now. But top-of-the-line hardware is always rediciously expansive, and if you ask me FX53 and SATA-drives sounds more of a Doom III-machine than a webserver.

Yup, choosing the 2nd or 3rd generation CPUs like the Opteron 248 or the 246 (in a dual setup) you are probably better left off than when you would choose an FX53 (which I see more as an enthusiast gamer choice). And AMD has the big advantage of a superior memory management compared to the Xeon processors. But I would really go the dual cpu route since you can control the workload pretty smoothly between cpus when it comes to things like databases and other server applications. With that new server you could go and install the old server as a pure webserver, add fiber or Gigabit NICs to both of them and make a nice internal link for database requests from/to the web server which will have the link to the outside.

 

[spectre72]

I have spent quite a few years developing and maintaining reporting/OLAP systems in both Oracle and SQL Server.

It sounds like new hardware is a definite need.

From my experiences I have found better performance when I have been able to remove the database from the application server. Sometimes this is not possible due to limitations in money or the application but when possible it really seems to boost performance.

My next question is on the disks...

Are you planning on OS mirroring or Hardware mirroring?

I am assuming Hardware, but I also know the limitations of funding.

I have never found OS mirroring to be very efficient in the Windows world.

Again, a bad assumption since you could be running any number of different OS's.

On a DB server I have found that disk subsystems are usually the cause of most of the grief you will encounter. Many times you will see bad performance and in the end you will trace it back to the disk subsystem even though that is not where you would have thought it was.

The other killer for performance that I work with all the time is sub-optimal design. SQL that causes iterative full table scans, indexes that don't work as expected, ...etc. This is a killer and one that will cause the best hardware to come to seem incredibly slow. Happens to me all the time.

I am also sure that everyone who is working on this project is putting 500% into it so don't take my comments as anything other than comments. I have quite a few years building and maintaining DB servers in the hundreds of GB size (on Windows ) so I just wanted to throw some things out for you to think about.

Good luck, and may your downtime be minimal....

 

[Simon Collins]

Well, it seems that there are already people here with good technical knowledge.

I'm not a techie myself, but I'll reiterate the offer to run any solution past the technical manager in the IT company where I work (the biggest indigenous IT company in Ireland - which, admittedly, is fairly small in US terms, but hey!).

I'll add my vote to the dual processors and hardware RAID argument whilst I'm writing, too.

Cheers

Simon Collins

 

[Jarrod]

Memory, memory, memory.

But you knew that

It's been mentioned before, but have you thought about multiple systems? A database backend, a dynamic page server, and a proxy cache may be cheaper than a single monster box.

 

[IronWolf]

First, let it be said you guys are doing a great job! It is much easier to sit outside and offer ideas and thoughts than be the one that has to make the final decision and put it all together.

But, I have to agree with Psionicist. Why not go dual server class CPU's (with fat cache) and make the move to SCSI drives? Your dual CPUs will help distribute your load caused by mysql and apache across two processors. One really fast CPU is still going to only be able to handle so many threads a millisecond. With two CPU's you can spread the load across the two of them a bit.

As for the SCSI drives, you'll get faster RPM and faster seek times. Seagate shows their SATA drives with an 8.5ms seek time while Seagates 15k SCSI drives 3.6ms seek times. While you say disk I/O is not an issue now, for future growth and performance I would be very concious of making sure my I/O system does NOT become a bottleneck in the future.

In addition, moving to SCSI drives will help move more processor load to the SCSI controller to handle locating various blocks of data. When the CPU does need to send requests for data from the disks it can take advantage of SCSI's TCQ to send multiple requests at the same time freeing itself up for other tasks. With the IDE drives, the CPU's processor is having to handle requests for blocks of data on the disks from start to finish with no help from the "smarter" SCSI controller.

So the SCSI drives will insure future needs for growth are met and most likely result in less CPU time than the IDE disks currently.

And finally, there are some really good troublshooting tips in the O'Reilly book 'High Performance MySQL'. This could help you folks track down which queries are especially problematic and causing spikes in CPU usage and disk usage.

Just my 2 cents! Again, you folks are doing a great job and it is much easier for me to sit on the outside and make suggestions than be the one that has to implement!

 

[Krieg]

At peak use times MySQL occupies 75% of the processor's overhead. Speeding the processor up is key to speeding MySQL up.

Ditching the SATA drives in favor of SCSI will obviate this somewhat. Don't underestimate the advantage of SCSI's better CPU utilization.

Not to mention that intentionally going with the less reliable option is usually asking for trouble...

PS - When in comes to databases, random seek time is ALWAYS an issue.

Edit: Well crap IW beat me to it. lol

 

[Psionicist]

I showed this diagram in the other thread. Here we go again.

The SCSI-drive is twice as fast. And that's the best SATA-drive right now, a WD Raptor II (74GB).

Michael Morris: "What MySQL isn't eating Apache is. PHP script isn't like C - it essentially has to be loaded and compiled from scratch EVERY TIME the page loads."

This can be solved by an accelerator, such as turck mmcache or PHP Accelerator.

Turck MMCache is a free open source PHP accelerator, optimizer, encoder and dynamic content cache for PHP. It increases performance of PHP scripts by caching them in compiled state, so that the overhead of compiling is almost completely eliminated. Also it uses some optimizations to speed up execution of PHP scripts. Turck MMCache typically reduces server load and increases the speed of your PHP code by 1-10 times.

We use it at http://forum.sweclockers.com/ (48 000 members, 7 200 000 hits/day) and it really helps the load down. We are also in the middle of a server upgrade. A Dual Xeon 2.4GHz with SCSI-drives worked perfectly for 45 000 users (with search "free for all"), but now things are getting slower.

 

[Zoatebix]

Hey Psionicist!

What are the axes of that diagram, again?

Thank you,
-George

 

[Rackhir]

I have a Perc-2DC dual channel Ultra-2 SCSI controller here at work that isn't seeing any use. I might be able to persuade my boss to let me donate that. Would it be useful?

I'm putting together a test opteron system at my job to get an idea of how it performs vs a Xeon System, if that information is helpful. Hopefully, all of the pieces will have arrived by early next week and I'll have it up and running shortly there after.