Technical thread for Server Discussion (merged)

[drothgery]

Deset Gled has some interesting suggestions, but I am confident the best performance will be achieved with a dedicated ENW-server. You can customize the hardware _and_ software for ENW's needs. (linux+apache+mysql+mmcache is a winning combo).

I'm partial to Win2K3 + IIS6 + SQL Server + .NET, but that takes a bit more of a budget than ENWorld's got.

 

[Piratecat]

I'll likely have some of the connection info from Cyberstreet later today. I agree that the best performance is achieved with a dedicated server, so (as stated in the FAQ) this isn't a discussion of "Shall we leave Cyberstreet right now?" I'm interested in comparing the two options, though, just to know what's out there.

 

[BSF]

OK, I just want to preface everything with the disclaimer that I know squat about specifics regarding EN World. Desert Gled has a few good questions and I think they should be discussed. I typed up a huge reply yesterday and then lost it when I had my user ID cache get hosed. (Just for the record, that is nothing on EN World's side. That was me at work doing other things.) So now I am doing what I should have done yesterday (and do for many of my longer posts), I am typing it up in a text editor and will copy/paste when I am done.

Desert Gled,
First of all, my apologies on the insufficient answer yesterday. All of us know that EN World uses a lot of bandwidth and I should have included that in my earlier post as a consideration. I do believe a dedicated server, leased monthly, is a strong contender versus a co-located server. I am not convinced it makes better sense, but I do not know since I can only speculate on several things. But it is good discourse and if nothing else, it does help demonstrate all the factors the server team has to take into consideration. They are doing a great job and will continue to do so, I am sure.

I will continue to use Cedant as a baseline because I am passingly familiar with their product offerings. Note, I have never used Cedant, but I know their name and have looked at them as a possible solution for a couple of different projects over the years. Also, I am in my "work mode" so I apologize in advance if any of this comes across as brusque, it isn't intended that way.

A dedicated server option certainly has some robust servers available. You can always make a server that is a little faster and a little better, but the dedicated server options are certainly good performers. On a purely hardware basis, it is difficult to argue against them. Since they are hosted, the ISP will have replacement hardware available. The servers will likely be on an upgrade schedule that is part of the monthly cost, etc. These are some fairly compelling reasons to consider a dedicated server at an ISP.

Basic hardware & OS troubleshooting will also be covered in the monthly fee. Especially since all the systems will be baselined to be the same. Places like that do everything to keep costs down by standardization. So, their server builds will try to use identical hardware with an identical configuration on top of that. Trust me, this _really_ helps keep maintenance, troubleshooting and repair as easy as possible. It is one of the things I strive to achieve with the servers I maintain.

This is all good stuff, but it does have drawbacks. For a site like EN World, there are performance optimizations that can be achieved by tweaking the kernel and applications for your specific needs. I am not sure if the server team has any of those planned, but in a dedicated environment, you will not be able to tweak things the same way. This is particularly notable with some of the PHP caching software because I am not sure we would be able to install it on a dedicated server, nor would there be any assurance that it would still be installed when the server is moved to a different piece of hardware. Because some of the performance optimizations could be significant, this might be a big concern.

Another concern is that a dedicated server does not give you direct access to the hardware itself. This can make troubleshooting difficult. If you need to see the console, or do anything from the console, Cedant does give you a small window of time with one of their techs. But if the problem cannot be quickly & easily solved, then you need to restore from backup, or pay for higher level troubleshooting/support. At $150/hour, this can add up pretty quickly. It is also notable that Cedant lists this rate for software/OS troubleshooting, customizations and patches. So, it might be possible to tweak the kernel and the applications, but it may require paying for the time of a tech to do that for you. This devours the $3000 buffer that might be realized by not purchasing a dedicated server.

There are several upgrades and value-added-services that are offered that can make the dedicated server a little more robust. OS upgrades, RAM upgrades, Hard disk upgrades, increased resiliancy, as well as backup options and resetting the server to presale conditions are all things that can be purchased. These are great services, however they also tack on more costs to the setup & monthly fees. It doesn't take long before before this starts adding an extra $40, $50, etc to the monthly cost.

There is also the bandwidth concern. I have no idea on the needs for bandwidth, though I _think_ I recall hearing that EN World would saturate a T1 connection. Keep that in mind a little further down and let's look at average bandwidth consumption. For just a moment, let's assume that EN World consumes an average of 1 Mb/minute. That seems a bit conservative, but it is an easy number to work with. 60 Mb/hour, 1440 Mb/day, 43,200 Mb/30 day month. 43 Gb/month easily fits in most of the plans. As I said, I think that number is conservative, probably grossly so given that we have 600-1100 users online most of the time. So, we just need to adjust for real numbers if the server team wishes to make those available.

Unfortunately, that is not the only consideration. Bandwidth at the moment is another issue. During peak times, EN World needs more bandwidth available than the average. So, let's take a look at this from a different angle. Co-location.

Cedant offers co-location at $75/rack unit/month. So, a 1U server would be $75, a 2U server (or 2 1U servers) would be $150, etc. Then you pay for bandwidth.

We know EN World's costs are currently ~$400/month. Piratecat has also stated that the hosting costs will go up by $50-$100/month. Let us take a guess and say that Cyberstreet charges $50/rack unit/month. It is just a guess, but it fits the range that Piratecat mentions if the server team hasn't made a decision of a 1U or 2U second server. Let us also assume that EN World also has a 1U server currently. These assumptions would put current costs at $50 for co-location + $350 in bandwidth. Ouch!

Let's look back at Cedant's offerings. Cedant offers a T1 connection at $299/month. If Cyberstreet's rates are roughly the same, it would seem that EN World has at least a T1 connection. However, I think EN World is getting a better deal than that from Cyberstreet. Though, I only base that on a vague recollection that EN World can saturate a T1, and shaky guesses on how current expenses break out. If I had to guess, and I have to without any other details, EN World is probably getting a good deal on an Ethernet connection that allows high burst rates and very good sustainable bandwidth.

I am not privy to the details of Desert Gled's conversation with aplus.net, but it is important to establish that EN World needs sustainable bandwidth for normal use + enough bandwidth for peak periods. Just because a given machine might have a high speed interface does not mean the ISP is going to allow the server to use all of that potential bandwidth. Bandwidth throttling on routers will be used to assure that each customer receives the bandwidth they were promised and paid for.

As an example, where I work we have an 8Mb internet connection that is growable up to 1Gb. Our LEC throttles the connection so we don't use more than 8Mb. I can also choose to throttle the bandwidth on our router to only provide a certain amount of bandwidth to our servers. In this case, I might throttle traffic to our SMTP and HTTP servers and leave a guaranteed level of bandwidth for the rest of our systems. It is important to understand exactly how much bandwidth would be offered to a dedicated server at any given peak moment. It would be possible to see site performance plummet during peak hours because the bandwidth was saturated.

Bandwidth is a big concern and the wrong bandwidth solution could eliminate most of the benefit of a faster server.

Back to the dedicated server vs co-located server debate. There is another issue to consider: Ownership of the server. With a co-located server, you have access to the physical hardware and if you need/want to relocate, you pick up the server and leave. With a dedicated server, you cannot do this. If EN World decided to move to a new host, the data and configration would need to be dumped to media, or transferred via the internet. We know the database is in excess of 1Gb right now. Perhaps that could be compressed far enough to be burned to CD, or maybe the files could be moved to a portable hard drive (at $150/hour), or maybe the files could be transferred, but this is a different level of ownership and complication.

It is an interesting assessment overall. I think it really helps point out the issues that the server team has to consider. A dedicated server, as opposed to a co-located server, is a compelling option. But there are many issues that might make it less effective. In the end, these are issues that the server team has to consider and assess. There are likely issues that I haven't even touched on yet.

My word this has become long. Look at the way I spew when I get to thinking. Stop me now! All of us want to see a robust, speedy, stable EN World community for a long time to come. I am sure the server team will make the best decision they can for EN World.

 

[Michael Morris]

The site's media is well in excess of 3 GB, not 1

 

[Deset Gled]

Thanks for the points, BSF. I appreciate the analysis and explanations. (BTW, it's Deset, though, not Desert. Don't worry, it's a common mistake). You make a lot of good points for both sides. And the price comparisons are quite interesting, but it is hard to know exactly how acurate they are without knowing better statistics of ENWorld.

I guess the thing that bothers me the most at this point is this: from the side of things that the user (me) is seeing things from, there's no evidence of the web team actually looking into any of these options. We are continually told how nice Cyberstreet is to us, but we've never been told that anybody has actually looked into these other options that are being discussed here. Nobody has come in to say that they checked with companies like Cedant and Aplus and found that they can't provide ENWorld what it needs for a competitive price. Instead, we're told how an admin works at Cyberstreet, and how friendly they've been. And quite frankly, from a user standpoint, it doesn't really seem that Cyberstreet is meeting our needs (I often have trouble connecting, experience slowness/timeouts during peak usage times, and can't help but notice the number of times this place has gone down in the past year or so).

If somebody were to come in with a cost analysis of both sides and show that getting another dedicated host wouldn't work, I wouldn't question it. I wouldn't even ask to see the report; knowing that someone is doing it would be good enough for me, and I trust them. But the fact that we can sit here and have a long discussion about it like this, and it seems like it's the first time such a proposal has come up is just, well, scary.

I would like to apologize in advance if I've hurt the feeling of any of the web team. I'm not trying to attack anyone or claim they're not competent in any way. I'm just calling things as I see them. It's even possible that somehas has done the kind of analysis I speak of above, and just didn't think it was necessary present the info to all of ENWorld's users. And PC, I realize that this discussion isn't the true purpose of this thread, but I don't know where else to take it at the moment. If you would rather start this up elsewhere, I have no problems with it.

 

[Morrus]

And quite frankly, from a user standpoint, it doesn't really seem that Cyberstreet is meeting our needs (I often have trouble connecting, experience slowness/timeouts during peak usage times, and can't help but notice the number of times this place has gone down in the past year or so).

But that's the whole point of the fund drive - it's not Cyberstreet's fault this is happening, it's ours: because we don't have an adequate server. Cyberstreet really have no blame at all in any problems we've had. If it were Cyberstreet's fault, we wouldn't be having a fund drive - we'd be switching in a heartbeat.

One of the problems with hosted options is the fact that you don't usually get full access to the box. This means that a) tweaking it exactly to your requirements is not facilitated, and b) any troubleshooting means we depend on some CS staff. My experience with such staff in the past has been very, very negative. And, if *real* problems occur (and trust me, they will), you end up with a lot of downtime and paying through the nose for support.

That's not the be-all and end-all of it, of course, but they are two factors which I consider to be important.

On top of that, the price savings suggested really aren't substantial. Bear in mind that we ideally want *two* boxes to run the site, and that at peak usage our bandwidth requirements rocket.

I looked into all of the options when I first moved the site to Cyberstreet. Even back then, (I was using CI Host), with far less traffic than we have now, Cyberstreet at $400/month was working out cheaper *and* vastly more convenient. While it is possible that prices in the industry have substantially dropped since then - and this is why PC asked for details of what people had found - my past experiences make me really, really hesitant about such a thing.

Added to that - the helpfulness and friendliness of the Cyberstreet staff, and Adlon's proximity to the box, are not to be underestimated. While that may seem inconsequential to the average user, when you're the one actually running the site and server, the difference really matters. I remember with CI Host I was sometimes up all night trying to get them to do what I wanted them to do, arguing with customer service staff via crappy online chat systems, not able to get a direct phone number to talk to a human being (and when I finally did, the human being was not adequate to my needs!) Nowadays, with Adlon and Walter nearby for any hardware issues, and with Thomas Heretic having direct access to the box for software issues, I have no such problems. It does matter.

 

[Piratecat]

Cyberstreet really have no blame at all in any problems we've had. If it were Cyberstreet's fault, we wouldn't be having a fund drive - we'd be switching in a heartbeat.

Very true. It's worth noting that there was one time in the last year that our outage was sort of Cyberstreet-related: it was during the hurricane, when the owner and Adlon personally took our server to dry higher ground because Cyberstreet is relatively close to a river. Other than that, all of our speed and outage problems have been the fault of our hardware being inadequate for keeping up with our popularity.

I think Russ did a better job than I could summarizing the pros and cons of our current arrangement. One thing that's hard to quantify is that when a problem rears up we have a very knowledgeable EN World member who has a vested interest in fixing the problem as quickly as possible while not charging us for his time and effort. Compared to some of the dreadful CI Host techs we've had to deal with in the past, this asset is hard to underestimate.

 

[BSF]

Thanks for the points, BSF. I appreciate the analysis and explanations. (BTW, it's Deset, though, not Desert. Don't worry, it's a common mistake). You make a lot of good points for both sides. And the price comparisons are quite interesting, but it is hard to know exactly how acurate they are without knowing better statistics of ENWorld.

Deset Gled, my apologies. I could swear I saw an 'r' in there, so I am not sure if I need new glasses, or just more sleep so my brain is less likely to skip over such things.

I completely agree that it is difficult to really understand what the best option is without all the details. It brings the discussion more to an academic debate than anything based on real-world concerns. But really, I am fine with that. I am not on the server team and I don''t expect to have all the details. However, I have offered to help in the past, and that offer is pretty much a standing invitation. I have confidence that if Kevin or Russ really thought I could bring something valuable to their discussions and decisions, they would ask. That they have not felt the need to ask just tells me that they have people already helping. Really, that's a great feeling to me. They have enough people helping and they have the wisdom to know that too many helpers becomes counter-productive. Too many cooks and all that.

I can understand your perspective though. We don't see all the discussions going on behind the scenes. We also want EN World to be as good as it possibly can be. It is hard to simply have faith that all options have been reviewed. You see problems, you see an offered solution, but were all the T's crossed and all the I's dotted?

As far as the problems we have seen to date, perhaps I have a slightly different perspective? My job is network and system administration and there might be times when it is easier for me to see the difference between an ISP issue and a server side issue. Though, I don't really know, since I don't know what background you have. By my observations, EN World doesn't have many issues on the ISP side. The issues come from hardware that is being taxed and needs to be retired. For what little value my assurances can offer, Cyberstreet hasn't been the source of our problems.

There is another consideration that comes into play when evaluating a hosted vs dedicated server vs co-located server decision. Morrus & Piratecat both mentioned that Adlon gives them a certain amount of peace of mind. That is one consideration that is very hard to factor into a value assessment. It is hard to put a dollar amount on that relationship. But there is a lot of value to be had from somebody working to resolve problems that actually cares about our community.

Problems will occur. Whether it is a hardware component shorting out, data corruption, etc, something will happen. When a problem crops up, you want somebody who will work beyond the call of duty to fix that problem. All other things being equal, you want to go with a solution that has the strongest relationships attached to it. I think Adlon, Thomas Heretic and Cyberstreet provide a very strong base to work from and we are pretty lucky overall.

Deset Gled, you are asking questions and you are trying to be part of the solution. In the scheme of things, I am just another user who is proud to be part of the community here. But I appreciate that you are willing to ask the questions and if the server team seems a little quiet on all of this, I suspect it might have to do with them already hashing a lot of issues out in the backchannels. I hope nobody takes offense at the discussion because in the end we are all after the same goal. We want EN World to be a great place to visit.

OK, maybe my post sounds a lot like "Have faith" and "Yeah it's good to have a guy on site." I suppose it is, to some degree. But really I am just trying to provide reassurance that I haven't seen the server team stumble in this process. If I thought they were really making a mistake, I would come out and say so.

 

[Piratecat]

Okay, I don't want to count my chickens before they've hatched, but it looks like Elocin has arranged for us to be given - GIVEN! - two Opteron 848 processors that retail at between $3600 and $4400 EACH. If this comes through, and I'm praying it does, we'll have a dual-Opteron server with two SCSI drives and probably 4 Gig of RAM. Fingers crossed, gang - unless I'm being stupid and missing something, this is a donation worth 2/3rds as much as all the other donations put together.

The noise you hear is my evil laughter as I contemplate the possibilities.

 

[BSF]

Wow! That's nifty. Dang Elocin, if that comes through you and your friend are heroes. Heck, you are pretty special for even trying! That would make a very robust server.