The ULTIMATE Dedicated Server and Hosting Guide - READ FIRST BEFORE ANYTHING ELSE!

[CodyLoco]

-=[ Clip-Share's Server Requirements ]=-

I have finally broken down and decided to write this very very very in-depth guide to choosing what server you should be buying for your Clip-Share site. The guide assumes you will be using Clip-Share (or any other similar script) and you are intent on having a real, professional server set up to run your real, serious video-sharing website from. This does not give you any advice on the $99 Pentium machines you see advertised. If that is all you are willing to put into your website, I'm afraid you have wasted your money buying the Clip-Share license.

When choosing your server, here are the things you are going to want to know before you make your final decision. I'm going to point out what each resource does for your website, and why you aren't going to want to skimp out on it. This will hold true to any media sharing script/website that uses mencoder or ffmpeg to convert video.

Processor (CPU)

The processor in the server obviously is responsible for converting your videos. Most web-serving applications were designed with multi-tasking on a single core in mind, example, a LAMP server (the normal linux server, consisting of Linux, Apache, mySQL, and PHP ) is able to happily run all it's required processes on a single core of a processor, provided the load doesn't demand more processing time than is available. In fact, at the moment Apache, and mySQL, the two 'heaviest' processes on a server, don't even usually come equipped with the ability to multi-thread (run across multiple cores).

To clarify, I want to define the following:

Linux: An open-source operating system that is usually FREE (don't pay for it!) and has proven to be one of the most stable platforms to host busy services like web hosting servers. In the PC world, it is the equivalent of Windows. The required video conversion software only runs reliably on Linux. If you want to start using Linux on your own computer to get a feel for it, be sure to check out www.ubuntu.com

Apache: Is responsible for actually handling the data transfer between your browser like Firefox or Internet Explorer (If you still use IE you don't know what you're missing!) and the web server. Apache is very powerful and incredibly versatile with the amount of extensions you can install for it. The Microsoft equivalent to Apache is called IIS.

mySQL: The database engine. All the dynamic information for Clip-Share and the majority of PHP based software uses the mySQL engine to store any data that is not directly code for the site itself. The SQL database holds all the user information, video names, descriptions, ratings, view count, locations, etc and is a vital part of your server. mySQL is also usually the process that requires the most memory (RAM) and is the most likely to slow down the speed at which your pages load in your browser.

PHP: The script is obviously written in the PHP language, but the engine that makes something out of that code is also called PHP. PHP doesn't require too many resources (unless you are calculating PI to the millionth decimal), however, if it has to wait for mySQL to send it data because that process is overloaded then it will take a long time to run the PHP code.
All components of a LAMP server (the ones mentioned above) generally come as a package with your operating system. The OS version, however, usually dictates what versions of these programs will be installed. For example, CentOS 4 has Apache 1, whereas CentOS 5 uses Apache 2.

Mencoder and ffmpeg were never designed to run as a converter for a web server, and thus were never intended to happily run efficiently alongside other resource-hungry processes (especially another instance of itself) on the same core. They both will use 100% of the available resources on the server, simply because they are designed to convert video as quickly as possible, which of course involves using as many resources as possible.

This also means, if one of these two modules is busy converting a video and using 100% CPU, it won't allow other processes, such as apache or mySQL, to operate efficiently, thus queuing their work. As you can imagine, if a server has to wait before apache gets to run, that means that someone who is requesting a file such as a web page or video to be served by apache is going to have to wait too. This translates to poor website performance, especially when multiple users are online trying to fetch pages.

As a general rule of thumb, a single server designed for Clip-Share will run efficiently as long as there is at least one processor core that is not being used by mencoder or ffmpeg. Because these processes don't live together well at the same time, you are only able to efficiently run as many video conversions as you have cores, minus one. If you have a quad-core processor, then you will be able to efficiently handle three conversion processes at the same time, while leaving room on one core for the server to allocate to the rest of the important LAMP processes. On an eight-core, the server can handle seven conversions. On a dual core (or dual single core processors) the server can only handle one conversion efficiently at a time, as soon as two people have uploaded a video at the same time, performance is going to drop.

When choosing servers, you will often be given the option to choose either a processor with more cores at a lower clock speed (Xeon 5310, 4 x 1.6Ghz), or less cores with a higher clock speed (Xeon 5130, 2 x 2.0Ghz)… usually for the same price. Although results can vary, it's generally better to go with more cores than higher clock speeds because you want that extra headroom in case you get multiple uploads at the same time, whereas higher clock speeds is only going to lower the time it takes to do the actual conversion.
Finally, definitely stay away from any server running a Pentium, AMD Phenom or XP, or similar 'consumer' level chips. They were not designed to be run continuously and you aren't going to get the performance you would need. I even wouldn't recommend the older Xeon processors (such as the 3040 or 3060) because they just don't have the power to handle doing too many things at once.

A perfect balance of power and speed is a dual-processor Xeon 5130 server. This will give you four 2.0Ghz cores which will provide both speed, and of course the extra cores for simultaneous conversion. You can also often get a dual processor setup with the newer Xeon 5335 processors which would grant you eight cores at 2.0Ghz for around an extra $60 - $100/month.
Also worth noting that processors usually aren't that easy to upgrade in the future on a leased server and generally require you to move servers if you outgrow your processors, so be sure not to go cheap on the processor just because you don't think you will use the power right away.

Memory (RAM)

Random Access Memory is used as a sort of buffer between the processor and the hard drives. Due to their mechanical nature, hard drives are very, very slow when compared to RAM. When something on the server needs to be processed by the CPU (such as a video to be converted to FLV), it is faster to load it to the RAM first so that the CPU has on-demand access without the hard drive lag.

As a basic, general rule, I recommend you provide at a bare minimum at least 512MB of RAM per processor core, which means that if you have a dual Xeon 5130 server (which would be 2 x 2 x 2.0Ghz, so four cores) you would want a bare minimum of 2GB of RAM. For a more reliable server, and to avoid paralyzing the server in the event of a sudden influx of traffic, 1GB of RAM per core is what I highly recommend.

If you notice that your server is running sluggish and there are no conversions in progress that are slowing down your server, then mySQL may be out of RAM and is being forced to pull data directly from the HDD all the time, which of course slows down the return of data. Luckily, RAM is easy to upgrade once you start to run low at the 1GB per core rule, at which point you can usually bump your server up to 2GB/core.

Hard Drives

Hard drives are a simple choice as well, but there is something that many people don't consider when deciding how much space they need. The first one, is data redundancy. On average, hard drives have around a 0.5% chance of failing each year. It sounds like a small number, but you should realize that every year, that means one in 200 hard drives will fail and if that is your drive with the only copy of your data, your entire website just went down the tube.

There are two ways to protect your data. One is with a mirrored Redundant Array of Independent Disks (RAID). This process uses two identical hard drives and a RAID controller (the controller usually costs around an extra $50/month) to maintain an identical copy of the data on both drives (thus mirroring the information). If one of the drives fails, it can simply be replaced and the RAID array can be rebuilt by copying the data on the one good drive onto the new drive. While this provides protection against drive failure, it doesn't provide any protection against other data loss caused by hacks, human errors, exploits / viruses, etc. The only way to effectively protect against that is…

---------==========(Continues next post!)=========---------

[CodyLoco]

Regular backups. WHM (more on that later) has the ability to automatically create daily, weekly, and monthly backups of your data on the server. The problem though is that depending on your settings, you are going to end up with between one and three (compressed) full copies of your entire website, which of course needs to be stored somewhere. There are a few options for where you can store these backups such as Network Attached Storage (NAS) systems, remote FTP servers, iSCSI drives, etc; but by far the easiest is a dedicated backup drive inside the server itself.

As a general rule, you should have about twice the drive space for backups as you do for your main website, assuming you want to retain a daily, weekly, and monthly version of your backup. If you are only interested in keeping a single version, then you should have at least the same amount of space available for the backup drive.

So, in summary, if you want to use both ways of ensuring your data stays safe, then you will need at least three hard drives. Two drives are used for the mirrored RAID array, and a third of twice the size (or two striped drives of the same size, useful when using all 1TB drives) is used to store backups. If you only want to use one or the either (I recommend backups then in this case) you will only need two drives.

Although it is usually possible, hard drives are NOT easy to replace in your system once you have set up your site because you will have to have all your data moved onto the new drive or array and that will mean downtime. I also don't recommend using a striped array (which takes two or more drives and makes it look like one long drive) because although it does increase your I/O performance, you multiply your chance of failure by the number of drives in the array. For more information, check out Wikipedia.

Finally, how much space do I need? This is a very difficult decision because it is completely based on your target niche and user habits. Try to guesstimate how often you expect your users to upload videos and go from there, nobody can honestly give you a real number. For example, a site that you want to use to upload soccer games for a single team will have limited uploads because there are only so many games and only so many people at the games with cameras, whereas a site that covers sportfishing in general might have many more uploads since anybody with a fishing pole, a body of water, and a video recorder can upload a video.

As for a raw number, unless you know exactly how many videos are going to be uploaded, I recommend staying away from anything less than 250GB drives and recommend at least 500GB or 1TB drives. Once your site is at a point where it is using up 1TB of space, it probably has enough traffic that you should start considering a second server to offload the work onto anyways.

Bandwidth

Bandwidth, of course, is the most expensive and the most critical part of your server. Without access to quality bandwidth, you may as well burn your website to a CD and store it in your drawer, since it is going to be doing the same thing there as it would on a dual-quad-core server. Bandwidth is the quantative term for the amount, as well as the speed and quality, of data that is sent over the internet, usually to a client's computer in the form of pages or video. It includes bandwidth inbound and outbound (uploads and downloads) to the server.

The general tendency is to jump to the conclusion that the best bandwidth is cheap bandwidth, and that is obviously not the case. Quality bandwidth is very expensive. Let me stress that: Quality = $$$.

Of course, that's not to say there aren't companies out there that are selling second-rate bandwidth at an expensive premium, but I can assure you that nobody is selling first-rate quality bandwidth for anything under $100/TB. It simply can't be done when you take into account the amount of infrastructure involved. I have lots of people ask me to find them a good server with unmetered 100mbit bandwidth for under $300/month… and all I can tell you is that it is impossible. If you want an idea of what it would cost you, try calling your ISP and asking how much a 100mbit dedicated connection would cost (tell them it is for an office building downtown and you are getting rough estimates) and it will give you an idea.

Now, in a real datacenter, there will be multiple (read: very many) redundant data connections with several different providers at Gigabit speeds. Add to that all the requirements that make your server stay online, such as multiple redundant connections to different power grids, massive diesel backup power generators, HVAC air conditioning and filtering, foam fire suppressants that won't damage electronics, 24/7/365 on-site security and techs, … etc and you can see why real bandwidth costs as much as it does. If your website is just going to be a hobby and you don't care about reliability and performance, then by all mean go with someone offering "unlimited" bandwidth for $200/month, but don't complain when things go wrong.
With most servers you buy, the package comes with a set amount of bandwidth for you to use, usually around 2000GB of bandwidth per month. For the majority of websites this is more than enough to get established, however, you do not want to find out one month that you have transferred more than your allocated amount of bandwidth. Most hosting companies tend to want you to pre-purchase your bandwidth ahead of time in chunks of 1000GB (1TB) so they can plan their network strategy accordingly, and will charge you a very expensive overage fee per GB if you exceed what you have purchased. Always try to gauge the amount of bandwidth you expect to use each month and avoid getting too close to your maximum. If last month it was within 20% of the maximum, it's probably time to purchase more for next month!

Watch for deals on bandwidth. Occasionally, especially after a datacenter has done a network upgrade or expansion, you will see offers for either double or triple bandwidth with your server. These are a great way to avoid paying extra in the near future and you can usually save hundreds or more down the road.

Lastly, regarding port speeds (that's the 10mbit, 100mbit, or 1000mbit speeds you see advertised), you should probably start off with a 100mbit port, which usually only costs around an extra $10/month. This is the maximum data transfer rate your server will be allowed and if too many people are downloading from your server at the same time and your port speed gets maxed out, videos will begin to stutter during download/playback and it will be a slow, unpleasant experience for everyone. Even if you don't think you would use 10mbit, I have seen a small site with ten visitors a day get featured on the first page of Digg.com and receive over 10,000 video views in less than six hours.

Keep this in mind also when you are choosing your server hardware- you might only get one chance to make it big and so you better hope your machine can handle it!

Software and Miscellaneous

In order to control your server, I highly recommend you use a control panel. The most popular variety is the WebHostManager (WHM)/cPanel combination, which costs around $25/month in license fees. It will allow you to control nearly every aspect of your server from a backend (WHM) and also gives you user control panels for each website account on your server.

The operating system, other than strictly being a Linux variant, is a question of choice. To each his own, but if you have no experience and need someone to tell you what to go with, I suggest CentOS 4.x if you want Apache 1 and PHP4 as default or CentOS 5.x if you want Apache 2 and PHP5.

You might also want to consider signing up for a server management service. In almost all cases, PSM is the perfect solution to handle anything to do with your server software.

They are willing to set-up, optimize, and secure (a.k.a. harden) the server, install the required modules, and monitor your server uptime for you.
At $30/month, you won't find a better support provider than them. They are, however, limited by a 24-hour SLA (it's usually closer to 1-2 hours max, but varies) and do not guarantee emergency assistance. If you need better response times, search for Linux Web Server Management on Google.

In summary, my overall point is that if your website is something you really want to put the time and effort into, don't waste your time by limiting yourself to your hardware. If you were going to open a paving company, I would think you would invest in a little more hardware than just a shovel and wheelbarrow. Even though those do work… how far do you think you are going to get?

And finally, a word on upgrades. In my experience moving lots and lots of websites from one server to another, it is not worth the time, money, or downtime to buy a machine when you know you will be forced to upgrade to more powerful hardware in the near future. The worst thing that can happen is finding out that you have hit a glass ceiling that now you have to spend the time and resources to break through only to remember that you were the one that installed that ceiling by not setting up the proper hardware in the first place.

-----------=========(CONTINUES NEXT POST... AGAIN... PLEASE READ!)=======---------

[CodyLoco]

----------------------======================{[ NEED HELP? ]} ======================----------------------

Want my recommendations? I'm here to help anyone who needs a hosting solution and I'm here to answer their questions. Because of the contacts I have developed with past work around here and elsewhere, I luckily have some pretty decent connections and can get some amazing quotes from several companies.

Don't be shy: send me an email to codyloco -at- gmail.com or add me to your MSN or Google Talk account with that same address. You can also send me a PM on these boards. I am generally a friendly guy and will try my best to help with whatever I can… plus I can almost guarantee I can help you save money! How does a free 4000GB of bandwidth sound on your next server order?

Check out the following if you want to read some information about The Planet- you should really read about their network and datacenter so you understand what you are paying for:
The Planet Alpha Dedicated Servers

If you are looking for a quote, take a look at these. I managed to get them put through with a free upgrade to 6000GB of bandwidth, plus with the hard drives discounted, the setup fee waived, and another $50 discount on top of that... Keep in mind the free bandwidth itself is worth over $500!

The order forms / quotes only work for ONE order. If you click on a link below and it says the form is no longer valid send me an email or PM and I will do my best to get you an order form. If you do purchase a server, whether through the site directly or through the order form below, make sure you send me an email right away with your order ID number and I will call up The Planet on your behalf and have them place a rush order on your server so it gets set up as fast as possible

Dual Xeon 5130 (The server that matches all the specs):
https://www.theplanet.com/quote/?q=s...nhcw%3D%3D

Dual Xeon 5335 (Twice the cores for twice the conversion capabilities!):
https://www.theplanet.com/servers/qu...8mMzDN2Q%3D%3D

This document will be a work in progress and I'm hoping to update it whenever I have new information. If you have any questions, please ask them below or send me an email and I will get back to you right away. Of course, any comments or suggestions are also very much appreciated!

[CodyLoco]

(this message is just a placeholder for future content, reply below)

ffmpeg hosting guide

[CodyLoco]

(this message is just a placeholder for future content, reply below...)

[dargre]

Originally Posted by CodyLoco

(this message is just a placeholder for future content, reply below)

Great job Mr CodyLoco

Little bit long story, but there's no other way to learn things.

[karlinge]

This was very interesting, makes some of the things you explained to me before make even more sense (I`m a slow learner lol ).

Thx again for all your help-we`ll talk again when our business speeds up and we need a bigger/faster server

[chris]

Very Nice Article - Stickied.

[CodyLoco]

Originally Posted by chris

Very Nice Article - Stickied.

Thank you!

If anyone has any comments or suggestions, please feel free to add them and I will address them in the article itself!

[zxcjason]

Very nice review. Cody recently helped us get a 25% discount when my client is looking for more powerful servers. We have a clipshare site that has seen increasing traffic the past couple of months and require much more bandwidth. We eneded up with a private rack with 3 servers on it and 18Tb bandwidth/month, and that costs like $2400/month. Cody helped us negotiated with the server company and the 25% discount he got amounts to $600/month, which is significant. He has been most helpful throughout the process, and I'd recommend him to anyone that are serious with their website.