March 20, 2013

FLO Cycling - Rackspace Saves the Day


When we started FLO Cycling one of the last things we thought we would have to worry about was crashing web servers.  As a new manufacturer of a product we assumed there would be production delays, design modifications and marketing/sales struggles, but we never assumed we'd have to learn how to manage high end web servers.  Well... you live and you learn ;) 

Our site traffic is pretty unique.  On a day to day basis our web traffic is low, but on our sales days we get hammered for about 30-60 minutes!  This massive spike in traffic caused our site to crash several times during our first three Pre-Orders.  Being a new company it was acceptable the first time, but when we crashed again during Pre-Order 3 we were simply embarrassed and knew we needed to fix the problem.  In the end Rackspace (our current and awesome hosting company) provided a solutions that was in short perfect... more on Rackspace below.



We've covered a lot of our start-up process through our blog and on our "About Us" page.  This article discusses how we've overcome a major road block in our start up process.  This really has nothing to do with cycling wheels but non the less we think it's a cool story.  Here is our web server evolution from Pre-Order 1 to present day.

Website Hosting Intro

For those who don't know, a web host is a company that "hosts" your website.  When you sign up for a hosting account, you place all of your website files on a server and those files are given an address (known as an IP Address).  When a user types a url like www.flocycling.com into their browser, the user is sent to that IP address where they can view the website.  



There are typically 4 types of hosting accounts that you can purchase...

1.  Shared - With a shared account, you share a webserver with many other websites.  All resources like RAM are shared between all of the sites.  If more than one site is having a busy traffic day, the server can quickly become problematic.  This option is very affordable.  Prices typically range from $4-$15/month.

2.  VPS - A VPS or Virtual Private Server is similar to a shared server where more than one website resides on the same server.  The difference is resources are allocated to you and only you.  You are given a guarantee of maybe 1GB of RAM instead of "getting what is left over on a shared server".  This option can be cost effective for sites that experience higher than normal traffic and want to manage there own server settings.  Prices typically range from $30-$100/month.

3.  Dedicated - With a dedicated server, you are the only website on a server and all of the resources of that server are yours.  This is great for high traffic websites but it's quite expensive typically ranging from $150-$500+ dollars per month with the more reputable solutions easily being $400+ per month.
  
4.  Cloud Servers - In a way cloud servers are like VPS servers but they are extremely flexible and have the ability to be modified on the fly.  Rackspace is truly the master of cloud web hosting.  With the click of a button you can convert a webserver's RAM from 512MB to 30GB when you expect a jump in traffic.  They also have a ton of features that give server administrators endless flexibility.  The best thing is, you pay for what you use.  You can pay as little as $17 per month and when your traffic is about to spike you pay pennies an hour for the increase in resources.  Our full solution will be highlighted below. 

The Evolution

Pre Order 1 - Shared Server RAM Unknown (likely less than 1GB)
After several lengthy discussions with our first hosting (not Rackspace) we were assured we could take our anticipated traffic load.  Well... they were wrong.  There were thousands of requests sent to our shared server within seconds of our store opening and the site crashed immediately.  After 5 minutes of shear panic, we got the site back up but there were many problems.  The site was horrendously slow which caused dozens of people to hit refresh on our payment screen resulting in dozens of duplicate charges.  Our database crashed, meaning customers details weren't complete and confirmation emails were not sent out.  It took about 4 days to dig ourselves out of that mess.  

Pre Order 2 - Dedicated Server with 4BG of RAM
We knew during our next sale we wanted to be on a dedicated server but as a new company $400 per month was simply something we couldn't afford.  We moved to a new hosting company who informed us that they could easily move us from a shared hosting account to a dedicated server whenever we requested it.  The best part was we only had to pay for the time we used.  What they failed to tell us was the process of scaling up or down would cause our site to be down and could take up to a week in either direction.  We didn't crash during Pre-Order 2 but we couldn't handle the downtime.

Pre-Order 3 - VPS with 2GB of RAM
We stayed with the same host that we used for Pre-Order 2.  After several discussion with the systems administration team we were pretty much guaranteed we wouldn't crash with a 2GB VPS.  Yup... you guessed it... they were wrong.  We crashed and we crashed badly.  We were down for nearly an hour.  We were embarrassed and needed a solution in a bad way.

The Solution - Rackspace 128GB of RAM!!!
I contacted Rackspace after a friend recommended them.  From the minute I placed my first call I knew we were in good hands.  Rackspace has by far the best customer service I have seen in the business and the flexibility of their solutions are endless.  I'd like to say that they are not paying me to say this, I'm simply a very happy customer.  After explaining our unique situation, Rackspace placed us in their "Start-Up" program and assigned a team to our project.  The team taught us the ins and outs of the Rackspace system, and best practice solutions.  After a month or so of tinkering (slow for me because I was learning), I finally had the system set up.  Let's take a look at it below.

Simple Server Architecture on the Left / FLO's Robust Rackspace Server Architecture on the Right
The image above details two types of server architectures.  The schematic on the left shows a very simplistic design.  The website and database both reside on one server and when a user visits a url like flocycling.com, they go directly to that server.  This system is what we crashed during both Pre-Order 1 and Pre-Order 3.

The system on the right is our much more robust Rackspace solution used during Pre-Order 4.  This baby DIDN'T crash!  Here's how it works.

A user visits flocycling.com and is first directed to a load balancer.  The load balancer in our case was connected to four web servers that all contained a copy of the FLO Cycling website.  The load balancer runs an algorithm that calculates which web server is the least busy and it sends the user to that web server.  When the user requests a read or write to the database, the web server they are on connects to the database server.  This system has several benefits.  They are as follows.   

Redundancy - Running four identical web servers gives you redundancy.  If one of your web servers becomes overloaded and crashes, the load balancer automatically removes that web server from it's que and directs all traffic to the other 3 web servers. 

Plenty of Resources - In the simplistic architecture design (shown in the above image) the web server and the database server both share the resources of one server.  When you separate the web servers and database server, they have their own resources and are less likely to crash.  

In total we had 128GB of RAM running for Pre-Order 4 which is 32 times larger than ever.  Was it over kill?  Yeah probably, but the site didn't even flinch.  Instead of our regular routine, crash, panic, upset our customers, begin disaster recovery mode, we sold our entire container of wheels in 12 minutes without a single website or server problem.  The best part is, this massive system only cost an additional $5.28 for the hour of our Pre-Order.  Yes you heard that right, only $5.28.  After the sale we clicked a few buttons and our site was back to a very affordable 1.5GB system.  Rackspace provides the perfect solution for our unique needs.

I'd like to thank Rackspace for their support throughout this entire project and the ultimate success of Pre-Order 4!  With our new website architecture we have one less thing to worry about!

All the best,

Chris


6 comments:

test said...

Congratulations! We're glad we were able to help, and happy you are part of the Rackspace family! Please let us know if there's anything you ever need help with!

Best,

Jeremy Wasner
Rackspace | the open cloud company
www.rackspace.com

joe clement said...

Yeah..............we just all want to know when we can order the Flo 30.......

FLO Cycling said...

Jeremy Wasner,

Hey thanks for stopping by! As you can tell we are very happy with the solution. Thanks for helping out.

Take care,

Chris

FLO Cycling said...

Joe Clement... very funny! They'll be out ASAP. We promise.

Chris

Gray Loon said...

What did the process look like to prep for another pre-order? How do you go from a single server back up to the scalable architecture? How do you get the files updated across the web servers and the DB updated on the DB server. Is it all done manually?

FLO Cycling said...

Gray Loon,

Thanks for writing. I've answered your questions below in bold.

What did the process look like to prep for another pre-order? How do you go from a single server back up to the scalable architecture?

The architecture is now permanently scalable. Our "non pre order" architecture features 2 web servers and 1 DB server. At any time we can add additional web servers or increase the RAM of each server. The process takes about an hour in total to go from a 2-1 configuration (2 web servers 1 DB server) to a 4-1 configuration.

How do you get the files updated across the web servers and the DB updated on the DB server. Is it all done manually?

There are very few changes to be made on the web servers so those are all done manually. It takes seconds to complete. There is only 1 master DB server that each web server reads and writes too. Running more than one DB server in a Master/Slave configuration in the cloud has it's own challenges. We feel our system works well for our needs.

Let me know if you have any additional questions.

Take care,

Chris