Fundamentals - having enough bandwidth
I work with many large retail customers, and this is the time of year when many of them are scrambling to make sure that they can handle their anticipated peak traffic days ahead--Black/Green Friday, the day after the US Thanksgiving holiday, and on until Christmas. One of the fundamentals that seems so 2000 is making sure you have enough bandwidth. Forget the 8-, 4- or 2-second rules--if you don't have enough bandwidth to serve up the transactions, even with a Content Delivery Network handling all of your static content, then you have major problems. At least this customer realized that internal testing does not give you the full picture, and that one must conduct end-to-end testing to really know what is going to happen. Unless your customers are shopping from your data center, then it does not make sense to test from there. My mantra "unrealistic testing yields unrealistic results" is entirely true here.
The customer was at approximately 70% of their target for estimated traffic for Black Friday. They have 3 OC-3 connections coming into their datacenter, and each was over 90%. Packets were dropping, the hardware queue on the router was hitting 37 (out of 40), etc. What are they going to do?
It is only 3 weeks until Thanksgiving. Getting in a new OC-3 or an OC-12 is 60 days if the stars align right the first time. They could get a local gigabit ethernet, but that isn't going to happen in 3 weeks either.