Amazon Killed the Internet

Tonight, these sites and perhaps hundreds more died. At least temporarily.

Amazon's AWS servers went down effectively disabling Netflix, Tumblr, Instagram, Woot, Pinterest, Skype, Zynga and more

The Internet was ablaze (almost ironically) tonight when a massive storm on the East Coast caused a power outage that affected Amazon’s “Amazon Web Servicescloud server system. AWS is what power so many sites and services these days. It’s their reliability, scalability, cost and speed that have so many startups jumping to their system.

Tonight became Survivor - Startup Edition

Tonight became Survivor – Startup Edition

However, tonight highlights what happens when you rely on a single fail point. What’s the adage? “You’re only as strong as your weakest link.” Well tonight it was a single Amazon server location’s power outage that has many of the most popular sites offline. Amazingly, Twitter is not one of the sites shutdown.

 

Once the power is back on, or once Amazon starts shifting bandwidth and syncing servers, all of these sites will be back up and running. What’s disappointing is that even though Amazon has tons of servers all over the country, it only took one power outage to cripple their system. 

Until they do, it’s not only annoying, it’s losing these companies tons of money. Oh and did you know this happened a week and two weeks ago? That enterprise cloud service Google’s releasing is looking sexier.

Having a backup in place is elementary. Even I, a simple blogger running WordPress, know better than to put all my eggs in one basket. I offload a ton of my bandwidth to Amazon’s cloud servers because their fast and reliable. However, when they go down, shouldn’t my site go down? Not in my case. I had a backup in place.

In fact, I use Amazon to literally backup all of my sites and databases as well. Those backups are mirrored and chronicled on two other locations. I wrote about my process over there. I wonder how many people cache/run their site off of AWS and backup their site on AWS. I’d wager more than we’d like to believe.

At the very least tonight is a wakeup call to web services that rely on a single cloud. A hurricane, earthquake, tornado, wildfire, flood… Any of these alone can cause these sorts of outages. There’s got to be a smarter way to run these clouds. If not, maybe we’re not ready for a cloud-based life.

Luckily Hulu, Twitter and Facebook are up, so none of us actually have to leave the house to find entertainment and socialization. What a relief.

90 comments
tracey14
tracey14

What’s disappointing is that even though Amazon has tons of servers all over the country, it only took one power outage to cripple their system. Thanks for sharing..

reddy2go
reddy2go

 @justex07 Shared this on my Facebook page. Sorry about the link not showing up the first time. It was a Facebook error. Link added and error corrected.

justex07
justex07

 @reddy2go Thanks so much, glad it was fixed. Facebook can be tricky.

justex07
justex07

@mikeasaursrex91 @RolandoCFC yay thanks for sharing my post! :-)

mikeasaursrex91
mikeasaursrex91

@justex07 yeah. No problem. I tried tag you. But then it was too long.

justex07
justex07

@TheNYGalavant @motoridersd well it's certainly evidence that the systems we have in place now aren't as rock-solid as we'd like to believe.

justex07
justex07

@motoridersd Thanks and thanks for sharing it :D

thejustins86
thejustins86

@justex07 This. Is. Why. I. Love. You. Haha always educating me!

phlpes
phlpes

@justex07 Overload, overload. Self-distruct in 10..9..8..7..6..5..4..3..2..1..

justex07
justex07

@JackPrinya haha, never have I ever been told I use limited "nerdspeak". Perhaps this is a new avenue for me :D

justex07
justex07

@drewmaniac thanks for sharing the link :D

drewmaniac
drewmaniac

@justex07 No problem man! Enjoyed the post.

TheDigitalNinja
TheDigitalNinja

There is always a point of failure. Even if you use multiple data centers and multiple providers a dns issue can still take out everything. I think the best way to stack is to use a group of racks servers that you physically control and spread the load out to the cloud as you need to scale. 

KristieKenney
KristieKenney

@justex07 @reginabeth thank you for clarifying!

justex07
justex07

@nate_pawley thanks again for sharing!

StacieCordray
StacieCordray

@nate_pawley Thanks for the info! I found it odd that all those sites were down. And how stupid of them to all be linked to one server.

nate_pawley
nate_pawley

@StacieCordray you don't realize it but many sites are linked to one server, reason because the server is so strong.

orchidhunter
orchidhunter

@justex07 @dws179 Or what Justin said, more eloquently than I. :)

justex07
justex07

@orchidhunter @dws179 haha, yeah power outage took out one of their 5 servers, but system failed, other nodes should have picked up slack.

justex07
justex07

 @tracey14 Exactly. Part of the reason that happened was because on that day, totally coincidentally, we added 1 second to the day. Literally, all atomic-clock set clocks added a second to the day. So when the outage happened and the servers reset, they were off by one second which kept the from actually syncing. Talk about a perfect storm!

tracey14
tracey14 like.author.displayName 1 Like

What’s disappointing is that even though Amazon has tons of servers all over the country, it only took one power outage to cripple their system. Thanks for sharing..

justex07
justex07 moderator

 @tracey14 Exactly. Part of the reason that happened was because on that day, totally coincidentally, we added 1 second to the day. Literally, all atomic-clock set clocks added a second to the day. So when the outage happened and the servers reset, they were off by one second which kept the from actually syncing.

Talk about a perfect storm!

tracey14
tracey14

What’s disappointing is that even though Amazon has tons of servers all over the country, it only took one power outage to cripple their system. Thanks for sharing..

justex07
justex07

 @reddy2go Thanks so much, glad it was fixed. Facebook can be tricky.

reddy2go
reddy2go

 @justex07 Shared this on my Facebook page. Sorry about the link not showing up the first time. It was a Facebook error. Link added and error corrected.

justex07
justex07 moderator

 @reddy2go Thanks so much, glad it was fixed. Facebook can be tricky.

reddy2go
reddy2go

 @justex07 Shared this on my Facebook page. Sorry about the link not showing up the first time. It was a Facebook error. Link added and error corrected.

mikeasaursrex91
mikeasaursrex91

@justex07 yeah. No problem. I tried tag you. But then it was too long.

justex07
justex07

@mikeasaursrex91 @RolandoCFC yay thanks for sharing my post! :-)

mikeasaursrex91
mikeasaursrex91

@justex07 yeah. No problem. I tried tag you. But then it was too long.

justex07
justex07

@mikeasaursrex91 @RolandoCFC yay thanks for sharing my post! :-)

justex07
justex07

@drewmaniac thanks! I’ve been so busy with work that my blog has lagged. Hope to get back into the swing again.

drewmaniac
drewmaniac

@justex07 No problem man! Enjoyed the post.

justex07
justex07

@TheNYGalavant @motoridersd well it's certainly evidence that the systems we have in place now aren't as rock-solid as we'd like to believe.

justex07
justex07

@TheNYGalavant @motoridersd well it's certainly evidence that the systems we have in place now aren't as rock-solid as we'd like to believe.

justex07
justex07

@motoridersd Thanks and thanks for sharing it :D

justex07
justex07

@motoridersd Thanks and thanks for sharing it :D

justex07
justex07

@thejustins86 My goal is to someday have answers to all of the world's questions available right here lol. Like a valuable @formspring :-P

thejustins86
thejustins86

@justex07 I usually retweet at the very least. I also appreciate when you link me to answers in your blog. Lol

justex07
justex07

@thejustins86 That's what a few other people are telling me. The post has been shared almost 100 times in like 12 hours! Makes me happy :D

thejustins86
thejustins86

@justex07 you make it simple. So thanks.

JackPrinya
JackPrinya

@justex07 Oh there were sprinkles of nerd jargon, but it was still easy to follow and made interesting points about life in the clouds.

justex07
justex07

@thejustins86 awe shucks :D

justex07
justex07

@drewmaniac thanks for sharing the link :D

drewmaniac
drewmaniac

@justex07 No problem man! Enjoyed the post.

justex07
justex07

@drewmaniac thanks! I’ve been so busy with work that my blog has lagged. Hope to get back into the swing again.

justex07
justex07

@drewmaniac thanks for sharing the link :D