PCPartPicker

  • Log In
  • Register

Wow, what a day

Forum Search

Guidelines

  • Be respectful to others
  • No spam
  • No NSFW content
  • No piracy or key resellers
  • No link shorteners
  • Offensive content will be removed

Topic

philip 39 months ago

Merry Christmas everyone!

What a day. It was fun (mostly), but I'm glad it's over.

I was up until about 1:30am last night finishing up wrapping presents (yay procrastination). My youngest child woke me up at 5:30am this morning screaming out asking if Santa had come. "Yes, he did. Please go back to sleep." (Last night we told him Santa wouldn't come unless he was in bed. I've never seen him run to get in bed so fast in my life.)

Then we had the Steam debacle. Wow. I... I can't even imagine being on the other end of that. Doing web based stuff, I can imagine some bad failure modes where things like that could possibly happen. But that's a nightmare. Seeing stuff like that makes me incredibly thankful we don't ever do anything with credit cards, addresses, personal information, etc.

Then later in the afternoon I got the ping that our site was inaccessible. Sure enough, all of Linode's Dallas datacenter was essentially offline. We serve a good portion of east and west coast traffic, so the bulk of our infrastructure splits the difference and sits in the Dallas Linode region. Today it seems amidst the Steam DDoS and other service DDoS attempts, Linode Dallas also was DoSed. It didn't last too long (maybe 15 min?) but it had some lingering effects that made the site pretty sluggish. I checked what I could on our end and things looked ok. So thinking the coast was clear I went on a bike ride with my kids. At literally the furthest point on the loop from the house, I got another notification that the site was inaccessible again. So we rode back, and my oldest's bike chain fell off. Literally three times. Oh no worries, no rush... lol. Got home, investigated more, poked around a bit, and got things back to normal operational status and performance again.

I think it's safe to say we're probably going to migrate off of Linode in the near future. In the beginning (for us) they were pretty robust with minimal downtime. But in the last year we've seen a number of issues particularly in the Dallas datacenter. A few years ago, AWS and US-east had lots of major outages, so we opted not to migrate there then. Since then their robustness has improved (but still isn't perfect). The difference though, is that when Linode Dallas goes under, people think it's your fault. When AWS US-east goes under, half the internet is failing and everyone knows it's Amazon's fault and not yours. But I think it's about time that infrastructure-wise, we start looking at what it would take to be able to survive AWS availability zone outages and keep running. That's a pretty big shift, but could be fun to do.

So now the site is running smooth again, the kids are in bed, and the day is almost over. Phew. About time to start working on the code again, gearing up for the next big go-live code push.

Comments Sorted by:

FH100 1 point 39 months ago

Steam debacle? Took a good ... 7 hour? car ride for some great french toast (got the recipe!) and haven't been near a computer until now... Quick search seems to say something about people being able to access other accounts? You mention a DDoS, so was that the cause then?

When AWS US-east goes under, half the internet is failing and everyone knows it's Amazon's fault and not yours.

Ah, yes, clever tactic... Once PCPP becomes large enough to do that themselves, people will think it's Amazon... I see what you're up to! :P

Anyway, have a happy whatever everyone!

BryanMUFC 1 point 39 months ago

Wow, what a chaotic day it was for a small portion of the World Wide Web...

I can't wait to see what new surprises await inside the PCPP headquarters... :-)

Anyway, merry Christmas everybody!

AceBalthazar 3 Builds 1 point 39 months ago

Christmas time has not been fun on the tech front lol. HDD failures, steam, blasted internet sporadically failing, around here everything hit the fan and shattered lol

manirelli staff 8 Builds 1 point 39 months ago

I know the feeling. I'm sitting here 1200 miles from my NAS watching errors pile up on one of my drives :/

AceBalthazar 3 Builds 1 point 39 months ago

oh that is no good.

i would be pulling my hair out of my head. literally. because the stress would be making me go bald.

i thought my local drive failing was bad enough

manirelli staff 8 Builds 1 point 39 months ago

Thankfully it is running RaidZ2 so I can lose a drive or two in a worst case scenario without any data loss. It just drives me crazy getting the emails with each and every error and I can't do anything about it.

AceBalthazar 3 Builds 1 point 39 months ago

i can imagine. i was sitting here watching files take five minutes to load, thinking "something isn't right," then starting the shutdown process as i watched the HDD become unresponsive. i unplugged it after shutdown and didn't plug it back in lol

manirelli staff 8 Builds 1 point 39 months ago

RIP [*]

[comment deleted by staff]
philip staff submitter 10 Builds 1 point 39 months ago

LOL no, he got a bicycle.

[comment deleted by staff]
philip staff submitter 10 Builds 1 point 39 months ago

I would confiscate said gifts for myself.

[comment deleted by staff]
[comment deleted by staff]
Xorex64 1 point 39 months ago

Only 70? How much did you have in your steam wallet?

If i was a hacker, i would spend all of the money in the wallet

[comment deleted by staff]
Xorex64 1 point 39 months ago

I guess they were nice enough to leave you 10 dollars