this post was submitted on
435 points (73% like it)
676 up votes 241 down votes
shortlink:
to anything interesting: news article, blog entry, video, picture...

434
435

When this post is 1 hour old, reddit will be taken down for maintenance (self.announcements)

submitted ago by [deleted]

Greetings redditors,

On this lovely Sunday morning, I'm going to be taking some time to upgrade our Cassandra ring to version 1.0. While this upgrade can technically be completed with no downtime, I'm going to be taking the site down for extra precaution, and to ensure a clean rollback path.

I will be taking the site down at 0100 PST / 0900 UTC. I expect the downtime to last up to 30 minutes. As always, simply use private browsing or incognito mode to avoid the downtime.

cheers,

alienth

top 200 commentsshow all 215
sorted by:
best
hot
new
controversial
top
old

[–]xxcaponexx 120 points121 points ago

Is this reddit way of screwing the Austrailians?

[–]alienth 111 points112 points ago

Unfortunately the evening hours of redditing in Australia also happen to be the hours when we have the lowest level of traffic overall on the site. The reason I do maintenance at these times is there are fewer changes on the site, and things are easier to clean up or roll back should post-maintenance problems arise.

Solution: Get more people in OZ to use reddit, and fuck some other timezone over ;)

[–]katzpijamas 25 points26 points ago

I had to wake up at 2:30 in the morning to watch the Federer - Nadal match in the Australian Open, and the Aussies really seemed to relish that control.

So suck on this down time Australia!

[–]roy1990 11 points12 points ago

and i wake up at 3 am to watch NFL, fair enough!

[–]dyslexicsuntied 1 point2 points ago

You guys actually watch that? I thought the rest of the world just thought it was crazy and stuck to rugby. Or I guess ya'll have your own bastard child "football" sport.

[–]andrewinmelbourne 3 points4 points ago

He was being polite.

[–]dyslexicsuntied 5 points6 points ago

Polite?? So now you're telling me Australians are Canadians?????

[–]anthologyofinterest -1 points0 points ago

What you thought is pretty much spot on. The Superbowl is a good excuse to get together and drink though.

[–]roy1990 1 point2 points ago*

Yeah I do bro and my brother too. Colts fan here btw!

[–]ImAJerk 3 points4 points ago

Damn, reddit is down for tennis right now.

Maria Sharapova posts are coming...

[–]fecalmatter 74 points75 points ago

I KNEW IT. I FUCKING KNEW IT! THATS WHY MY POSTS DON'T GET ANY RECOGNITION; BECAUSE I SUBMIT IT AT THE HOUR WHEN YOU HAVE LOWEST LEVEL OF TRAFFIC OVERALL ON THE SITE! FFFFUUUUUUUUU!

[–]Bloodywhitechucks 44 points45 points ago

Don't worry, fecalmatter. I appreciate your posts.

[–]fecalmatter 24 points25 points ago

Actually, don't look through my posts; they are embarassing. But thank you for such praise good sir!

[–]Thefourthdoctor 25 points26 points ago

Now we ALL get to look!!!

[–]fecalmatter 19 points20 points ago

NO FUCK

[–]perfectlysufficient 11 points12 points ago

^ furiously deleting

[–]Putnam14 12 points13 points ago

I'm looking the hell out of your posts.

[–]Lazerus42 7 points8 points ago*

to mess with maintenance i will proceed to upvote every post you've ever had... in the next 12 min.

*Damn you have a lot of shit to upvote

***5 min left to make you the next wolf

[–]Captain_Generous 0 points1 point ago

Well arent you Generous!

[–]Dredyn 3 points4 points ago

Bahaha - internet fist bump.

[–]CarTarget 1 point2 points ago

And, why not, upvote!

[–]sixfourch 2 points3 points ago

I'm going to upvote everything you've ever done, bro

<3

[–]Thefourthdoctor 0 points1 point ago

This kinda stuff should happen more often!! Or would that bring the value of the karma point down?

[–]perfectlysufficient 0 points1 point ago

It's already $0.

[–]ImTheManOkay 0 points1 point ago

Just tagged you so I remember to appreciate you more.

EDIT: Thanks Reddit Enhancement Suite!

[–]Kaizoku-D 0 points1 point ago

I just upvoted eveything. Australasia FTW

[–]123choji 2 points3 points ago

Same here mate.

[–]jevon 2 points3 points ago

Come visit /r/nightshift :D

[–]fecalmatter 1 point2 points ago

SUBBED

[–]Kurskovich 0 points1 point ago

Lower traffic likely means they disappear slower, I am pretty sure it balances out.

[–]mikethefecalmonkey 1 point2 points ago

I've finally found you...

[–]Loopyprawn 0 points1 point ago

You've got some pretty shitty posts........... badum tsh!

[–]Lilyqt 6 points7 points ago

We only have 22m people in our country :S

[–]codereign 1 point2 points ago

22 million potential bot masters.

[–]InactiveUser 0 points1 point ago

22 people online though - dial up.....

[–]lud1120 0 points1 point ago

Déja vú post.

[–]KryptosV2 0 points1 point ago

There's no logic in that post at all! He really just hates us Aussies for no good reason!!

[–]RetractedPenis 0 points1 point ago

Fuck you. Australia rules. Give a shit country like the US the downtime.

[–][deleted] ago

[deleted]

[–]kimbof 2 points3 points ago

We're all about quality over quantity down here.

[–]MindlessDream 11 points12 points ago

It's 30mins...100 pushups, 100 situps > Toilet > Shower. REDDIT IS BACK UP.

[–]dgamer5000 23 points24 points ago

Toilet. REDDIT IS BACK UP.

[FIXED]

[–]atrich 7 points8 points ago

But... what will I read on the toilet??

[–]AussieDaz 5 points6 points ago

Reddit is back up. TOILET [FIXED]

[–]IHAILSATAN 2 points3 points ago

/r/getmotivated is here

[–]crappuccino -1 points0 points ago

You take a LOT of time on the toilet, man.

[–]MindlessDream 2 points3 points ago

I'll be on Reddit?

Oh.

[–]joshistheman3 1 point2 points ago

Says crappachino

[–][deleted] 0 points1 point ago

seems like for our sakes, these reddit outages may be a good thing

[–]sdk16420 2 points3 points ago

The day is just starting really bad here... (GMT+1)

[–]monkeysmarts 4 points5 points ago

Australian Open will be on, maybe that will be enough of a distraction for the tennis fans at least.

[–]mrsbluesky 3 points4 points ago

New Zealanders everywhere are suffering

[–]OmegaVesko 2 points3 points ago

Or morning Europeans. :(

[–][deleted] 1 point2 points ago

they have to play the odds, mate.

[–]the_suffering 1 point2 points ago

http://i.imgur.com/ut9Wt.jpg

[–]biasedview 1 point2 points ago

so thats going to be 8:00 in New Caledonia, sigh, I should have learned French, then at least I could try television...

[–]VeryMacabre 0 points1 point ago

Yes

[–]zaneyhaney54 9 points10 points ago

I know you guys just did your state of reddit post, but there weren't any specifics about your Cassandra ring's specs. How many nodes are you guys converting over to 1.0? What's the instance size they're running? Are they all in the same availability zone?

[–]alienth 4 points5 points ago

21 nodes running 0.8, on m1.xlarges.

22 nodes running 0.7, also on m1.xlarges. This ring is kinda falling apart and can't really be upgraded. We've spent the last several months recomputing the data which exists on it and writing it to the new ring. I'm hoping to put an end to this ring soon :|

All in the same AZ, as of today. We're working on splitting things up this year.

[–]zaneyhaney54 0 points1 point ago

1.0 is faster and less buggy, my company just did moved ours over earlier this week with very few problems. It's amazing what the Cassandra devs have been able to optimize and cleanup in such a short time. Are you guys using any of the third party tools like Datastax ops center to manage things?

[–]omnigrok 0 points1 point ago

Interesting - any particular hurdle to splitting into multiple AZs? I wouldn't have thought there to be any particular advantage to having them in one to start with.

[–]MindlessDream -5 points-4 points ago

I'm just replying to your comment to feel important. Thanks :)

[–]onlypoststableflip 49 points50 points ago

（╯°□°）╯︵ ┻━┻

[–]finsterdexter 13 points14 points ago

As always, simply use private browsing or incognito mode to avoid the downtime.

What are these dark magics you speak of? How does this avoid downtime... I don't even...

[–]alienth 9 points10 points ago

Whenever you don't pass a cookie to reddit, you get a cached response from Akamai. When the site is down, you can go to incognito mode to not send any login cookie, and you'll still be able to see the last cached pages served up by Akamai.

Logging out has the same result, but of course, you can't logout very easily if the site is down :)

[–]wonderbreadz 0 points1 point ago

Ah. TIL. I thought you were implying... something else.

[–]nombre_usuario 5 points6 points ago

to put it in 5 year old terms, the thingies they change are only related to your username and stuff like voting or anything that requires to know that you are finsterdexter (I read that in german... maybe I'm still a little high).

in incognito mode, you are not browsing as finsterdexter, so reddit allows you to browse with no problems, but you can't upvote or downvote or post things while the usernames are put back in operation

[–]SecondGuy 14 points15 points ago

Skyrim it is.

[–]xDeaMuffin 15 points16 points ago

And that is how 30 minutes turned into 30 hours.

[–]maxcrabb 5 points6 points ago

In memory of secondman. May he quest on forever.

[–]bvanman 1 point2 points ago

First SecondGuy, now secondman? WHEN WILL THE MADNESS END?

[–]Hellsbagels 18 points19 points ago

Damn Damn Damn!

[–]givemebackmynose 2 points3 points ago

submitted 59 minutes ago by alienth (_) [A] to announcements

Hey, wait a minu-

[–]stonedandhungry 6 points7 points ago

As always, simply use private browsing or incognito mode to avoid the downtime.

Fuck, TIL.

[–]WarChimp 6 points7 points ago

What if we downvote this announcement and then maybe there will be no maintenance.

[–]from_the_sidelines 1 point2 points ago

While this upgrade can technically be completed with no downtime, I'm going to be taking the site down for extra precaution, and to ensure a clean rollback path.

Cheers from a fellow sysadmin! It doesn't matter how many times you stage things in a QA environment; whenever you touch production Murphy's law loves to take a dump all over your face. Best of luck!

[–]alienth 0 points1 point ago

/highfive

[–]phire 0 points1 point ago

So, did you end up having to rollback?

[–]EnjoyMyDownvote 0 points1 point ago

Hi Alienth. Just wondering if you could see what's up with my account. My new posts aren't appearing under reddit.com/new so no one ever sees them. Actually even on my other accounts the same thing happens. Perhaps it's an IP address problem? Thanks.

-EMD

[–]StymieGray 7 points8 points ago

The funniest thing was seeing everyone try and post a witty comment, then downvote everybody else to make theirs the best.

[–]i542 3 points4 points ago

those jokes were funny first one billion times.

[–]EpicPickle 10 points11 points ago

penis.

[–]pspalo -4 points-3 points ago

vagina.

[–]romkeh 0 points1 point ago

PEeenis

[–]TrustworthyAndroid 2 points3 points ago

Productivity approves this message

[–]maku450 -1 points0 points ago

34 Minutes... Clock is ticking...

[–]L551 0 points1 point ago

20 minutes to go..

[–]lookitssupergus -1 points0 points ago

Ten minutes....

[–]Cameron_D 1 point2 points ago

What will we do?

[–]atanok 0 points1 point ago

Just enough time to watch yesterday's episode!

[–]Cameron_D 1 point2 points ago

Episode of what?

[–]atanok 1 point2 points ago

You'll never know!

[–]matthew07 -1 points0 points ago

Right in time to get some actual fucking work done.

[–]Buttpudding -1 points0 points ago

I was going to bed anyway.

[–]anthdx -1 points0 points ago

'Tis a time of darkness.

[–]exoendo -1 points0 points ago

*procedes to load up 50 tabs*

[–]PenisFlyTrap 0 points1 point ago

OH FUCK 38 MINUTES AGO

[–]DLXII 0 points1 point ago

:(

[–]SandwichBoy 0 points1 point ago

NOOOOOOOOOOOOOOOOOOOO- wait...

[–]captron 0 points1 point ago

21 minutes to go!

[–]ConfuciusBateman 0 points1 point ago

I ONLY HAVE 20 MINUTES LEFT WTF DO I DO

[–]Sephiroth912 0 points1 point ago

And I just finished fucking my mind with Videodrome. Was gonna stick around here and discuss it, but I guess I may as well just move onto my next big mindfuck: Timecrimes

[–]really_Dude 1 point2 points ago

It would be so awesome if afterwards you did a small post on the procedure :)

[–]MingusFan 0 points1 point ago

this comment gets buried in jokes....ah, tis the way of reddit....

[–][deleted] 0 points1 point ago

BAH

[–]Talamant3z 1 point2 points ago

Guess I'm going to sleep

[–]BestReadAtWork 0 points1 point ago

Oh god I just got here. I have 20 minutes to feed my addiction before I go into withdrawal. D:

[–]mulduvar 1 point2 points ago

When this post is 1 hour old I will be going to bed.

[–]desktop_ninja 0 points1 point ago

Who are you, my mom?

Regulating my sleep patterns like this...

[–]forbearance 0 points1 point ago

Nooooooooooooooo

[–]kellums 0 points1 point ago

Downtime coming? Ready your bananas, everyone.

[–]Craven_Moorhead 0 points1 point ago

NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO

[–]Loxville 0 points1 point ago

RARRRAGHHHH!!! 46 mins to go! Glad I just got onto this. Upvote everyone!

[–]MONOTONE_ALLCAPS_GUY 0 points1 point ago

I CANNOT WAIT FOR THIS UPGRADE

[–]waxlion 0 points1 point ago

Nooo! I just got back from camping for 4 days. I need to catch up.

[–]zweizweifunf 0 points1 point ago

OH shit, gotta get my browsing done asap, 12 minutes to go

[–]new_abcdefghijkl 0 points1 point ago

i guess that is when i go to sleep

[–]sm_x 0 points1 point ago

you mean when this post is 1 hour old, reddit will be hacked?

[–]logicallychallenged 0 points1 point ago

Ah crap I just signed on.

[–]iloveyourname 0 points1 point ago

Woke up from a dream of our alien overlord(s) being teleported to a safe room with pensive scientists ("incoming! My god, what is it?!"), the head scientist being cold-cocked by conspirators, then fed to the emergent alien (nice throat grab, decapitation). Suddenly felt the need to hit the shitter. Opened reddit to find this message. Coincidence?

I, for one, welcome our downtime overlord, alienth.

[–]Baconadors 0 points1 point ago

If I try to avoid the downtime, I'll only last a couple minutes... :(

[–]Flexmeister 0 points1 point ago

Preparing for the "What I did during the blackout" posts.

[–]unohoo09 0 points1 point ago

OH SHIT ONLY 7 MINUTES WTF DO I DO

[–]1WithTheUniverse 0 points1 point ago

Damn I just woke and its about to go down :( First world problem.

[–]montagv3 0 points1 point ago

THANK GOD I can finally go to sleep! ...Australia's mid-day is around 4:00am ...huh...

[–]preo 0 points1 point ago

6 minutes :/

[–]jaxspider 1 point2 points ago

When I get back from taking a massive shit, reddit better be back up.

[–]codyhorne 0 points1 point ago

its at 55 minutes :O

[–]Prius 0 points1 point ago

It's stressful to see this three minutes beforehand. Hurry!

[–]Crogater 0 points1 point ago

OH NOOEESSSSSS

[–]maker00 0 points1 point ago

1 min left!

[–]mistressfalulu 0 points1 point ago

Noooooooooooo D:

[–]kareltjeftw 0 points1 point ago

Exams tomorrow, I'll have one solid hour of studying.

[–]fuzzyfroboy 0 points1 point ago

fuck i just logged on...

[–]madkiwi 1 point2 points ago

That was rough...

[–]chaz6 0 points1 point ago

Did everything work out okay?

[–]sonQUAALUDE -1 points0 points ago

LIES!

[–]narzy 0 points1 point ago

Booo ಠ_ಠ

[–]DrProv 0 points1 point ago

Looks like bedtime is postponed again!

[–]djluca 0 points1 point ago

submitted 1 hour ago by alienth

WHERE IS YOUR MESSIAH NOW, REDDIT?!

[–]JustAnAvgJoe -1 points0 points ago

Great, now I can go to sleep

[–]speedycat 0 points1 point ago

Get out your popcorn!

[–]Laeryken 1 point2 points ago

Thanks for the heads up!

[–]pill_popper 2 points3 points ago

Why do you hate freedom?

[–]TylerBDogs -1 points0 points ago

Well wtf do you want us to do? SLEEP?!

[–]theredpolak -2 points-1 points ago

Time to start drawing...

[–]Fortitude21 0 points1 point ago

Are you drunk? You should ask for requests. Oh wait...

[–]rockstar504 -1 points0 points ago

Well, time to go to sleep.

[–][deleted] ago

[deleted]

[–]iamaelephant -1 points0 points ago

No one cares.

[–]fukitimout -5 points-4 points ago

that is pretty gay, fuck off

[–]WhiskeySeven -2 points-1 points ago

Aw hells bells, Trudy!

[–]l0n3s0ld13r -1 points0 points ago

Quick! Open 30+ mins worth of reddit content before it goe-

[–]hurricaneR 0 points1 point ago

But what will I do while I an drunk...........

[–]spazzikarp 1 point2 points ago

Drink more! :D

[–]captainwalnut -1 points0 points ago

noooooooooooooooooooooooooooo!

[–]Pravusmentis -4 points-3 points ago

Well I'll just start my own site, with blackjack. And hookers, you know what? Im sorry for that outburst, you know I wouldn't be anywhere without you

[–]FrownSyndrome 1 point2 points ago

Actually, I'll be using Reddit at that time. Go ahead and hold off a few hours.

[–]Awful_Antagonist -4 points-3 points ago

Quick! Everyone! Cash in your karma before it's too late!

[–]line10gotoline10 -5 points-4 points ago

At first for whatever reason I read:

When this post is 1 year old, reddit will be taken down for maintenance (self.announcements)

and thought to myself, that's a really cool way to schedule some very aggressive maintenance. Think of the acclaim and discussion a year-long threat of downtime would generate on a site like Reddit. Hell, we only knew the SOPA Blackout was coming for a few days, and that was a news-worthy mega-event.

[–]fukitimout 2 points3 points ago

Jane you ignorant slut.

[–]geniusbean -5 points-4 points ago

now we can all go to church

[–]LovingJudas -2 points-1 points ago

Thank you for the warning!! I'll dust off my stumpleupon account I guess..

[–]l0n3s0ld13r -2 points-1 points ago

Funny, I got an emaail earlier today from them asking me to "come back" Wwll guess I just might.

[–]areo -3 points-2 points ago

But... but I just woke up! :(

[–]sixtyninenicely -1 points0 points ago

Noooooooooooooo

[–]PotentialTroll -5 points-4 points ago

Good thing I have this ambien.

[–]peaches017 -2 points-1 points ago

These types of posts are normally reserved for /r/starcraft

[–]SymphonicPsychosis -2 points-1 points ago

Dammit, I guess that means I should go to sleep.

[–]gamenutt5 -2 points-1 points ago

Quite the grave prophecy...

[–]PlanningAHeist -2 points-1 points ago

It's go time!

[–]cwcoleman -2 points-1 points ago

Nnnnnnoooooooooooooo

[–]Freeroot -2 points-1 points ago

http://nooooooooooooooo.com/

[–]dirmaster0 -2 points-1 points ago

Its okay, I'm going to bed at a decent hour for once XD

[–]Frankstg 0 points1 point ago

You probably won't even see this, but I'd like to know a little about how it is possible to patch a site without having a downtime.

[–]alienth 2 points3 points ago*

We actually patch the code of the site all the time, middle of the day, whenever :) The infrastructure is much more tricky.

In the early to medium future, we're going to be serving up cached pages during downtime. So, the site will still be served up for these type of maintenance, just in "read-only" mode.

In the long-term future, we should be able to engineer things in such a way that most maintenance can be done without taking the site down. It is a bit of an engineering feat when you have a tech team of 8 people and a site with the size and complexity of reddit.

Edit: In this specific case, Cassandra actually has the ability to a "rolling upgrade", where you upgrade one node in the cluster at a time. This method allows you to have a no downtime upgrade, if desired.

[–]Frankstg 0 points1 point ago

Thanks for the answer.

[–]Never_Better -1 points0 points ago

THE END IS NIGH. REPENT, SINNERS.

[–]staircar -1 points0 points ago

Now what I am going to stay occupied as my fiancee keeps me, wide awake with his snoring.

[–]mocisme -1 points0 points ago

Err. Scrath that. Its not maintenance.its another SOPA protest. Hurr durr

[–]veruus -1 points0 points ago

IT'S STILL UP. YOU'RE SO FULL OF SHIT!!

Good work on the upgrade, guys. ;)

[–][deleted] ago

[deleted]

[–]Esteam -5 points-4 points ago

THIS IS THE WORST PART OF BEING AWAKE AT ONE IN THE MORNING

[–]jrmcgee1 -5 points-4 points ago

yep

[–]jrmcgee1 0 points1 point ago

yep

[–][deleted] ago

[deleted]

[–]thatsmystapler -3 points-2 points ago

alienth seems to be taking down reddit a lot since he got here. Punish Him!!

[–]joll3 -1 points0 points ago

Djoko vs Nadal is on anyway

[–]Brobeast -4 points-3 points ago

My response.

[–]RyukaChi -3 points-2 points ago

Okay..... I should be sleeping anyway....

[–]thedarkpurpleone -3 points-2 points ago

What has my life come to... I need a website going down for maintenance to force me to go to bed.

[–]AstralPro -3 points-2 points ago

This is unacceptable, we need bigger notice time! 1 hour isn't enough, I plan my life around Reddit. It's not like I am addicted to reddit or something, but this really fucks up my whole day schedule so I get my 8 hour a day dose of Reddit.

load more comments (27 replies)

    Use of this site constitutes acceptance of our User Agreement and Privacy Policy. © 2012 reddit inc. All rights reserved.

    REDDIT and the ALIEN Logo are registered trademarks of reddit inc.

    π Rendered by PID 7510 on app-140 running 4810ec8.