Sun Comes Up Over Silicon Valley

The Morning Skies of Silicon ValleyThe Morning Skies of Silicon Valley Hosted on Zooomr

Saturday 1:30 a.m. the phone rings in the middle of the night.

Kristopher: “Dude, Zooomr’s down.”

Me: “Huh, what time is it?”

Kristopher: “It’s 1:30 a.m., Zooomr’s down, we have to go down to the data center.

Me: “Ugh, dude I’m asleep. I just went to bed 30 minutes ago. I have to be up early this morning to head down to Photowalking with Scoble.”

Kristopher: “Yeah man, I know, but Zooomr’s down. We have to go down to the data center.

Me: “Ok, I’ll be there in 20 minutes.”

So I swing by SF, pick up Kristopher and we drive down to our data center at Savvis in Sunnyvale. Grumbling most of the way. Driving is tough.

“Just stay awake!” Kristopher reminds me as I drift out a few times on the way down.

We get to the data center, check in, and Kristopher gets to work.

“Crap,” he says, it’s worse than I thought. A drive’s gone down. More hacking on the computer at the data center. A twitter here and there on my part (I’m pretty useless on these calls except as chaufer).

Finally we fix the problem. Take the bad drive offline. And wrap up at about 5am. As we head out we joke with the guy at the data center about these capsules that you go in and out of at Savvis. They are like something out of star trek. Little phone booth type doorways that seal you off as you go in and out of the data center. Once he said, three guys from Google got stuck in one of them. Another time he tells us that one guy freaked out who was claustrophobic and had a collapse in there. They had to call the paramedics to get him out.

He hands us our IDs back and we say bye. Good night he says. No, good morning, Kristopher laughs back. Ok, now, goodnight, he says back.

And then we get in the car. We drive past this big ass building with Yahoo on it and both laugh. Then Krisopher lays back in the seat to try to sleep while I’ve got Van Morrison blaring on the radio and I pull out my 5D and snap the photograph above.

My friend Dave Sifry says someday this will be one of those great stories that our employees will love hearing us tell. I sure hope he’s right.

Loading Facebook Comments ...

11 Comments

  1. Jeremy says:

    Yeah, he’s right. These are the things great products are made of. My last major product rollout had several late-night/early-morning downtime emergency events, and now we look back on them and laugh.

  2. Yuvi says:

    He’s right. Driving down to a datacenter in the middle of a night so that my friends can see my crudely photoshopped photos is, uhm, wonderful. Thanks!

  3. Maurice says:

    Hmm if it been me i’d have been calling you from the cab on the way there.

    Or was this a dual key setup to get in? the entry device is called a Mantrap BTW

    I usued to do some silly hours running the billing for Telecom Gold (a pre internet email system)

    sometimes 3-4 days with 4-5 hours of sleep at month end.

  4. Thejesh GN says:

    I appreciate the effort to my our pics online

  5. Jeremy says:

    Driving Mr. Tate, huh?

  6. Jim says:

    Yes, believe it or not millions of people do stuff like this every single day. What’s really interesting is how a single failed drive can take the site down. Wow. I’d like to hear more about that.

  7. Anonymous says:

    Don’t you have people for this kind of work? It doesn’t make sense for the CEO and founder to be doing this work.

    Also, why don’t you get a data center closer? There are several in SF. We use 365 Main.

  8. Ben says:

    Yeah, according to the Zooomr blog, it was several drives in the RAID array that went haywire, and not just a single drive.

  9. Thomas Hawk says:

    Ben, actually when the one drive kind of freaked out the controller shut down all of the drives on that server. So this server also houses Zooomr’s system software. We had to use a backup of our system software, move it to another drive and then restart the system with all drives on that server working except the one that had failed. We will have to have that drive replaced with Dell.

  10. Ian says:

    Great story Thomas! You guys certainly put a lot of love and care into Zooomr to bring us the best photo service. Then you do what sounds like an amazing Photowalk! Dedication indeed. Thank-you!

  11. Anonymous says:

    Kristopher doesn’t drive?

    Multiple drives going bad in a RAID array is not nice (nor completely unknown to happen). Been there, done that.

    How’d 3 google serfs get caught in a mantrap? Usually those things have weight sensors etc to make sure that only one person at a time (based on your historically measured weight and an allowance for varation based on turkey day, notebook bag, heavy coat etc) is going in/out and that you aren’t significantly heavier on the way out (removing equipment via the trap is a no no).