Alex Brugh’s Blog

This totally needs to be proof read, apologies for anything that doesn’t make sense or read especially well.

I met a super great girl on a flight back in December. The only problem is that she lives in Chicago. I started looking at jobs in Chicago pretty early on, quite idly however. One job that caught my eye was for Software Developer (or Software Engineer?) at Google. I’m kind of a Google fan boy, working there would be super dreamy. So I applied. Even if things with this girl didn’t go well, a new city and a new job (especially at Google) would be awesome. I fully expected to never hear from them.

Fast forward a bit and things are still going really well with this girl, so much so that she starts sending me links to jobs in the Chicago area. One that caught my eye was ‘Senior Network Security Engineer’ at Groupon. I haven’t been keeping up with the news or anything, I thought Groupon was still employing fewer than 1,000 people. I applied after skimming the post, also not really expecting to hear from them; They’re a tech darling after all, right? I’m sure tons of people are applying.

Skip head only 3 days and I’m already getting a call from a Groupon recruiter. He loves my background and thinks I’d be great. A few days later and I got an email from Google (zomg!) saying they want to talk to me as well. My mind beings to think fanciful thoughts about pitting two companies against each other in some sort of high stakes bidding war. I have a phone interview with the Google recruiter who doesn’t seem quite as excited about my background but she certainly wasn’t negative either.

I then have technical phone interviews. My phone interview with Groupon was rather informal and non technical in comparison to my previous experience with Amazon. The guy interviewing me had himself only joined the daily-deals site 4 months prior and spent at least half the interview telling me how he was barely keeping his head above water. He was also my potential supervisor and kept slipping in phrases like “We could get you… er, whoever we end up hiring, doing X extremely soon.” It seemed like they *really* liked me. Plans were made shortly thereafter to have me come out for an onsite interview.

All this time I’m doing lots of research on Groupon and I’ve discovered lots of things that had made me uneasy. The refused buyout from Google for $6 Billion, the overvalued stock, the fact that lots of businesses were discovering Groupon wasn’t actually helping them, not to mention the change in the culture after the IPO. Then there was the competition from Google and Amazon and the fact that Groupon wasn’t really a technology company and the fact that they employed almost 10,000 people world wide.

Before flying out to Groupon I had my Google technical interview over the phone in conjunction with a shared Google Doc. I’d been reading up on “Cracking the Coding Interview” and felt more confident since I’d be typing on a keyboard and not standing awkwardly in front of a white board. The interview was set for 2 o’clock but my interviewer didn’t call me until 2:08, those eight minutes felt like an eternity. He was on the Image Search team, in mountain view, so no one I would be working with, at least not immediately, in Chicago. He asked a few boiler plate questions and jumped right into the coding question. The question at first seemed easy enough, but of course nothing is that easy. I struggled, a lot. Thinking out loud is still super counter intuitive, but everything I read said to do so, otherwise they have nothing to judge you on. I wrote my answer in Python, right in the shared word processing document, so I never got a chance to try stuff, also counter intuitive. I noticed that once in a while he was selecting all of my code. I’d say he did it 3 or 4 times, and it wasn’t until the end that I realized he was *trying* my code on his computer, a luxury I wasn’t afforded. He would then ask me questions about lines that had syntactic errors, things I would have the answers to in literally a second had I been able to try them myself.

Eight minutes no longer seemed like an eternity compared to the 45 I spent floundering this one coding problem. I could tell he’d long made up his mind but still afforded me the opportunity to ask him a question or two.

I’d blown it, probably, but maybe not, but probably. So much for the dueling job offers… But there was still hope for getting to Chicago! A company of 10,000 people and a market worth of nearly 12 billion dollars isn’t going anywhere any time soon, right? Off to Palo Alto! Which isn’t Chicago, but that’s where the people most interested in interviewing me would be on the day the interview was scheduled.

I flew out to Palo Alto, interviewed and flew home all in the same day. Its not an experience I’d recommend to anyone, especially if you don’t have direct flight. I got to the Groupon building, found it to be rather modern and recently redone. The whole place had a very green theme to it, which reminded me a lot of my first real technical job at a place called Digital Green. I had arrived somewhat early but I was hoping my interviewers would want to utilize that time since I had to turn right back around in a few hours. They did not, however, and I waited in their lobby for something like 20 minutes. Once we got down to interviewing it was much like my phone interview, where we discussed how Groupon grew so fast that lots of things didn’t scale up with it, network security being one of them. I only met with two of the guys but they both seemed pretty excited about me and I actually felt better about it as a place to work.

The waiting game began.  I didn’t hear anything from Google for two weeks nor Groupon for three. This was sorta giving me hope on both fronts; maybe they were working on more interviews (Google) or hiring packages (Groupon).  But as time kept passing that hope started to wain.  The Google recruiter eventually got back to me in a voice mail, she sounded extremely sad but she informed me that they weren’t going to pursue hiring me.  So I did blow my technical interview…

I had sent follow up emails to Groupon no response.  I ended up going back to the recruiter who originally called me to get an answer.  Three weeks after being flown out I was finally told that, in spite of being super awesome, they weren’t gonna hire me.  They thought I’d get bored in the position, an assessment with which I didn’t entirely disagree, but he said they wanted to keep in touch in case a position opened that I would be a better fit for.  He then went on to assure me he wasn’t “just saying keep in touch” but that he “actually meant it,” and said a few more times just how awesome I am.

So that’s my story.  I’m back to having zero job prospects in Chicago, but that’s only because I haven’t applied anywhere else yet.  If you’re in Chicago and want to hire me, do contact me.

Amazon provided some closure on the matter of my interviewing. They politely said ‘No,’ via email, explaining that they didn’t think I was a good fit for the position.

Rejection aside, it was good practice. I can now say I’ve interviewed with a Fortune 100 company. Amazon is also one of the Big 5 tech companies entire books have been written to prepare interviewers for. I also can’t feel too bad as I’m in good company with those that Amazon has failed to offer employment.

I was contacted by my recruiter informing me that I had done well enough that they wanted to fly me out.  It was done in a form letter that began “Dear NAME,” which had me a little worried; what if all interviewees got some canned form letter by mistake.  After a quick email they confirmed that I really was to be flown out.  Setting up travel with them was painless, they worked with my schedule really well.

After this all seems lined up I get another email from my recruiter telling me that the guy who would be my boss if I were hired couldn’t be around during the day of my interview, but he really wanted to do a phone interview.  Again with the awkward panic, but it wasn’t as bad as with my first phone interview.  I don’t recall the specifics of the interview aside from any solution to any problem I proposed was met with “but now scale that up a million times.”  Of all my interviews though I probably got the most feedback from this ‘hiring manager.’  After that I was super confident my being flown out was nothing more than a formality; a chance for me to see the town, on their dime, to find out if I could see myself living there.

Fast forward to the day of the interview.  My morning started with not being able to acquire a ticket for the street car (exact change only and its $2.50, all I had was singles), but no one else was swiping passes or buying tickets, so I just acted like I knew what I was doing and got a free ride for 1 mile to the Amazon.com office building (sorry Seattle, I’ll make it up to you some day).  I got there a little too early and didn’t know who to ask for.  Its a good thing I’m not a goalie in the NHL, my confidence was eroding quickly with each faux pas.

Promptly at 10:15 am the gauntlet began.  Every interviewer asked questions about stuff I never claimed to be an expert in, or even know about.  Application level security.  Host level forensics.  None of these things are even on my resume.  My favorite was “What does communication mean to you?”  The interviewer followed up and gave that question some context, certainly I hadn’t thought to prepare for questions like this.  They all acted like stone-faced poker players too, I seriously had no idea if I was on the right track when answering a never ending stream of open ended questions or not.  Every interview felt like I would have gotten a C or worse had they been homework or tests.  Only the very last interviewer asked algorithms questions.  So I had to actually write code on the white board.  It was awful code, and like my phone interviews, it was sort of a progression of a problem, meaning I had orphaned code blocks with arrows into the main set of code.  Then he took a picture of it with his iPhone.  I was so embarrassed.

Halfway through the day I met with an HR person, different from my email contacts, who said I should hear something by Friday, and if not, I should ping her with an email sometime on Friday afternoon. ” Such efficiency,” I thought!

After getting out of there at 3:30 I felt like I’d totally blown it and would probably be banned from ever visiting the entire Seattle area ever again.  A friend of mine, who works for Microsoft and has done her fair share of interviewing pointed out that I really had no idea how I did, good or bad, and that I was just assuming the worst.  She’s right, I’m usually not a “glass is half full” sort of guy, but its only because I like seeing full glasses.

Fast forward to last Friday.  I hadn’t heard anything and I emailed the nice HR lady as instructed.  No response to this day.  This past Tuesday I attempted to call the aforementioned HR rep. and got forwarded to voice mail.  I left a message and I’ve yet to get a reply through that avenue as well.  Yesterday I attempted phone contact again with the same HR person and one of my email contacts.  Neither answered and the latter’s voice mail message stated that she never checked voice mail.  I emailed the latter rep after that, I’ve yet to get a reply that way either.

Its been a week since I was told I’d hear something and I’m not really sure what to think now.  My Microsoft friend finds this to be pretty appalling behavior for a company as large as Amazon to be treating me.  I would just like some closure on the matter, offer or rejection, neither would be entirely shocking at this point.

I recently applied for a position with Amazon.com.  It is a software developer position, doing security related stuff with EC2.  I didn’t even expect to hear back from them as my attempts with applying to Microsoft and Google haven’t yielded a single response yet.  They actually got back to me rather quickly and I had two phone interviews scheduled a few days after I was initially contacted.

It was during this period that panic started to set in.  I had never had to really apply for any of the jobs I’ve taken.  Landing my job with Digital Green involved brining some free pizzas to a LAN party hosted at their office.  My job at the lab started with a student internship, which did require a phone interview, but I don’t remember being that stressed about it.  There was an in person interview with management at the end of my first summer but I wore a white t-shirt and shorts and the questions were more along the lines of “so, you’re gonna finish school then come right back here, right?”  I had an interview with a company out East, but *they* wanted to hire me, so it was virtually no pressure.

I started to research the Amazon interview process and I wasn’t thrilled with what I read.  Data structures and Algorithms was all I kept reading.  I got an A in algorithms in grad school, but only through lots of studying and office hours and that was over 2 years ago now.  If I want to sort something I call the sort method and I’m done with it, I don’t really care how it works at this point.  Panic aside, I picked up a copy of Cracking the Coding Interview and started to read.  What I read actually cheered me up, I could come up with an approach to all the sample questions I’d come across.  I decided the two diplomas on my home office’s wall maybe did mean something.

The first interview came and I was nervous as hell.  The first few questions weren’t technically challenging at all: Why do you want to work for Amazon? Tell me a little about yourself.  My answers felt stupid, mostly because I hadn’t rehearsed, I was shooting from the hip.  Once he started asking technical questions I calmed down and got into it.  This first interview happened to be with a Security Engineer, not a developer, and not a single data structures or algorithms question was asked.   My resume mentions that I understand Internet protocols, such as DNS.  I’m really familiar with how DNS packets are structured and how to parse them.  He asked a questions about DNS, but how it works, not how the packets are formed.  I had to sorta dig deep and reassure myself I knew what I was talking about, but the confidence wasn’t in my voice at all.  Then came the encryption questions.  Again something I know but haven’t had to explain or think about in years.  Here I choked a little more.  Specifically on how signing works.  The interview ended after about 40 minutes, 5 minutes faster than he said it would last.

The next day I had an interview with a developer.  He asked me some softball questions to start.  How many bits in a byte?  Eventually he got to some data structures questions.  The first I’d actually seen in Cracking the Coding Interview, which I mentioned, but he had all these twists that made it a lot harder.  The second question seemed easy, but again he had all these refinements.  I almost didn’t get it until he mentioned hash tables, which I didn’t even realize I could use.  In the end I think that interview went as good if not better than the first.

Neither interviewer gave me any indication of how I really did.  They simply told me that the recruiter would get back to me within a week.  Even if I don’t get brought out for the 6 hours of white-board programming it was really good practice.  I know now that I know what I need to get a job at a big place like Amazon, I might just need to study a little more before any future phone interviews.

If you’re like me you might have an old Ubuntu 8.04 LTS server still running somewhere on your network, and you might think its time to drag that sever into the current decade.  Many of the docs on the web make the process out to be a simple matter of apt-get update/upgrade/install update-manager-core and then simply run do-release-upgrade.  I’m here to tell you that these instructions should in fact work, but you might still be seeing: “No new release found”

This is pretty frustrating and everything I found via Googling didn’t fix the issue.  The problem, it turns out, is the fact that we run an HTTP proxy where I work.  First I dismissed this as the problem since apt-get update and friends worked, but it turned out apt has its own conf file where a proxy can be specified, that apparently do-release-upgrade doesn’t consult.  So if you live behind an HTTP proxy and can’t get the upgrade from 8.04 to 10.04 to work, try setting the http_proxy environmental variable first, then run do-release upgrade.

I started writing this blog post right as Blackhat 2011 was wrapping up.

I’m half way through my second circuit of Las Vegas Computer Security Conferences, namely Blackhat and Defcon, and I have some thoughts on the subject.

This year was the first time I took one of the extremely over priced trainings.  It was two days and $2700.  At the end of it I received a certificate stating that I had completed 16 credit hours.  I found this rather funny; during my first semester in undergrad 11 years ago I took 16 credit hours all semester and it cost almost $1000 less than this 2 day course.  Based on my own experience and from what I coworkers have shared with me about their current and historic courses I’ve come to the following conclusion: Blackhat training (and really, any professional training), for anyone with half a brain, is simply an extremely expensive short-cut to very specific knowledge or experience.  For instance, the course I took this year was on wireless hacking, the first couple of labs went over stuff I had already successfully done one day in a coffee shop with a considerable amount of googling.  The second half I found to be a little more valuable but it became clear that this wasn’t outside of what I might have been able to figure out on my own, this simply told me the things I could have read but in a very straight forward manner.

The talks have been nearly a complete waste of time, the time alloted seems to be about twice what most speakers actually need and they end up doing everything then can to drag their talks out.  Most material ends up online and anything worth mentioning probably ends up on slashdot or reddit or something.

This is where I stopped and I didn’t have a chance to finish while in Las Vegas.

Attending DEFCON 19 was actually quite refreshing from my experience at Blackhat.  The overall vibe is more laid back and people actually seem interested in computer security at a tangible level that I can appreciate.  One thing I had forgotten from last year was that half of the Blackhat talks end up at DEFCON, which is sorta nice if one missed a given talk at the first conference but really goes to show just how little value there is in attending Blackhat at all.

The big thing DEFCON has going for it is just how immersive it is.  If talks don’t look interesting at Blackhat you can go to the vendor area and exchange your personal information with vendors for crappy swag or you can go back to your hotel room or casino or whatever.  At DEFCON if you don’t like the talks there are: competitions to take part in or observe, vendors selling stuff you not only know the price of up front, but that you can actually take home with you that very day, and puzzles strewn about the conference area to captivate you if all else fails.  And even if none of that really interests you if you blow off the afternoon to go sight-seeing in Vegas you don’t really feel guilty since the conference is only $150 for 3 days.

In all I’ve come to find Blackhat to be a waste of time and money and DEFCON to be a bargain and a more legitimate experience.

I’ve often found myself wondering if some set of tabular data, based on a given column, is sorted.  If it is, great, then I don’t have to (sort it)!  An example might be illustrative:

The above table isn’t sorted if you consider the first and third column, but it is sorted when you look at the second. Now lets say the above table was in a flat file, tab separated and you wanted to check it for sorted-ness. Sort allows you to specify columns, known as ‘keys’ in its nomenclature, with the -k flag. -k1,1 for instance could sort the data by the Name column. -k2,2 could be used to sort the file by the second column if it wasn’t already sorted.  The -c flag checks an input for sorted-ness and will complain if it isn’t. At this point you might think, much as I did, that -c -k2,2 would be ideal for this task. However sort will complain right away that the first two lines are out of order.

/tmp> sort -c -k2,2 test.input
sort: test.input:2: disorder: Charlie     17     0xE3D

You might be saying, “But, they’re not out of order, 17 and 17 are in order!”  And you’d be correct!  However, sort apparently performs a ‘last resort comparison’ to resolve ties, and to do this it apparently goes out of its way to look at the entire line, in spite of your use of the -k flag.  I’m not 100% on what the last resort comparison entails, but if you swap the X-Ray and Charlie line, then the file *is* considered sorted in spite of the third field being unsorted, so it must start from the left and work its way across.

But don’t fear, dear sweet Googler, for there is an answer!  Its the -s flag!  -s disables the last resort comparison and just moves on in the sorting.  I didn’t discover this until today, and part of the reason might be because the man page doesn’t really spell out how sort works deep down, nor that the -s flag might in fact be the solution to this problem.  Having found this I figured I’d share it with the world, because once in a great while someone Googles a problem I once had and I hope they find the answer useful.

When faced with a large number of unlabeled or mislabeled machines I fall back on a trick I’ve been using for a long time: eject.

‘eject’ as a command in most Linux distros (and probably most flavors of Unix) typically ejects the optical drive.  What better way to identify a machine than to have it basically stick its own hand out?  I admit that this method only works if you have some sort of access to a command prompt or desktop either remotely or via KVM, but it’s really handy when faced with a rack full of identical servers someone else installed and then didn’t bother to label.

The method also works in Windows by right clicking an optical drive in the explorer and selecting ‘Eject.’

Back during grad school this xkcd comic came out and inspired an app idea in a classmate of mine.  His idea was for an app that simply said “Don’t Panic.”  It sounded like something I could do pretty easily – after all, the Android Developer plug-in for Eclipse did all the work to get to the point of displaying text on screen.  One fateful Saturday afternoon, when I had every intention of working on school work, the school’s network was down and the project I was needing to work on required ssh access to a server there.  This was all the motivation I needed to make the “Don’t Panic” app happen.

I probably spent all afternoon making the app, looking for a suitable icon, registering a domain name, registering as an android developer, finding a free style sheet to use for the website, setting up Google Analytics and Ad Sense, etc.  The cost of becoming a developer was $25 bucks and the domain name was $32 for a 2 year registration, I’m really not sure why I needed 2 years but it seemed like a better deal at the time.

Once I had it up I was excited and afraid; I knew I had made a big pile of crap, but Google has a 24hr return policy so I didn’t feel terribly guilty about it.  I was in no way misleading in the description but I didn’t come out and say it was a “hello world” app either.  The sales figures can be seen in the table below.  I put $57 dollars into this project and I’ve almost gotten that much back out.

125 people have hit the ‘buy’ button in the market for this app, 77 kept it for more than 24 hours and 38 still have it installed.  It has somehow managed to keep a 4.5 star rating as well – I guess Hitchhiker’s fans get the joke.  I created a free version of the app dubbed “Don’t Panic Lite,” in hopes of drumming up interest in the pay version.  Instead of saying “Don’t Panic” in large, friendly letters, it says “Panic Less” in smaller but equally friendly letters.  Don’t Panic Lite has been downloaded by 2293 people, remains on 192 devices and has a 2 star rating.

For months, every time I saw someone with a G1 I wanted to tell them I wrote an app and that they should get it.  I managed to restrain myself every time, if only to save myself from the harsh criticisms I’d expect for such a trivial application.  The experience of having a published app has its perks, though.  For instance, I’m the most “experienced cell phone application developer” within my group at work now – mostly because we do nothing with embedded devices.

There are many more apps on the market now than there were in Feb of 2009 and I’m sure this sort of thing would be harder to do now than it was a year and a half ago.  But I get the feeling that useful or fun apps can pretty easily break even if not make money for a sole developer.

EDIT: I forgot to put my ad revenue in here, I’ve made 53 cents with Ad Sense on the domain name I put up for the app.

Back in late 2009 the decision makers in my group at work read the freely released plans for the BackBlaze Storage Pod.  Less than $8000 could get you 67TB of raw storage (45 1.5TB disks).  As noted in the blog post, this price point is far, far less than the average enterprise product of similar capacity.  The part list needed to make one of these machines was laid out entirely, down to the nylon motherboard stand-offs.  The authors of the blog post stressed that they had done lots of experimentation and that the given list of components worked really well together.

The aforementioned decision makers I work under decided we needed one of these pods, but that we could afford more than $8000.  Instead of the tested and recommended 1.5TB drives, we’d order some 2TB ones, but we’ll get the ‘Green’ models because they’ll generate less heat (even though the original design handled the head of non-green drives just fine) plus we’ll get all that extra storage!  It also sounded like the kind of project that should be given to a student because staff member time was far too valuable for such a project, a project with the goal of being (sort of) cheap.  The ordering of parts started in November, 2009.

Fast forward 9 months.  We still don’t have a working storage Pod.  The plan was to build two Pods.  One for run of the mill storage and one for… testing?  Or science, or something.  In fact I think the second one was going to be used in a high-performance storage project, which is an inappropriate use for the design, as performance isn’t even a real consideration.

Initially the top two most expensive staff members spent a day assembling one of the pods when the hardware first arrived.  Half way through the day 2 other staff members joined in when the first hurdle was encounter:  The thing wouldn’t even POST.  A top of the line ASUS mother board was ordered, one of the SuperComputer models.  A really great board for a lot of reasons, but the student that had been given this project ordered incompatible RAM.  The plan was to max out the machine with 32GB, but it only supports 24GB via 4GB sticks and everything we’d bought was 6GB.

Fast forward to a working computer with five 20-count cases of Seagate Barracuda LP 2TB drives unpacked and ready to go.  The original BackBlaze design calls for software RAID, which we were going to stick with.  Personally, I find that Linux software RAID is somewhere between witchcraft and back-ally dice games.  The disks we picked out didn’t help with this perception.  About the time we’d get a RAID set up configured just the way we liked it, drives would begin to disappear.  At first we thought it was the weird expander boards that make this project possible.  Then we suspected the cables, then the SATA cards, then the power to the expander boards, then the disappearing drives them selves.  After all, with 100 of anything one or two of them are going to be bad, right?

Fast forward even further, a hardware RAID card has been ordered for evaluation and a new server enclosure is on order.  There’s a new problem, our fancy hardware RAID controller doesn’t even detect the Green drives.  After much Googling I find a suggestion of adding a jumper to the drive to force it in to 1.5Gbps mode.  This actually works and the fancy RAID controller sees the drives, now where do I find 89 other jumpers?  After scrounging up a dozen or so, we start to conduct some tests.  An 8 drive RAID5 should be a good place to start.  Within a minuet of defining the array and watching it start to initialize the (un)expected happens: a drive disappears.  Just gone, without warning or explanation.  The RAID card is beeping, quite loudly, over this.

The drives suck, apparently.  More googling brings me to the support forums.  Plenty of people had a similar idea to our own: use cheap, huge drives in a RAID.  Most weren’t at the scale we were wanting to operate at, but even in a 4 drive configuration people were having troubles.  It turns out the drives really hate vibration and Seagate doesn’t recommend using more than 2 of these disks at any one time.  There is however a new firmware, and a CD that can be burned and booted from to update the drives.

I burn the CD and boot from it.  One problem: the firmware flashing software can’t see the drives through the fancy RAID controller.  No problem, I’ll just use some of the originally specified controllers.  Problem, the original controllers are sort of but not entirely compatible with the software used to update the drives.  The motherboard itself has SATA connectors but they’re inaccessible due to fan placement and I had no desire in fully dismantling the machine to get at them.

A desktop with a sole SATA connector was found and the firmware updater worked!  It took over a minuet to boot from the CD and flash the firmware.  1 down, 104 (before we ordered the full 100 we got 5 to evaluate) to go.  I flashed a dozen more and then went and built a test RAID6 setup with the fancy new RAID card.  I hammered the volume with a bunch of tests and none of them failed.  I’m letting it initialize all weekend but we’ve already set uptime records without having a drive disappear at this point.  With such promising results I went looking for a computer with more SATA ports, eventually finding one with 4.  I became a well oiiled machine for the next few hours and in total I flashed 103 of the 105 green disks we’ve bought on this endeavor.  1 of the 103 had already suffered a click-of-death death and the two I didn’t flash were in use elsewhere or hard to get to.  I’m actually fairly confident now that we’ll get a storage pod working come Monday.  But this sorta brings me back to the original blog post.  67TB for $8000 compared to enterprise hardware at hundreds of percent more in cost for the same price.  Even if we were building our pods at twice that in parts alone, what’s been spent in people-time has far out weighed any benefit for going this route.  Then again, last year I spent the better part of a day with another staff member and a field tech trying to get one of these expensive arrays to work, and they had worked on it for the week prior to that.  But maybe even that is a small price to pay though compared to this project.  Perhaps also, we should have just stuck with the precise formula BackBlaze developed, and maybe we could have had working storage within a week of getting the parts, instead of a better part of a year later.