Building an engineering internship program

ClickTime is a small 20 person, privately-held company that works on SaaS time and expense tracking. It’s not a startup, there are no high profile investors, and revenue growth, while consistent, is certainly not explosive. Intern salaries are significantly below what Facebook, Google, and other elite giants pay. There’s no fancy swag, no relo, no housing, no free food.

And yet, ClickTime’s internship program has consistently been able to land amazing students. A significant portion stayed on for multiple summers and converted to full-time. Others went on to work at Google, Microsoft, and Apple. One student even went on to cofound a YC company.

So, how did ClickTime do it?

First, let’s talk a bit about what interns care about. In no particular order, the list looks something like:

  1. Street cred/social proof/working at a company that looks nice and fancy on your resume
  2. Money/help with logistics (relo, housing)/perks
  3. Impact
  4. Getting to work on cool stuff/getting code into production
  5. Good mentorship and learning new things

I worked at ClickTime for over 4 years as an engineer and was very heavily involved in intern hiring and recruiting, so I hope I can share some insights into how we addressed these things.

In our case, we couldn’t do much with the first two items[1][2]. Instead, we focused on the rest of the stuff and became really, really awesome at it. Even if we couldn’t leverage a huge brand or pay people top dollar/stuff interns’ faces with free food, we could give them a great work culture, meaningful projects, and a sense of having walked out a better engineer. And that’s what people remember.

Here’s what we did.

Engineers bought in to how cool it was to have interns.

This buy-in probably mattered more than any other single factor. Before you do anything else, make sure that you have buy-in from engineering, or your program will be dead in the water. And I’m not talking about perfunctory buy-in where people pay lip service to how cool having interns would be. I’m talking about at least one influential engineer being really, really fucking hyped about the internship program. Why does this matter so much? The thing is, you can sell the level of impact interns have til you’re blue in the face, but until someone actually comes up with what projects they’re going to work on, who they’re going to be working alongside, and how much access they’ll have to higher-ups, it doesn’t matter. At the end of the day, you really need an engineer to be like, “Oh hey, here are some hack day projects we started working on but didn’t have time to finish. Perhaps an intern could work on some of these.” You need people who are excited about the program to be the ones out recruiting and doing internship interviews. Because their excitement is going to be contagious. Few things are more compelling than organic enthusiasm. The worst case scenario is managing to get top talent and then making them fix a bunch of obscure, all-over-the-place, low-priority bugs that won’t see the light of day until 5 releases from now.

The internship program itself was really well-designed.

Every summer, we’d bring in a marketing intern, a product management intern, and up to 3 engineering interns.  Over the course of the summer, all the interns would work on an intern project together, effectively forming a mini-company within ClickTime proper. After familiarizing themselves with the product, they’d find ways to make it better or think up brand new features that it was missing.  We’d also pitch some ideas to them, and ultimately, they’d pick something they were the most excited to build. The marketing intern would work on ways to engage users (and how to measure user engagement), ways to publicize the feature, and so on. The PM intern would come up with a spec. The engineering intern(s) would build it. By the end of the summer, the students invariably had a fully functional prototype, and many intern projects have ended up in production with only minor tweaks/additions (stuff like scalability, hooking the feature into the rest of the application, etc).  Past examples of intern projects included a way to photograph and upload expense receipts, a mobile app for expense tracking, an incentive module for employees to submit their time, and a way to easily export time and expense data in QuickBooks.

In addition to working on the intern project, engineering interns would be a bona fide addition to the ClickTime development team. They’d get tickets to work on, and their time and contributions would be budgeted into release planning. Historically, in addition to working on tickets and the intern project, our engineering interns would find some part of ClickTime they thought could be better and improve it. We did our best to foster an environment that was receptive to new ideas, and we’d support the interns in building stuff they thought up. Because we tended to hire very smart and driven students, at the end of summer, we’d be left with amazing work that we never tasked them with or saw coming.

Interns had dedicated mentors.

During the summer, interns would regularly meet with the dev lead (PM and marketing interns would meet with analogous leads) and with the CEO, who would regularly provide product-level feedback on iterations of the intern project. In addition, interns had an engineer who was their dedicated mentor. Sometimes being a mentor meant doing code reviews. Sometimes it meant acting as a sounding board. Sometimes it meant answering questions. And sometimes it meant just checking in and saying hey. One of the biggest complaints I heard during internship interviews when I asked students what they didn’t like about their past experiences was feeling like they got left alone all summer. Some students are super proactive, but many are shy and are afraid of looking stupid by asking a lot of questions. You want a go-to person whose job, in the eyes of the intern, is to work with them. To ensure that the mentorship aspect runs smoothly, a trick you can do is to assign more work to a mentoring engineer than he could complete on his own — this kind of thing encourages organic delegation and create a real dependence on the intern. Outside of the direct benefit to the intern, having engineers taking on interns is a good way for more junior people to get a taste of what it’s like to have minions (or, more politically correctly, start learning how to manage and delegate).

The CEO was out recruiting interns.

ClickTime’s CEO is a Cal alum, and he’d attend Cal engineering career fairs and stand there for hours to talk with students. I remember when I went to my first career fair with ClickTime, and I made a point to eavesdrop on the conversations he was having to see if I could pick up some pointers. It was actually quite remarkable. Alex made every student he spoke with feel like the most interesting man (or woman) in the world. At career fairs, when you have students lining up outside your booth, you start to feel the pressure to get through them quickly and keep the line moving, but Alex took what seemed like a long time to find out exactly what each student was interested in. Sometimes, students didn’t have an answer ready, and he would then either tease it out of them, or failing that, give them advice about how to figure out what they wanted to do. It was really quite amazing. Students would regularly walk away with different body language than they had approached the booth; many of them looked a few inches taller.

Everyone selling the program was able to talk about it in depth.

When recruiting, everyone representing the company could give specific project examples, describe to students exactly how their summer would look, and share anecdotes about previous summers.

We were open to hiring underclassmenif they were remarkable.

This is especially handy if you don’t have a huge brand to leverage. Sure, the conversion rate drops off a bit if you hire underclassmen instead of juniors, but if they have a great summer, they’ll likely come back, and at the very least, they’ll tell their friends. A huge reason why ClickTime’s internship program was able to land top talent was because of the reputation it established on the Cal campus over the years.

[1] Having a great engineering brand is probably the most important thing you can do to attract top talent (both interns & f/t people), but that doesn’t happen overnight and is out of the scope of this question.

[2] With respect to money and perks, we decided that because we couldn’t afford to pay for housing or relo, we would focus primarily on local candidates. If you do have some means, though, I’d say that personalized, thoughtful gestures matter a lot more than sweeping perks. If you’re not close to public transportation, you can buy your interns bikes. Or if you don’t have the means to secure intern housing, you can have someone in the company take on and own the task of helping your interns find summer sublets.

Note: This post was adapted from an answer I wrote on Quora.

How different is a B.S. in Computer Science from an M.S. when it comes to recruiting?

Note: This post can now be found in Forbes!

Recently, someone on Quora asked what the difference was between a BS and an MS in computer science from a recruiting perspective. My answer ended up with > 200 upvotes, so I’m reproducing it below:

In my experience, an MS degree has been one of the strongest indicators of poor technical interview performance.

There are multiple confounding factors at play here. In my analysis, I didn’t bother to separate out an MS degree that comes out of a combined BS/MS program (e.g. MIT’s prestigious MEng) from an MS degree from Bumfuckshitsville, Indiana. I also didn’t separate an MS in CS from an MS in some other, related field (e.g. “Information Systems Management”, more on that below). With perhaps the exception of the combined BS/MS programs from top schools (which tend to have stringent undergrad performance criteria), though, I’d be surprised if separating this stuff out made too much of a difference. Whereas MS degrees used to be a means for departments to begin vetting future PhD students, I believe that the purpose has, in some ways, shifted to be a cash cow for the university in question. Stanford’s admissions bar for an MS CS degree is significantly lower than their undergrad admissions bar. Schools like CMU do things like open a satellite Silicon Valley campus where they offer an MS in “Information Systems Management” (which to naive hiring managers sounds impressive until they interview a few people with this degree) and rake in all the moneys.

Part of the problem is that CS fundamentals instruction tends to happen in undergrad computer science courses. If your undergrad degree was in some other field, you can get through an MS in CS without ever taking an algorithms or data structures class. Or you could take a graduate-level algorithms class where the grading curve is going to be different because a good portion of your classmates have never done any programming either. A lot of MS courses I’ve seen on people’s resumes seem to involve taking some machine learning toolkit and using it out-of-the-box on some data set. Is this interesting? It can be. Does it show that you know how to make hard, unexpected design decisions or write code that’s not brittle? Probably not.

The other thing is, if you already have an undergrad CS degree, employers may wonder why you chose to go back to school rather than working (unless you’re going into academia, which is an entirely different animal and out of scope here). A possible line of reasoning might be, “Hey, if this person were really good, he’d be working already instead of waiting out the recession or whatever.”

So, let’s say that you weren’t in a position to get into a top undergraduate program. One tempting option is to try to get an MS from a top computer science school to legitimize yourself on paper. If you actually are passionate about programming, I would urge you not to do that[1]. Although you will look more legit on paper, many startups are catching on to how useless an MS degree can be. Until the market catches up, an MS will probably get you an extra $10K in your base salary. However, keep in mind that you’ve just taken 2 years out of your life and paid some amount of money (how much you pay depends on whether you can TA while studying). Instead of going back to school, take a few months to build something really cool. Teach yourself things. Take Udacity and Coursera classes and potentially use their career placement services. Work with a recruiter whose recommendation will help you get a first round interview even if you don’t have a pedigree. Once you start working, raises you get will quickly make the initial 10K boost pretty insignificant (especially given the opportunity cost and potential tuition expenses).

So, to answer the question: BS >>> MS (if they’re from the same school, assuming you have to choose one or the other)

[1] There are exceptions. If you don’t really love programming but want a coding gig for the stability/income, this could be a good option. Also, if you’re a foreign student looking to turn your MS into a work visa, this could be a good option as well (though you’ll want to distinguish yourself somehow from all the other foreign students trying to do the same thing).

Note: This post was adapted from an answer I wrote on Quora.

What computer science knowledge/concepts do software engineers use on a daily basis?

For as long as I can remember, there’s been a disconnect between kinds of stuff people ask in software engineering interviews and the day-to-day work that software engineers do. As a recruiter and a former engineer, this disconnect is particularly salient because when I evaluate candidates I have to ask myself two separate questions:

  • Will this person be able to pass a few rounds of coding interviews?
  • Is this person actually good?

It’s a bit funny that whether the person is good is somewhat immaterial if he isn’t going to make it past the CS fundamentals gauntlet.  That said, I think that the standard interview process isn’t the worst predictor of success on the job — people who are good at solving toy problems, have solid knowledge of various data structures, and for whom evaluating code efficiency is second nature also tend to be good at being engineers.  This is perhaps a classic correlation vs. causation problem — people who are good at being professional coders also tend to be good at being interviewed.  However, I’m not sure that they are good coders entirely because they are good interviewees.  In other words, I would guess that this approach has a pretty high false negative rate.  Unfortunately, I don’t have a good solution to this problem, and surely  lots of false negatives is better for a company than the same number of false positives (especially if the company has a good brand among engineers that affords it a revolving door of applicants).

Recently, someone on Quora asked about what CS concepts are actually used on the job (as opposed to what candidates are expected to bring to the table).  Bulat Bochkariov did an amazing job at attacking both this question and discussing coding interviews.  With his permission, I’ve reproduced his answer here.

Answer by Bulat Bochkariov:

The set of questions people ask is frustratingly disjoint from what’s important day-to-day. There are reasons, but I’ll try not to get sidetracked here.
Here’s a tiny list of things an engineer should be prepared for.

  • It’s 2pm. The release gets cut at noon tomorrow. (Or we could delay, but that would suck kind of a lot.) This piece of code is running way too slow for us to ship it with our names on it. How would you decide what to do? And then how would you do it? Remember that there’s still that other feature we were working on.
  • We need to do X. You know our stack and our toolchain. Research the options, then determine the next step and take it.
  • MySQL is seriously melting with this growth. We can’t keep doing all these joins.
  • There’s a critical bug in production.
  • We’re losing velocity. Our crufty code is keeping us from making changes quickly. What’s the root of the problem? What would really good abstractions look like? What are we trying to do? And how do we keep doing it while cleaning up this mess?
  • EventMachine is leaking more than normal.
  • Someone took out Git. There’s a crater.
  • Is it ok not to encrypt this? Like, what’s the worst that could happen. Right?
  • I think we’ve been hacked.
  • We’ve got an interview today. They even mentioned some security experience. What do we ask? How? What will it tell us?

For the most part, no one cares if you can make a red-black tree. It’s not something you’ll be doing. It’s just a proxy for a few things that do matter: your ability to solve new problems and an understanding of the content in a core CS curriculum—or whether you can write code that compiles and runs. But those are just the basics. They’re necessary, but they’re not sufficient.

What’s more important is your understanding: whether you can see the forest for the trees and find your way to where you’re going. You don’t think in JavaScript or Python but in problems and solutions, on multiple dimensions at once. You know the details are important but they aren’t the point—except when they are, and you can tell the difference.

You don’t do stuff that’s clearly bad; consider that a given. You don’t leave O(n^2) code turds on the floor or weird recursion time bombs that will blow the stack if n gets bigger than a thousand. But in a Maslow pyramid for programming, that stuff is on the bottom steps. It’s important to keep reaching for the top.

The most important question you can ask is “why.” What problem does this project solve, and what problems is it made of? Why should I consider Factor A over Consequence B? Why the hell is there a ServletBeanAdapter here? And keep in mind that sometimes it’s because an engineer took time to fully understand the problem, examined the upside, noted the risk to his or her own timetable, considered the security concerns, thought about maintainability, sketched out some more powerful solutions, decided not to waste the effort here, and made a quick decision that looked right when faced with finite time and lots of ambiguity. That’s what software engineering is about. You develop judgment and you learn from your mistakes.

And that’s what no one asks about.

Here’s where I should say a couple things. First: I’m a “new grad,” which means I’ve interviewed enough that I can’t count them on my fingers and toes, but I wouldn’t need to grow a lot of extra hands. That leaves me biased in a number of ways. Second: The context is important. If you work on machine learning, you’ll definitely need some statistics. For distributed systems, you’ll need data structures and concurrency. I don’t mean to say that core CS is not important. I’m talking here in generalities, so please don’t take them for specifics.

In my experience, what people ask is more like this.

  • FizzBuzz variants; easy looping questions. (Because it would be rude to check your pulse.)
  • Counting nodes or levels in some sort of tree. The goal is usually to see if you can think recursively or how you’ll try to hack the problem if you don’t. I love to harp on these, but honestly they make pretty good screens.
  • Basic questions from an algorithms class. Can you solve a searching problem if you aren’t told how? Can you mentally connect the properties of BFS with level-order printing in a binary tree?
  • Data structure questions with a fairly clear right answer. Given some data and a time or space constraint, can you process it a certain way? Can you make a stack with constant-time maxValue()? What structures would you hook together for an LRU cache? A lot of these are just fill-in-the-blank if you know the standard structures’ time and space costs, but that’s already showing that you know something worth knowing.
  • Sometimes they’re just random made-up problems that you probably won’t guess in advance. Maybe some weird string questions or a twist on a dynamic programming problem. Some companies might ask more mathy stuff like rolling averages or random number generation.

And of course there’s plenty more. There’s a bunch of different lists online that people study with. I just don’t think what’s in them is the point. The most important thing I have to share, strictly true or not, is that the skills you’ll need for interviews are not the skills you’ll need at work.

If you’re interviewing, brush up on your textbook knowledge. Practice your dynamic programming. Go code up a union-find in C. Worry less about what skills you really need and more about the ones you need to demonstrate. They’re different. The standard problems people test you on are proxies for the things they really want. They just have a funny and imperfect way of finding out.

View Answer on Quora

How long should my resume be?

Disclaimer: I do eng hiring for startups so my advice will be based on startup experience. Big, established organizations may do things differently.

I would try to keep your resume as concise as possible without sacrificing readability. Sometimes, this means you will go over one page, especially if you’ve been working for a few years. It’s not the end of the world. I would rather see a 2 page resume that looks like it was written by a functional, thinking human being than a one page resume that lacks any kind of content or soul. Try not to go over 2 pages, though, as your reader’s attention span is short. In other words, you should be aware that the people reading your resume are not going to read everything. Make it easy for someone on autopilot to internalize the most impressive things you’ve done, and always err on the side of conciseness. This means that you probably should:

  • Refrain from listing every project you’ve worked on. Pick the ones where you took some initiative, built something from scratch, or had major impact on the final product. At the end of the day, I’d rather see one impressive project than 5 projects where you upgraded some framework or followed detailed instructions to fix a subproblem of a subproblem of a subproblem.
  • Explain what you did in plain English. I can’t harp on this point enough. People outside of your company aren’t going to know what any number of internal acronyms or tools are. Think big picture. Ask yourself, “Why does what I worked on matter? How did I contribute to the overall mission of the organization? What is the context for what I did?” and so on. This doesn’t mean you shouldn’t go into technical details (you should), but it does mean that you shouldn’t use obscure terminology or the languages/tools/frameworks you used as a crutch.
  • Don’t keep repeating what tools/languages/frameworks you used. This wastes precious space, and in most cases, what you did trumps what you used to do it (unless you did something very specific like migrate from one language to another or fix a bug in some framework). Rather than going on about how you used Hibernate to do ORM or whatever (this is a particularly poor thing to go on about because you’re not adding any value — you are using a tool for its intended purpose!), just put all the things you used/know in one concise Skills section. For bonus points, delineate which things you’re very proficient in, which ones you know pretty well, and which ones you’ve used but would need a refresher on before diving in.

To illustrate these points, here are some project descriptions (taken from real resumes). My comments on the bad ones are italicized. I think the good ones speak for themselves.

Bad ones:

Designed software application including: data modeling, software architecture design, software- hardware integration, user interface design, and database management

I have no idea what the candidate did or how he contributed. This is super vague and uninspired.

Developed [product name], using C# in .NET framework, for marketing and allowing end-users to experience [another product name]

Again, I can’t tell what the candidate did. He’s hiding behind products/technologies to potentially mask the fact that he is either not too excited about his work or that he didn’t contribute much.

Good ones:

Created and launched a service that collects product opinions and recommendations from Twitter. The service finds related tweets, removes spam, analyzes sentiment and creates a structured database of everything that was said about particular products [link to demo]. The service is exposed as a consumer website and as widgets that can be embedded in online retail websites.

Evaluated and identified [OS name] network stack performance bottleneck in latency, system jitter, per-packet processing overhead, and scalability of different network IO models through various system measurement and profiling techniques

Note: This post was adapted from an answer I wrote on Quora.

What are the characteristics of great engineering recruiters?

A great technical recruiter should be able to do 2 things: (1) present engineers with companies that fit their criteria and (2) present companies with engineers that meet their standards and are likely to accept an offer, should one be made.

Before I come up with a list of bullet points (though you can skip to the TL;DR section below for bullets), I'd like to pose the question of why an engineer or a company would want to work with a 3rd party recruiter. I'll then go on to explain what value a good recruiter would add.

From a candidate's perspective

As an engineer, what's the point of even working with a recruiter when you can just apply to companies directly? In short, a recruiter should be able to help you flesh out what you want in your next job, come up with a short list of positions, deal with interview logistics/minutiae so you don't have to, advocate for you throughout the process, and help you understand your offer.

People want all sorts of things from their jobs, though sometimes it takes some time to really distill what someone's requirements are. Sometimes it's more money. Sometimes it's more impact, a specific type of challenge/problem to solve, or a specific kind of team structure. Sometimes someone wants to change over from an enterprise product to a user-facing one. Sometimes it's a matter of entering a new industry. It's the recruiter's job to know which questions to ask and figure out what's going to be most important to a candidate. In figuring this stuff out, I'm of the opinion that having industry engineering experience is going to be invaluable (how else would you really understand what it means when someone tells you they are tired of monolithic code or want to work somewhere that is more receptive to using 3rd party tools than their current gig?).

Once that's figured out, a good recruiter should help you go beyond your personal network and a sea of badly written job ads to find the job that will make you the most happy. To do this, a recruiter should have a pretty good mental model of what companies are out there, what they do, how much traction they have, what projects are currently being worked on, what the culture is like, and what compensation looks like, and then use that info to handpick a short list of jobs that would be perfect for you.

Sometimes this process takes several iterations, and it's the recruiter's job to elicit useful feedback and go back to the drawing board as many times as necessary to get things right. If a candidate says that he wants one thing, and when presented with that specific thing, he doesn't want it, that doesn't mean the candidate is lying or not being helpful. It means that the recruiter didn't ask the right questions.

Once a list of companies/jobs is in place, the recruiter should continue to be useful by abstracting away some of the interview scheduling back and forth. In essence, the recruiter should tell you to put on pants and to show up to interviews at some specific place and time. After each interview, the recruiter should be advocating for you to figure out what next steps are and make sure that you hear back and hopefully get any relevant feedback in a timely fashion.

Once you get to offer stage, your recruiter should be on hand to provide you with any non-confidential data about salary and industry trends and help you navigate the equity component of your offer.

From a company's perspective

As a company, what's the point of working with an external recruiter rather than doing all your outbound sourcing yourself? Chances are, if you're a startup, you're not going to have a huge recruiting department. You may not even have a dedicated in-house recruiter (though you probably should). Regardless, there's more to recruiting than just sourcing candidates. You have to deal with the candidate pipeline (screening people, making sure subsequent interviews happen in a timely fashion, and make sure that candidates are moved through the process, made offers to, and rejected). You have to deal with placing job ads, campus recruiting, figuring out ways to promote the company's engineering brand (meetups, tech talks, hackathons, wrangling people to write blog posts), incentivizing internal referrals… the list goes on. Having a bit of help getting candidates in the door is not the worst thing in the world.

Let's say that you now have some agencies working for you to find engineers. There are 2 pain points I've encountered as an in-house recruiter working with 3rd party agencies: influx of either low-quality candidates or of high-quality candidates that have a low chance of accepting the offer.

Consistently sending high-quality candidates to companies isn't easy. There is definitely some correlation between a candidate's pedigree and technical ability, but in my experience, it's not as strong as one might like. Therefore, getting an idea of a candidate's ability before pitching him to a company is invaluable. To be able to screen a candidate thoroughly, a technical background is going to be necessary, no two ways about it. Even with the advent of a single screen, it's pretty hard to tell who's good. It isn't too hard to tell who's bad, though, and ideally a recruiter would be able to filter out the poor and mediocre candidates, leaving the fine-grained filtering of the decent from the great to the companies themselves. Even if a recruiter isn't technical, he can still filter out some amount of poor candidates by asking them to describe what they worked on in a way that a non-technical person can understand. Doing this well is non-trivial and, in my opinion, tends to correspond to future workplace success more strongly than pedigree. Regardless of how the filtering is done, a great recruiter is going to find a viable way to do it.

Let's say that a recruiter is working with a candidate who is clearly strong technically. At this point, the recruiter has to decide where to disseminate the candidate's info. A great recruiter will take into consideration what the candidate wants (see portion above about the candidate's perspective) and will only pitch him to companies who match what he's interested in. Doing this is non-trivial because, in addition to understanding what the candidate wants, the recruiter must also understand the companies' business models, cultures, be familiar with their tech stack, and so forth. Unfortunately, because effectively matching candidates to companies really is non-trivial and requires deep, specialized knowledge, many agencies don't do this and turn the recruiting process into a giant feeding frenzy. The same candidate's info can get sent out to any number of companies, and for each company, the odds of closing said candidate drop off. This approach is particularly poor from the company's perspective because interviewing candidates is expensive, both in terms of engineering time and logistical expenses (such as flying people in).

As with candidates, it often takes several iterations to nail down exactly what the company is looking for. Sometimes, there's a disconnect between what a company claims to look for in a candidate and what candidates are actually successful. To mitigate this, it's always helpful to get an idea of the current eng team demographics. In general, it's also good to keep a frank, open stream of communication in both directions (company providing feedback to the recruiter and the recruiter potentially sharing insights about inconsistencies between what the company advertises to want and what it wants as well as any industry salary/equity trends that might be helpful).


Great tech recruiters:

  • grok engineers' and companies' criteria
  • are familiar with salary and equity ranges at startups in different stages and for different positions and understand/are able to offer meaningful advice about these things to both companies and candidates
  • are able optimize on candidates' and companies' constraints within a preexisting, broad network of companies and engineers
  • are able to filter candidates' technical ability to some extent (separating bad from decent is much easier than separating decent from great)
  • are able to elicit useful feedback from both candidates and companies
  • care about people/take pride in doing their job well so that when it's tempting to just play a numbers game, they don't

Note: This post was adapted from an answer I wrote on Quora.