[author here]
Hi all - I hope you find this useful!
If you have questions or want to know more I’m hosting a live stream Q&A today on Twitter at 1 PM Pacific. Tune in and bring your interviewing and hiring questions! I'm @startupandrew on Twitter.
back
2 comments
Your claim that a take-home test is "lower stress" is undercut by the fact that you time your interviewees on their homework. Why do you think it is important to gauge how fast a person might be able to solve your toy problem, and why do you expect your candidates to dedicate 6-12 hours (unpaid) of continuous time to your company before you've even met them in person in most cases? Did you consider that this might inherently discriminate against those with significant time constraints in addition to their current job, biasing your results to new grads and people without hobbies and/or families?
How does an employeee having hobbies and a family help with the survival of the company?
It seems that hiring people with a life outside of work could be in the active disinterest of squeezing the most work for the least pay out of the human resources that corporations feed on.
So true. If you can’t code for 6 hours straight without ignoring the fact that a job change can have an incredible impact on all your life goals and personal relationships, why are you even a coder bro?
This process is so weak, when I heard about it in 2013 I was like...why would I ever want to work at firebase?
The alternative to a take-home test is on-site interviews with white board problems. These also take (unpaid) time and are timed (you have until the end of the interview). Doing this part of the interview at-home rather than in-person gives people more schedule flexibility, not less.
The total time spent is higher than most interview processes I think, which can be a struggle for some folks. But I do think it's the right tradeoff.
Regarding why time taken is an important data point -- we found a strong correlation between people who were able to get good solutions working quickly and people who were able to effectively defend their solutions during the "GoldMine review" in their on-site interview.
> The alternative to a take-home test is on-site interviews with white board problems.
That's not the only alternative though. Why not use a more humane interview process involving conversation about previous work, ideas about the future, strengths and weaknesses?
> The total time spent is higher than most interview processes I think, which can be a struggle for some folks. But I do think it's the right tradeoff.
I don't agree but I'm glad you thought about it. I personally think making your interview process more inclusive allows you to get better candidates, as opposed to structuring the process in such a way to exclude those that have different priorities and life choices.
> Regarding why time taken is an important data point -- we found a strong correlation between people who were able to get good solutions working quickly and people who were able to effectively defend their solutions during the "GoldMine review" in their on-site interview.
Correlation doesn't equal causation though. I also want to point out that if you are expecting someone to "defend" themselves, that means you are on the offensive. Interviews don't need to be adversarial.
Edit: I do want to put in one good thing I got from your article to offset my rather negative responses. You mention that even if you reject a candidate you gave them the full code review and feedback about why you rejected them. The culture of secrecy surrounding rejections is something that needs to die, and I'm glad that although I think your process as a whole is unfair, you took the step to change that toxic part of our industry's interview culture.
The problem with interviews that rely on discussions of experience alone is that they aren't effective.
It's easy for candidates to exaggerate or misrepresent their accomplishments. This is especially true when they worked on large team projects. For example, if they say "I was part of the team that built Google Search", that could be a really impressive thing if they played a key leadership or technical role, or it could be a really unimpressive thing if they hung out while others did the heavy lifting, and it's very hard to tell the difference.
Yeah this is the common excuse for torturous technical interviews but it just doesn't fly with me. What is the end game of the interviewer who is lying? They'll get found out when asked to deliver and lose their job eventually. This is not without cost to the employer, but it gets harder to explain so much turnover in your resume as you accumulate early exits.
In addition, there are levels of technical chats that become very difficult to fake, further reducing the pool of lying interviewees people claim to be defending against.
To be clear, at Firebase, we held these kinds of discussions in addition to the technical assessment. A portion of my section of the interview process was usually to do a deep dive on a recent interesting project that the candidate had worked on, either sourced from their resume or just by asking. You're right that you can often tell their level of involvement by seeing how quickly the well of details dries up.
There are tradeoffs, but I found it to be an overall net positive in addition to the technical assessment. Doing this effectively requires that the interviewer have a pretty broad exposure to lots of different technology, otherwise you can't distinguish between someone BSing their role vs something you just don't know a lot about / can't ask intelligent questions about. It's also incredibly subjective, which would make me hesitant to rely on it without more objective criteria.
> What is the end game of the interviewer who is lying? They'll get found out when asked to deliver and lose their job eventually. This is not without cost to the employer, but it gets harder to explain so much turnover in your resume as you accumulate early exits.
That's a pretty idealistic view of people. Most people exaggerate on their resume. The most reliable tool (at the moment) to cut through and establish some common metrics is the technical test. How do you compare technical "chats" between individuals? Duration of call? Gut feel? How do you ensure that the person on the phone has the breadth and depth of experience to thoroughly discuss any and all projects on the phone with the interviewee? Is it the same person on all of these calls? It can't be, so how would you compare between two interviewers' subjective feedback?
> This is not without cost to the employer...
This is a HUGE cost, especially at a startup. A mis-hire at a startup can sink the entire ship.
I do think that your approach has merit, and I personally dislike technical interviewing, but for a startup the OP's approach is, IMO, the best tradeoff.
> They'll get found out when asked to deliver and lose their job eventually.
It's the "eventually" part that's problematic, since it can be long, uncomfortable, and expensive to the company.
I read that comment differently. The idea is to look at the candidate's work history, and if there's a large number of short stints you can use that as a red flag that previous employers found them lacking.
> Why not use a more humane interview process involving conversation about previous work, ideas about the future, strengths and weaknesses?
Because I'm hiring someone to build technology, not to schmooze about it. I need proof of ability.
Do you hire people to write code like in the example, plotting mine routes? These things don't test for relevant technical ability, they test for compliance, they test for algorithmic knowledge, and they test for being a young person without technically being age discrimination.
From the article: "In retrospect, I think our test was a bit longer and more difficult than ideal. If I were to do this over again, I would design the test to take about 4 hours."
Seems reasonable to me.
But the test time is actually unbounded, although of course they're hovering over you with a stopwatch.
Which means the candidate will be under incredible stress still, no matter what the supposed target time is.
Was there ever a time the testing and quizzing was wrong? How did you know?
For example, “Later that candidate did X, which would have been great for Firebase.”
How often did you take subordinates that weren’t fresh college graduates or former/current FAANG engineers?
Yes, we definitely had false negatives. It's always hard to be sure if you made a mistake though -- someone might be super successful at another company but not work out well at yours. I'm sure we made mistakes, though I don't have stats on this.
We definitely hired non-new-grad non-FAANG engineers. In fact, new grads and FAANG alums were a minority for us.
> someone might be super successful at another company but not work out well at yours
I don't actually buy this. Environment makes a difference, sure, but if someone goes off and succeeds technically[1] at a another company in a big way they would have succeeded at yours as well. If that person can't succeed with you, then there is something fundamentally wrong with your organization. My experience, though, is that such organizations have much lower hiring bars.
[1] I say "technically" because it's quite possible that the person is not very good technically, but landed at a place where that didn't matter and some combination of non-technical skills led to their success.