Item — HN Reader

The problem with work-sample testing (which is commonly administered as a take-home problem for the developer candidate to solve) is two-fold:

a) it discriminates against people who cannot spare 4+ hours of focused time on evenings/weekends to work on the problem. People with multiple jobs, single parents, etc.

b) in the age of AI it is no longer a reliable measure of someone's skill, for obvious reasons

Unlike Yegge, I haven't worked at FAANG, but the companies I have worked at all followed the same hiring practices and suffered from the same problems as he describes.

Provisional employment (or, if that's not possible, then well-paid internships) solve all of those issues. The candidate gets 3-6 months of stable employment, you as the employer get a large number of work-sample tests, and you can see how they use AI and how much.

5 comments

I think a very real problem is that these take home problems are often way more than 4 hours. And to that they often add the traditional 4-6 hour interview loop.

Provisional employment doesn’t work for most cases, though. It might attract people who have no job and it might attract people who have so much saved that they are okay with potentially being let go after 90 days. But I imagine the vast majority of the potential employment pool are not willing to quit their current job to accept a “maybe” job from another company.

Adding take-home problems to a traditional 4-6 hour interview loop is odious.

But the "way more than 4 hours" thing smuggles in a premise: that every candidate should be able to finish the challenge in the allotted time. But candidates with greater aptitude or conversance with the problem domain will complete work sample tests faster than candidates without, and selecting for those candidates is the point of hiring qualification.

It depends on the details of the work sample test.

If I ask you to write me a python function to convert OSGB easting/northing into WGS84 longitude/latitude the task has a very clearly defined scope. If you knock it out in a quarter of the allotted 4 hours, you've saved time. You can't use the remaining time to go further and demonstrate your mastery.

On the other hand if I ask you to write me a website for organising photos, there's no such thing as 'done' - no matter how good you are, after 4 hours you'll still be able to think of ways to make it faster, more beautiful, more featureful, more scalable, cheaper to operate, etc

Obviously, as a hiring manager I'll notice if you've spent 40 hours on the 4 hour task - but if you've spent 6 hours maybe I just think you're a fast worker with relevant experience and sharp tools. And my sense of how far you can get is calibrated by other prospective hires; if lots of people are spending 6 hours and claiming to have spent 4, my expectations will naturally be high.

Again, the premise is that you're exercising professional judgement. If you can't let a project go until it's perfect to a standard far past what's called for, that is itself signal. Either way: if the project budgets 4 hours, it is on you as a professional to stop at 4 hours.

If you run an interview process where candidates who take 6-8 hours and claim to have taken 4 hours score highest, those are the candidates you will hire.

All these objections rely on removing agency from the professionals applying for jobs. You look at the work sample. You use your professional judgement. You decide if it's reasonable to execute it to what you think a professional standard would be in the time allotted. You make a decision.

This isn't a college application.

I think you're answering a different objection than they're making. Their concern is that people will choose to spend 8 hours on your 4 hour problem but then tell you they only spent 4. Then you'll think they're a leet hacker because their solution is so awesome and they did it so fast.

Why would you design a hiring process that scores unprofessional people (by your own definition) higher than professional ones?

That's absurd thinking if putting in 6-8 hrs outta what everyone else is doing and what is needed to get you a job.

For all its flaws, part of the benefit of an interview is it's time bound and equal for everyone. Similar to a test.

Look, if you want to make people do work samples from an uncomfortable conference room at your office, be my guest. I am pretty confident I speak for the majority of candidates when I say that that my preference would strongly be for the ability to work on this stuff from wherever I want to.

I mean, that doesn't have to be how it works. You can have a both fixed amount of time, and the ability for a candidate to work in whatever environment they want.

This is theoretically true but it’s also rife with misaligned priorities. The people putting together these take home assignments have little incentive to ensure that they can be completed by a competent engineer in the allotted time. The engineers completing these assignments are definitely incentivized to underreport how long they spent on the assignment.

With AI coding this is also largely useless. These “build this thing in 4 hours” assignments come with a literal prepared prompt so that they can be churned out in 10 minutes.

We don't ask or check how long candidates take. You're a professional, we give you a challenge, you can decide (up front, 30 minutes in, whatever) whether you're likely to be able to finish in the budgeted time. Maybe you can't because you've got a lunch date and don't have the contiguous block, and want to do it in chunks. Fine by us.

Again, the underlying smuggled premise here is that candidates have to finish the work sample. No they don't. In fact, that's a strong sign it's not an effective work sample. A test that everybody passes isn't a real test; it's just a hazing ritual.

Just anecdotally, I can confirm that this method works great - it screened me out by showing me exactly what day to day work was like at your company, and that I was not nearly nerdy enough about specifically containerization to want to do it all day.

So you saved yourself/the team several possible hours of interviewing, and me quite a few hours - I think it took me about 10 to 15 minutes to see what you wanted in an engineer and that I was not it, and a total of 1 email which felt quite automated (whether it was or not) so there was a very low social cost as well.

You are assuming the assignment is reasonable and the candidate is lacking if they cannot complete in the expected time. And for all I know that’s entirely true for you. What I know is that I’ve seen assignments from others where the assignment scope was unreasonable for the allotted time. And for those teams, the filter becomes not so much “is this person capable” but “is this person willing to put up with our shit”, and the teams likely don’t even realize that’s what they’ve done (because they also “don't ask or check how long candidates take”).

> Again, the underlying smuggled premise here is that candidates have to finish the work sample. No they don't. In fact, that's a strong sign it's not an effective work sample. A test that everybody passes isn't a real test; it's just a hazing ritual.

Now this is an interesting take. Usually when people talk about these take home assignments, they talk about assessing the quality of the work. How good is the design? How is the coding? Is it efficient/elegant/whatever?

Here you take a much different approach, saying that the completion itself is the filter. If one person completes your assignment in the allotted 2 hours and another needs 12 but never tells you that, do you not care about that discrepancy?

We do assess all of those things. Again: you're a professional. We give you a work sample test. You look at it, and use your best professional judgement to decide if it is (a) reasonable and (b) doable in the budgeted time given your capabilities. If either is untrue, you don't do the work sample.

I'm having a real hard time seeing how this isn't strictly better than an interview, which, as the article (and basically everything written in the last year about interviews) points out, is basically a random function.

A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room.

The AI thing is an interesting problem, but a solvable one. We continue to hire resume-blind.

The question is whether companies would tolerate their own process if their employees did that recruiting process at a different company. They obviously do to some extent via plausible deniability; I have a 1 hour "doctors" appointment this afternoon, or I'm taking leave on Monday. Using it as cover to attend an interview.

Would this company permit an employee taking 3 months unpaid leave to provisional hire somewhere else and have free choice whether they stay or go at the end of it.

>> A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room.

Yes, standard interview loops also discriminate, and the more time they take, the more discriminatory they are. Any on-site requirements compound the issues.

Like Yegge says: provisional employment/internships solve all of these issues. You get the best of all worlds: stable employment for the candidate where they get paid a regular wage and aren't under a stressful interview setting, and lots and lots of work samples for you, the employer. It's not perfect. For example, it can be difficult to entrust the provisional employee/intern with anything impactful if you don't know whether they'll be employed at the end. But it is significantly better than the alternatives in most contexts.

Provisional employment does not work. It requires candidates to leave their jobs before they know whether they have a secure job with your firm. I concede that provisional hires are higher-signal than work sample testing (or rather: that they're the platonic ideal of work sample testing), but the entire problem of hiring qualification is to make decisions in the context of a candidate doing a job search.

>> Provisional employment does not work. It requires candidates to leave their jobs before they know whether they have a secure job with your firm.

It works. I’ve seen it in two different places.

At the second one, the fundamental realization I came to was that it is virtually no different than “regular” employment, where the new employee can get fired for not meeting expectations within an arbitrary time period after being hired. This can be months, or even weeks. From the perspective of the candidate, regular employment and provisional employment have roughly the same level of risk: in both cases they take a job where they might be let go at some point. The benefit of provisional employment is that they know how long they will be evaluated for and against whom. It turns out a lot of candidates do in fact like the all-cards-on-the-table approach and enjoy being given the opportunity to prove themselves on the job.

Everything will "work" in the sense of you'll eventually get somebody in the job. To me, it's self-evident that a process that disqualifies everybody unwilling to work for weeks speculatively isn't actually working, at a minimum because it DQs most (probably all) competitive candidates. But yes, your mileage may vary.

I’m curious though… these days I’ve to apply for n number of jobs to get 1.

Let’s say I get someone to do this provisional work. What about my existing job? I can’t just ignore it, and I can’t just causally say at my current company that I’m interviewing anywhere else.

And god forbid you like 2 potential gigs, and they both want you to do the campfire approach. Then my current job is really screwed!

Provisional employment only makes sense if you work remotely and have full benefits (esp. healthcare). Moving is costly.

A committed or indefinite term of employment is itself a benefit, so if you're going to work on a temporary basis you should expect contractor returns, not FTE comp; revise your baseline up 150%-200%.

Nobody does this, of course, but then provisional employment is a silly idea to begin with.

It has to be more than that. 150-200 is what your rate is _if you have a pipeline for more work_. If you don’t, you need to account for building the pipeline. So if a non-contract job wants me to work like a contractor, they have to account for the risk they’ve asked me to assume.

FWIW I came up with a provisional employment scheme that I thought would work at a job where I was responsible for hiring pipelines. It involved making an annual bonus (in an industry that was bonus driven) garaunteed and accelerating it so it was 50% sign on bonus and 50% delivered at 6 months. The company balked. Because provisional screening is a very good mechanism. Its must more expensive than hire and fire, at least on paper.

I feel like you could get around the AI bit by asking about components and what they do, rationale for decisions, etc. If someone can't speak to it, it should be a clear tell.

We hire entirely based on work sample testing, and there's a lot of stuff you can do to make it work in with AI-equipped candidates; I'm not prepared to write it up at the moment, but you start by recognizing that everybody is going to be using AI and designing the tests accordingly, and by relying on unassisted interactive challenges as a component of the process.

As long as you are talking to them face to face; over the phone they will use AI with speech recognition and parrot its response, erasing all signal. Then the interview becomes all about AI detection.

I've been hearing these kinds of things since 2014 (when I wrote a long post about the work sample process we had used at Matasano). I've been hiring continuously since 2008, so 18 years, and in that entire time I have never come close to hiring a scammer.

It might be a more salient concern now, in the era of AI agents, and we are much warier today than I was at Matasano, but generally I think this risk is more talked about than experienced.

I read that post back in the day and have remembered it ever since, and used it as a basis for the process at my current employment. Thank you!

(For those interested: https://sockpuppet.org/blog/2015/03/06/the-hiring-post/)

Ive had people show up at onsites that were clearly not the same person I screened over the phone earlier as far back as 2013 at google. It’s real. Just ask any recruiters from well known companies and they’ll tell you. It’s happening to more and more startups too now because it’s gotten easier

I believe you, but again: I haven't experienced anything like this, ever. One of the clearest results we got at Matasano adopting this approach was a drastic reduction in turnover, which is exactly the opposite of what you expect to see if people are scamming their way into jobs they can't do. That's held up (I don't have the before/after comparison, but I do have the industry-wide comparison) over the years afterwards. We hire these people, then we work with them, and they're amazing.

And coding interviews often bias against people who can’t spend days of time grinding on hackerrank or whatever to prepare for the interview. Provisional employment biases against people who currently have jobs.

There is no perfect interview process, so it’s important companies think about what they’re biasing for or against. A work sample gives a more realistic picture of someone’s ability than a coding interview, but less than provisional employment. But, it’s much easier for people to commit to finding the time for it than to spending 3-6 months in limbo.

Regarding AI and work samples, it seems like this problem is largely solved by having an in-depth conversation with the candidate about the assignment, no? They should be able to explain why they made the decisions they did, pros and cons of their approach, etc. If they can do that, and their solution is good, does it matter if it was LLM-assisted?

>The candidate gets 3-6 months of stable employment

To me, "stable" implies I don't even worry about having a full-time job for the next 3-5 years. Anything less isn't stable, unless this would be my second job immediately after the McDonald's burgerflipper one where I was scheduled for random 4 hour shifts seemingly designed to maximize personal inconvenience.

If I knew that my job would only last 6 months, I would have to immediately begin conducting a job search and prepping for it. I put my odds at finding another job before the 6 months is up at less than 60%. Even that number seems naively optimistic now that I've typed it out. Offering me 3-6 months of employment only sounds like a deal if I'm currently unemployed and rent was due yesterday.

We do all our work samples in person at a our offices as part of the in person interviews. Takes 2-3 hours, never been a problem so far.

If you are going to take a day off to do 3-4 in person interviews at a company then this slots in well.