How to make a coding challenge to hire a new engineer?
How to make a coding challenge to hire a new engineer?
We're a very small team with little experience in hiring but got approval for a new engineer. Basically HR will look for people through the usual channels and I think we have a reasonably good job description. Unfortunately the coding challenge (a 30h+ take home) is atrociously difficult and doesn't really reflect what we do. On the other hand I think the false positive rate would be low. FWIW it's a Linux application and it might be difficult to only count on experience from the CV.
Any ideas how to build a good challenge from scratch and what time constraints are reasonable?
A 30h+ take home that doesn't even reflect what you all do is a waste of everyone's time. I'd think most qualified applicants are going to ghost you when they are tasked with that. You have to keep in mind you're not the only place they're applying. Are you sure you want the engineer who has time for a 30h+ coding challenge for a potential job, that might then make a competitive offer?
Good point, I fully agree with you. Didn't come up with it myself but it has been used for years already but I think HR also complained about it.
Actually I did the challenge a year ago and it was only circumstantial that I didn't decline. Yet, we need a good replacement for the challenge (or the whole interview process) because most of the workload is on me at the moment.
I interviewed for a position that I was comfortably qualified for. As soon as they mentioned a 3 hour whiteboard interview in person I politely hung up the zoom call.
On the flip side, I had a company give the best interview process of all time. They told you how many people were remaining in the rounds. The programming task was to implement a hugging face model as a FastAPI. There was also a short video interview that took 5 minutes if you had basic ML knowledge. Likely took 1-2 hours tops and it was actually fun.
On the other hand I think the false positive rate would be low.
Yeah, because any good engineer will noop the hell out of there and anyone left will likely not pass. Cannot have any false positives if you filter everyone out...
1 hour is more reasonable, 2 hour max I would say.
What is your goal for the coding challenge? Right now you're screening for the most desperate and willing to cheat with a 30hr busywork assignment.
Short answer: don't. Ask questions that demonstrate familiarity with concepts that you and your team encounter in a regular basis.
If you insist on a coding challenge, give them an old but difficult problem your team encountered in the job and assess how they would solve it.
I’ve only ever had one job ask me to do a homework style challenge and I thought it was weird and off putting.
I have a full time job, and a family, and a lot of other shit going on. If you give me homework I’m not going to bother with your company.
I feel like giving out homework in a job interview is a good way to filter out anyone who is busy being gainfully employed already, and leaves you with a weaker pool on average to draw from.
Sure there are some stinkers out there, but if you can’t suss that out in an interview that’s on you.
In Europe it's actually quite standard to give take home challenges (1-3h) and leetcode tasks are rare but becoming more common. Also many companies only do 1 or 2 rounds of interviews. (HR and technical) One could also argue that to prepare for leetcode style interviews much more time needs to be invested upfront, at least if it's not easy questions.
I would probably not want to avoid any challenge at all, but 1-2h seems reasonable to me. (live or take home)
In my experience, interview culture differs depending on the country. So, it would be better to provide some context around that.
I wouldn't spend more than 2-3 hours on a take home challenge. I would politely decline such a requirement and move on to contact other employers even if the potential employer offers paying my 30h+ hour work week preparing for the interview. In my opinion, a "take home" should be a conversation starter so that you can ask questions and try to understand the candidate's thinking. But, as I said, it may differ according to industry/country, etc.
Working culture is rather demanding and things tend to be quite ambigious, so to be honest the challenge reflects reality to some degree. But our team works with niche technology and therefore the pressure doesn't fully apply to our small'ish team. (Honestly, I wouldn't recommend the position to a not so experienced engineer or someone who doesn't know how to limit their working hours.)
You still run into the problem that generally experienced, skilled engineers are not likely to put up with a 30-hour coding challenge. I won't entertain anything over about 4 hours full stop, and it has to be a very compelling job to get me to spend more than 2. Among the people I know, the more skilled they are the less likely they are to be willing to do more than an hour or two of "homework", and some of the best people I know won't do that kind of thing at all because they don't have to. They can still get good jobs if they exclude every company that does a take-home challenge.
You're also biasing yourself against people who don't have 30 hours of free time -- anyone with caretaking responsibilities, anyone with health issues that means they need a lot of downtime after work, people whose current job requires a lot of overtime, etc. A lot of those people end up being the people tech already tends to have issues hiring, so it's just reinforcing the existing biases. Not great!
I'd look at timed problems on hackerrank/leetcode for inspiration, and aim for a 1-2 hour challenge. If there's a particular skill that you think is particularly important on the team, try to target that.
You will lose the best candidates with an onerous coding challenge.
Our process, which has been heavily influenced by debate on r/experiencedevs on reddit involves a short phone screen, a 30 MINUTE coding challenge, a tech interview consisting of pair programming, and a non-tech interview with management. Very light.
The coding challenge is a FILTER only. It's not to evaluate who to hire, but instead it's to filter who not to continue interviewing.
You learn a lot during pair programming in a short period of time, including personality and team fit. We let them drive and we just watch and discuss. The assignment is to fix a bug, and refactor the code the caused the bug.
There are a ton of good challenges on leetcode. I'd use a [virtual] whiteboard and have the candidate ask questions. When I'm hiring, I want engineers who can ask clarifying questions. I don't want someone who takes an incomplete set of requirements and goes off and builds what they think is the solution. I prefer they use pseudo code and not worry about syntax. Their IDE of choice would correct the minutia. I once had a candidate ask me if I thought design pattern X would be a good fit for the solution. Excellent question. That shows me they are thinking critically about the problem and open to peer input.
If you go with the programming puzzle type challenge, have a few in your repertoire. I've seen candidates totally blank on puzzle A, but rock puzzle B and vice-versa.
Things that have turned me off in interviews:
- ask me to write code in a specific language, then nit pick spacing, exact syntax, etc. They can fuck right off. I'm not working for your micromanaging ass.
- invite me to a 1 hour interview with HR, 1 hour interview with my potential boss, followed by a two hour technical interview. After about 2 hours, I'm done, and need a break. A 30h take home would be a hard pass.
Good luck on the hiring process!
Just a note by setting up a 30 hours home project you effectively removing "people with lives" from your hiring pool. People who can do a 30 hours either have a lot of freetime currently, or code after the job. And if you really want those people in the team then go ahead, but you are missing on 8 to 5 crowd and that is a very good and diverse talent pool. From my experience 8 to 5 minded people are very good in solving tasks in sustainable manner. They just don't have time to fuck with the system and doing effective "dont-call-me-at-night" solutions.
if you are doing a lot of interviews you need a common set of questions and measures and this take a lot of time and effort to setup.
Personally I would suggest to setup interview as a two parter first ask some theoretical questions and then ask to create a simple code with simple problem related to the questions. This helps to find out if people are really understand what they talking about. This again require a lot of thought to setup an to have small practical tasks relevant to the questions.
For example in most recent interview I asked candidate about algorithm complexity, data structures, garbage collection and then asked them create a simple dictionary to store a hierarchical structure. This helped to see if candidate knows what he is talking about and can use his knowledge in practise. I have seen a lot of people without good theoretical knowledge, but they create a code that is good and working despite their gaps and other way is also correct people have a good theoretical knowledge but fails to apply it in practice.
So figure out who you are searching for. Create an ideal checking solution for their skills and start combing the desert. There is no shortcuts in hiring, sadly.
Thanks for the comment. Yeah good point, I really miss people with the 8-5 mindset. At the moment there are too many people who just throw things over the wall, giving me nightmares once a month. (And users hate it) Already settled for something smaller but that definitely makes me feel better with my decision.
Time constraints should be 1 hour maximum and it should be a pair programming thing done over webcam. Why? I refuse to do "take home" assignments. They take far too long (even if it is only 30 minutes) and there's no guarantee that anyone will look at it. Don't waste your candidate's time. You're going to push good candidates away doing this.
You guys pay for that time or do you try to get it free like new code challenges?
I had to hire a whole sweet of devs in a pretty large organization. We didn’t do a code challenge, but curated some questions to ask in the interview to get a gauge on their knowledge. This worked well for us and all the devs we hired have been doing great! I think your dev team should be allowed to interview and ask technical questions that are relevant to the work they will be doing, the hire the ones you like from that.
This sounds interesting. Actually my job before was also in a large org and while there was a coding part, it was very basic. (String splitting, joining, a design/modelling task and pair programming on the actual code with the team lead) That team was quite large and everybody contributed their part.
Anyway in the current company the actual day-to-day challenges are figuring out things in the Linux ecosystem and also getting things done.
I am not gonna do a 30hr challenge. I barely have 30 hrs to spare in a month and I'm not gonna spend it for a chance of getting a position. 4 hours max, and not even something too difficult. As another commenter suggested, use it as something to discuss in an interview to get an idea of how they think and make decisions.
So what we do is, between the first and second interview we have new candidates recreate Twitter over the span of a week. We stress that they can put in as much time into it as you want. By no means does the site need to be functional at all by the second interview. If they spend 30 minutes thinking about it and are able to have a decent conversation, great! 30h assignment is a bit much and a programmer with that kind of time, is a bit of a red flag actually.
The point of the assignment, for me, is not to have some barrier of entry for a candidate. Instead, I use the assignment to:
- Have something to talk about
- See how good they are at structurally dissecting the problem
- Do they get bogged down in details
- In what order do they attack the problem
- Are able to effectively communicate some basic concepts around web-development
- Request sequences
- Authentication
- Database Schemas
- Asses their personality
- Do they want to try some new tech
- Do they polish
- How broad are their technical interests
- Do they do tests, did they host the project, did they do something interesting with UI
- How deep does their knowledge go
- did they use the right tools, do they have experience
- Have room for some hypotheticals
- How would you do it in a team
- what would you do with a month of time
When you look at it like that, the project doesn't really need to be that complicated. A candidate may be able to fake a challenge, but they can't fake an interview.
I used to let engineers submit code/a project of their choice for my team to review. We would then have a technical interview and it would be partly inspired by the code provided. Do something strange in your work? Well now you're going to have to explain it and justify why you didn't do it some othet way. It wasn't perfect, but I hired a bunch of devs that way.
You should recognize a good engineer by asking to create a class, a function, to design a simple application, and how to use
find -exec. That's less than half an hour and you are done. Longer than that, no true good senior will applyRecognizing class/functions/basic functions is not a criterion for a good engineer, it's a criterion for any developer. Not saying it's bad to check that... In fact yes, quiet a bit bad. Nowadays there is google to help you understand how to use find; you don't need to know by heart how it works, and maybe you are a good engineer that can find how to use that with
manor internet in a short time.Ask him also a relatively hard problem (easy to understand though). Not something to code particularly, but a problem to see how he process it, how he's thinking about it, what will he look for in the project, what questions will he ask you about the problem.
It will not take more than 10-15minutes, and you can easily pick who will be able to Engineer a solution (=> think about it, how it works, how it interacts) from people that can output code that works but need someone else to give them detailed instructions to be able to work.
This is exactly what I am saying, you don't need a 30+h coding challenge. Good questions to understand theoretical understanding, style, practical experience, software design, and few quick technical assessment tests to confirm they didn't lie, over few key basic tasks, those that one should know without googling, and that can show coding style and quality.
I believe 30+h of unpaid work are unfair just to prove me something I can understand with less effort on their side
I do a code review in the problem domain for my applicants. For instance, for my devops/sre interviews, I have some shell, docker, python, terraform, helm, and circleci files in a repo that all have some issues. For frontend dev, I do the same but with the frameworks we use (next/react, jest, cypress), and the same for backend (python, sam, flask, flyway, and pytest).
Some of these problems are logical, some are syntactic, and some are obvious poor practices, and some are esoteric issues that would only be obvious to an SME (and may not be obvious then, because it should become a discussion of the tradeoffs of one approach over another). The projects are not any bigger than they need to be to function for the interview, and they should not take more than an hour or two to read and assess for the caliber of candidates we seek.
I give candidates access to the repo a week ahead of time and ask for them to come to the interview with a list issues to discuss.
I loved our test at my last company.
Do an API exposing a fizzbuzz. Make it "prodable".
The definition of production ready is completely for the candidates to determine .
The subject is simple, doable in a few hours. What we were looking at was how code was organised, tests, usage of libs, ...