Programming Tests - Advantages And Disadvantages
How does a programming test assessment work and what are the limitations of it.
The most important job a hiring process has to do is assess programming competence of a candidate. One of the most popular way to do that is by using online programming tests.
How does the assessment process work?
A programming test consists of solving one to three problems in 45 to 120 mins. This is accomplished by using online tools like hackerrank, codility, and others. Problems to be solved in the tests are algorithmic in nature. They provide a way to evaluate a candidate across four major areas - data structures, mathematics, algorithms, and programming1. The solution is evaluated automatically for correctness, as well for efficiency. Based on these automatic tests, solution is given a score, usually between 0 to 100.
If a candidate scores above a certain value, say 70, she is automatically selected for next step in the hiring process. If she scores below a certain value, say 50, she has failed the test and is eliminated from hiring process. If she scores between 50 to 70, then a manual review might be done to determine whether to reject her or select her for next step in the process2.
Things to be mindful of with programming tests
Be careful when picking questions, do not pick questions which are too hard or too easy. Account managers at tool providers, like codility, are quite knowledgeable about which problems are better suited and you should take their guidance seriously.
Even though tools allow any programming language to be used, be sure to inform candidate if there are certain languages that you would like them to use.
Time constraint is tricky. Some people thrive under time constraint and some lose their nerve. Generally better to have at least 2 or ideally 3 problems, with first being fairly easy. Having an easier problem to start with will help calm nerves, build confidence, and ensure that candidate has gotten used to the tool.
Do not keep changing the problems frequently. Makes it harder to benchmark talent sourcing.
Advantages of using programming tests
It is very easy to setup. Programming test providers integrate well with popular recruitment process software, and require minimal operational support.
Large bank of preset problems with varying levels of difficulties. You can use your own problems as well, within the limitations of the tool you are using.
Allows candidate to use a programming language of their choice.
Does all required checks such as correctness, efficiency of algorithm, and so on.
Provides detailed analytics to understand how the sourcing is working based on how many candidates pass the test.
Limitations of programming tests
Candidate writes a solution, but does not provide any tests. This is one of the biggest drawbacks, as you do not get to understand their knowledge of boundary values, edge cases, and whether they understood time and memory complexity requirements.
It is very easy to find solutions online and simply submit them. Tools do try to detect such cheating, but it is very hard to be accurate.
They do not test a candidate on real world scenarios.
It does not allow you test skills on everything else that goes with programming to deliver a production quality solution such as - packaging, source control, unit testing and code coverage, documentation.
Time pressure - some quality candidates do struggle under time constraint and will not perform well despite being well suited for the job.
A motivated candidate focused on passing these tests by taking lot of practice tests will end up getting high score, but may not always be well suited for job.
A 10x programmer has number of skills and traits, programming tests only focus on the bare minimum skill of being able to write correct and efficient code.
Programming tests work really well for entry and junior level programmer, but for senior engineers they do not provide enough information.
If you are using programming tests, it is important to understand the limitations and ensure rest of your hiring process is focused on evaluating candidate on skills which are not assessed by these tests.
How to score these problems can be weighted on complexity. So candidate must solve easiest problem with 90+ score, but may have 50 as acceptable score on very complex problems.