What to do about nondeterministic tests?
(No Ratings Yet)
Loading ...We’ve all had these. Annoying tests, that work perfectly 85% of the time, but fail mysteriously when the moon is half full and someone is using decaff. We (myself most certainly included) usually tend to ignore it, rerun the test and pray the problem will just go away.
This is simply not the way! It’s an acceptable solution for tests that work 99% of the times, but as soon as a test starts failing sporadically, you’d better do one of these:
1. Analyze the test, understand the source of the indeterminacy, and make it deterministic! This is not always practical because it’s usually the most time-consuming solution.
2. If the test is really fast, you can consider rerunning it automatically on failure / introducing more sleep() to eliminate fuzinness. Not a real solution, but will get the test green some times.
3. If all else fails, just comment out the problematic part of the test or even the entire test. This is not what you really want, but it’s better than just leaving the test failing sporadically.
If you do nothing, you’ll quickly experience CI degradation – your test suite will become meaningless. People will no longer care about it, not even enough to fix tests that are easy – because “The CI is broken anyway”. An all green CI is a wonderful productivity tool, and it is reachable and worth the ROI.
Friend:
You probably talk about “Random” tests and not non-deterministic tests, since the letter cannot be achieved (maybe in quantum computing), this is just a mental experiment.
6/2/09, 18:17ripper234:
It’s a matter of definition. My definition here: a deterministic program emits the same output for a given input. This can certainly be achieve without quantum computers, but it requires a specific definition of input.
Try opening 100 threads and have them all print their sequence number 10000 times. I assure you that on most platforms the output will not be the same between different runs of the program.
6/2/09, 20:50Friend:
You have a misconception, you talk about probability distribution, that is, the probability for a given output, while non-deterministic is all about making every possible outcome rather than a distribution. This is not a matter of definition, since there is a huge difference.
7/2/09, 11:18Actually, there is no non-deterministic machine available, otherwise you cloud have prove that P=NP!.
ripper234:
I wasn’t talking about Turing determinism or in Turing terms.
I’m well aware of nondeterminstic Turing machines, but one can talk about a deterministic result, outcome or program for that matter, without defining acceptance models in the Turing while. The word determinisim was around way before Turing, so we must be careful to choose our context properly.
The important thing about nondeterministic turing machines is not that they make every outcome possible (because they don’t have to, the indeterminism is bounded). The real kicker is the acceptance model – the machine is defined to Accept if at least one possible run accepts. This is the problem in actually implementing nondeterministic Turing machines.
7/2/09, 18:32A Quantum Immortal » Blog Archive » Keep fighting the voodoo:
[…] developers dislike Voodoo, but sometimes all the good will can’t stop a dark spawn of evil. For example, one of our […]
19/3/09, 20:03yonatan:
hi! better late than never. many times problems in such tests arise from timers/synchronization problems.
26/7/09, 19:47another method for handling those unpredictable tests is via use of mock objects/substitutes for the timers and synchronizers.
you can write your custom timer to tick manually from your code and set the fake clock to any ol’ time you need.
and you can set your mock synchronizer to enforce a certain order of execution to match the scenario you are testing.
my two cents.