A Quantum Immortal

A devilish problem I’ve had in the past week is the the Windows “System” process was locking a folder or two. This was on a build agent, and was causing problems deleting that folder as part of the build. This was all happening while I was introducing a new build type to our collection – automated system tests using Selenium – and I initially thought some voodoo in Selenium was causing the problem.

LockHunter (a 64-bit Unlocker equivalent) wasn’t able to unlock the folder (though it did well on folders locked by other processes). The problem seemed to go away after a reboot of the build agent, but then came back to haunt me. I tried asking this on SuperUser (StackOverflow’s colorful twin), but to no avail.

Well, last night I finally found the problem – my own computer had an open explorer window on one of the folders in the build server. I believe this window persisted even after an agent reboot. Closing it solved the problem, and achieved world peace.

I just attended my first alt.net conference (some would call it unconference). The story is about a group of 40 people that came to talk about … whatever they decided to talk about. The conference is self-organizing, with no predetermined lectures or lecturers, and with one healthy rule – if you don’t feel you are learning or contributing at the discussion you are currently having, you have to get up and find another discussion.

Here are some of the talks I attended (here is a semi-readable list of all the talks):

Aspect Oriented Programming

Usages other than logging, AOP frameworks.

Links: Cthru, Post#, Wicca.

Mocking/Stubbing

Reiterate the basic paradigm, emphasize on TypeMock. They are considering a UI tool adding to Visual Studio to help create mocks – meant for people just starting with mocking. The intended usage is:

1. Write a test, without any mocking
2. The test will usually fail because some deep class is not configured correctly.
3. You will see the chain of calls that caused the exception, and be able to automatically generate a mock for any class in the chain.
4. Rinse & repeat until your test passes

High Scale & Distributed Caches

The discussion focused around what I’d call medium scale – 2-10 nodes that used shared caches like memcached & Azure.

Multithreading

There was a comparison of Microsoft CCR and Parallel Extensions. It seems people still think of parallelization as simply utilizing all your cores, when it’s much more than that. Some applications benefit from multithreading even on a single core machine (think web crawler).

One interesting link – PowerThreading library (see this video for a demonstration of Asynchronous Programming Model using PowerThreading).

Good developers dislike voodoo, but sometimes all the good will can’t stop a dark spawn of evil. For example, one of our build machines always bluescreens when running a certain system test. Why? Probably some hardware/driver failure, we’ll look into that.

In the meantime, let’s just ensure the build agent doesn’t get these nasty builds, shall we?

1. Build Configuration –> Agent Requirementse –> Add requirements for environment variable –> SHITTY_AGENT (condition = “does not exist”)
2. Go to the shitty agent and add a system variable named SHITTY_AGENT
3. Restart the build service:
   1. net stop “TeamCity Build Agent Service”
   2. net start “TeamCity Build Agent Service”

We’ve all had these. Annoying tests, that work perfectly 85% of the time, but fail mysteriously when the moon is half full and someone is using decaff. We (myself most certainly included) usually tend to ignore it, rerun the test and pray the problem will just go away.

This is simply not the way! It’s an acceptable solution for tests that work 99% of the times, but as soon as a test starts failing sporadically, you’d better do one of these:

1. Analyze the test, understand the source of the indeterminacy, and make it deterministic! This is not always practical because it’s usually the most time-consuming solution.
2. If the test is really fast, you can consider rerunning it automatically on failure / introducing more sleep() to eliminate fuzinness. Not a real solution, but will get the test green some times.
3. If all else fails, just comment out the problematic part of the test or even the entire test. This is not what you really want, but it’s better than just leaving the test failing sporadically.

If you do nothing, you’ll quickly experience CI degradation – your test suite will become meaningless. People will no longer care about it, not even enough to fix tests that are easy – because “The CI is broken anyway”. An all green CI is a wonderful productivity tool, and it is reachable and worth the ROI.

You know the classic workflow to fixing bugs?

1. Find bug
2. Write test
3. Make sure test fails
4. Fix bug
5. Makes sure test passes
6. Profit!

Sometimes, after you fix the bug, the test still fails. This means the test was flawed!

In this case, besides fixing the flawed test, you should go back to step 1 – undo the bugfix, write a better test or fix the test, make sure it fails with the original code, and only then fix it (the “lazy flow” is just to fix the test and watch it turn green, but then you don’t really know it tested what you wanted).

After having used it for a few weeks, I have this to say following my last post:

1. Great Support! We’ve had a couple of issues with Teamcity, some because our owns unit tests "maliciously" killed all java processes as a form of cleanup, and this killed the TeamCity Build Agents, and some other problems related to which version of the JVM was to be used. The TeamCity support staff and developers were extremely helpful and responsive, and aided us to get a stable Continuous Integration system up and running.
2. Enjoyable UI – the AJAX UI is usually very responsive, and rather feature-complete. It gives easy access to builds, projects, a sane way to manage settings and source control bindings, and a quick way to access the full log for every individual build (can be quite a few megs!) What I am missing is the one specific feature – I want the UI to clearly spell out the name of the person who broke the build (and speak it very loudly using text-to-speech, but that could be considered too esoteric to be included in the main build but rather implemented as a plugin).
3. Impressive code duplication finder – buggy at first, it now semi-stabilized. While we haven’t had the time yet to actually work with it and remove duplications from our code base, it’s just a matter of internal priorities at Delver. The power this gives a developer that wishes to improve his code base is huge.
4. VC integration – a bit less smooth than the other features. It’s mostly worked for me, but sometimes it didn’t. The Personal Build feature is very attractive, but I found one major problem using it: I ran a personal build that succeeded, but then it failed to automatically merge the code. I don’t know if anything can be done about this – I can GetLatest before running the build, but the content of the branch may still change, and unless the merge is automatic human will be needed to solve it. However, I think it’s a powerful tool that should work most of the time.

That’s it for now. All in all, great product – keep it up JetBrains.

Funny – just today I read that the great Donald Knuth, in a recent interview, dissed unit-testing:

“As to your real question, the idea of immediate compilation and “unit tests” appeals to me only rarely, when I’m feeling my way in a totally unknown environment and need feedback about what works and what doesn’t. Otherwise, lots of time is wasted on activities that I simply never need to perform or even think about. Nothing needs to be “mocked up.”

(In this interview he also talked against multi-core paradigm (which may be the only way to keep up with Moor’s law due to our CPUs getting hotter and hotter), but that’s a different story).

Why is this funny? Because just today at work I installed JetBrains’ TeamCity – a very fine unit test and automated build suite. Sadly I didn’t have time to finish configuring it today, but I already see the great benefit this will bring to all our development team. It goes beyond a simple automated build system (which is important enough, making sure no member of the team messes something up by accident, opening up new venues for brave code refactoring).

TeamCity aims to take all resource-intensive activities off the developer’s computer. Just some of its features are (I didn’t have the chance to test them, but I believe JetBrains – these are the guys that make Resharper after all):

• Code Analysis (finding code and style errors)
• Code Duplication – automatically find the copy-pastes I hate so much, with configurable granularity (java only, coming soon for .NET)
• Pre-commit Build – Suppose it is now 8PM, you’re about to leave for home, and unfortunately you have a rather large piece of uncommited (checked out) code. What do you do? You do not commit the code. Why? Because there’s a good chance it will screw up the build and you don’t want to leave the common integration area messy. To the rescue comes the Pre-commit Build feature – you can commit the code, have TeamCity catch your commit before it actually happens, test it automatically on a build agent (dedicated build machine), and then only if the code compiles and all tests pass it will check in the code for you. You do not have to wait for tests anymore! This frees up developer’s time for actual development, while preserving the high benefits of unit-testing (that Knuth doesn’t see, for some reason).
• I’m sure it has load of other features I haven’t dug up yet – among them the ability to run tests without any commit process “just for fun”, it’s ability to smoothly handle multiple Agents (each running a configurable number of builds concurrently), supporting multiple source controls, IDEs and development languages, ICQ/RSS/Jabber support, and last but not least a button “Run this test on the fastest available test machine”.

So far the systems I’ve used for unit-testing were:

1. Manually running unit-tests a couple of times, then forgetting about them 🙂
2. A patched-up system that automatically runs tests for you on checkin, with a build status indicator that Gal Golan in my development crew in the army wrote himself
3. CruiseControl.NET – an open source project that does something similar to the above, only slightly more configurable

As you can see by the length of this post, I’m excited to try on a professional solution to this problem for once 🙂 And for those of you that got this far, let me add that it’s free or charge – for up to 3 Build Agents and 20 developers. Huzza!

This concept has been revisited in many Extreme Programming articles so I will not blab about it much, but I do want to raise the point – mainly because it is still not a universal practice.

A key feature of a good programmer is courage. A “cowardly” programmer will see a non-critical problem or an ugly hack begging to be refactored and will shy away from it on the grounds that it his not his concern at the moment and it will probably break the world and introduce bugs to the system. A “courageous” programmer (backup up with properly written unit tests, of course), will refactor the ugliness away, thus incrementally beautifying his code.

This overall process, reiterated over the product lifetime, will either produce a convulated codebase with loops, noodles and baggage (for the “cowardly” programmer), or will result in a clean, easy to use and modular code for the “courageous” one.

While refactoring is a time consuming activity at times, it is well worth the momentarily increased coding time, because the overall coding (including maintenance!) time for him and his entire team will be decreased significantly.

The existence of well written unit tests is crucial to this, for without it you really can’t know if you introduce bugs or not. With unit tests, you get a significant “courage boost”, allowing you to do major changes that affect many files – for the better – because you are reasonably certain that any change will indeed be detected.

On the other feature of a good programmer – laziness – in another time.