This blog has moved to Medium

Subscribe via email

Posts tagged ‘LINQ’

Playing around with PLINQ and IO-bound tasks

I recently downloaded Visual Studio 2010 beta, and took the chance to play with PLINQ. PLINQ, for those of you in the dark ages of .Net Framework 2.0, is parallel LINQ – an extension to the famous query language that makes it easy to write parallel code (essential to programming in the 21th century, in the age of the many-core).

A code sample, as usual, is the best demonstration:

public static int CountPrimes(IEnumerable<int> input)
    return input.AsParallel().Where(IsPrime).Count();
private static bool IsPrime(int n)
    for (int i = 2; i*i < n; ++i)
        if (n % i == 0)
            return false;
    return true;

This code sample, regardless of using an inefficient primality test, is fully parallel. PLINQ will utilize all your cores when running the above code, and I didn’t have to use any locks, queues, threadpools or any of the more complex tools of the trade. Just tell PLINQ “AsParallel()”, and it works.

I hit some gotcha when I tried to compare the parallel performance with the sequential one. Do you spot the problem in the following code?

public static void CountPrimesTest(IEnumerable<int> input)
    // parallel benchmark 
    var timer = new Stopwatch();
    Console.WriteLine("Counted primes in parallel took " + timer.Elapsed);
    // sequential benchmark
    timer = new Stopwatch();
    Console.WriteLine("Counted primes sequentially took " + timer.Elapsed);

This is all fine and dandy when the task at hand is CPU bound, but works pretty miserabbly when your task is IO bound, like downloading a bunch of web pages. Next, I simulated some IO-bound tasks (I used Sleep() to emulate IO – basically not using a lot of CPU for every task):

private static Random _random;
public static List<string> FindInterestingDomains(IEnumerable<string> urls)
    // select all the domains of the interesting URLs
    return urls.AsParallel().Where(SexFilter).
                Select(url => new Uri(url).Host).ToList();
public static bool SexFilter(string url)
    if (_random == null)
        _random = new Random();
    // simulate a download
    var html = "<html>" + _random.Next() + "</html>";
    return html.Contains("69");

Testing this with a list of 10 URLs took 5 seconds, meaning LINQ again spun only two cores, which is the number of cores on my machine. This really sucks for IO bound tasks, because most of the time the threads are idle, waiting on IO. Let’s see if we can speed this up:

// Use WithDegreeOfParallelism to specify the number of threads to run
return urls.AsParallel().WithDegreeOfParallelism(10).Where(SexFilter).
              Select(url => new Uri(url).Host).ToList();

This appeared not to work at first, because WithDegreeOfParallelism is just a recommendation or upper bound. You can ask PLINQ nicely to run with ten threads, but it will only allocate two if it so chooses. This is yet another example of C# being more magical than Java – compared to Java’s rich ExecutorService, PLINQ offers less fine grained control.

However, further testing revealed the damage is not so horrible. This is what happened when I put the above code in a while(true):

Tested 10 URLs in 00:00:05.0576333
Tested 10 URLs in 00:00:03.0018617
Tested 10 URLs in 00:00:03.0013939
Tested 10 URLs in 00:00:03.0013175
Tested 10 URLs in 00:00:04.0018983
Tested 10 URLs in 00:00:03.0024044
Tested 10 URLs in 00:00:01.0004407
Tested 10 URLs in 00:00:01.0007645
Tested 10 URLs in 00:00:01.0007280
Tested 10 URLs in 00:00:01.0003358
Tested 10 URLs in 00:00:01.0003347
Tested 10 URLs in 00:00:01.0002470

After some trial and error, PLINQ found that the optimal number of threads needed to run this task under its concurrency guidelines is ten. I imagine that if at some point in the future the optimal number of threads change, it will adapt.

If you found this interesting, wait till you read about DryadLINQ – it’s LINQ taken to the extreme, run over a cluster of computers.

Most annoying Euler problem yet

In Problem 54, you receive a set of poker hands, and are asked to count the number of times Player 1 won. No algorithmic or mathematical intelligence here, just coding. I advise skipping it in favor of more interesting problems.

Some say the Chinese word for “crisis” is composed of “danger” and opportunity (others do not). I took the time to play with LINQ and lambdas for the first time (in C# at least), and it was a great deal of fun – even at my home computer which is currently suffering from lack of Resharper.


Here’s a bit of LINQ goodness. The method I used (SPOILER SPOILER) to calculate the winner of two similar hands is a weighted sum. I summed all the best cards with a high weight, then added less important cards with lower weight (for example, 3 twos get more weight each than 2 kings in the same hand).

Once I found which value is the best in a given hand (2 in the above hand), I used this little one liner to count the number of cards equal to the maximal value.

_cards.Where(c => c.Value == bestValue).Count();

Terse is the new black.