HTTP/1.0 Is Dead
Update – it seems the pesky problem we were having for HTTP/1.1 is actually still relevant to us. The problem is we cannot force .NET framework to properly close a response stream within the HttpWebResponse – the thread just waits forever. Sucks, but it appears we have to use HTTP/1.0 for the time being, at least until we manage to hack it someway (understand how the class works internally and hack it away).
Working for the Diggers team in a search engine, a large chunk of our tasks is writing sophisticated web crawlers.
We were using HTTP/1.0 because we didn’t need any features in 1.0, and to circumvent a problem we thought we had with 1.1. On such a crawl mission, I discovered today that MSN’s site does not support HTTP/1.0.
After another check we found that the above problem does not affect us, because it is only relevant the Microsoft Compact Framework. But it’s still a rather shock to find a large site (even though it’s Microsoft) that utterly does not support HTTP/1.0.
Try it yourself – go to about:config in Firefox and set network.http.version to 1.0, and then surf to MSN. You’ll get a blank page. What actually happens is MSN assumes you’re using HTTP/1.1, and sends you a chunked response. Your web browser will not parse a chunked response, and rightly so.
(Needless to say that HTTP is supposed to be backward compatible).
