I’m sure you agree with me that errors which are hard replicate are difficult to resolve until you figure out what is going on. This was one of those worth blogging about it.
The problem:
To cut a long story short, some time ago I set up a website to receive any errors that occur on the live site by email. After passing some rigorous testing procedures I was surprised that I was receiving many Unhandled HTTP Exceptions by email with Error Message being : Exception of type ‘System.Web.HttpUnhandledException’ was thrown. I started scratching my head because on the development environment the site was working well and I double checked configuration settings and ensured that they were fine. Anyways, this error wasn’t helpful so I decided to include the stacktrace of the inner exception.
The error message of the inner exception was something along these lines: Exception of typeSystem.Web.HttpUnhandledException’ was thrown. Cannot use a leading .. to exit above the top directory. ‘
I figured out from the User-Agent http header that this exception was being thrown when a google or yahoo slurp bot tried to crawl the site. I confirmed this by testing pages using a googlebot spoofer and boom, I was getting an unhandled http exception.
The solution:
Believe it or not, the jist of the problem was Url Rewriting. Instead of improving search engine ranking, Url Rewriting was preventing crawlers from indexing any pages. ASP.NET 2.0 uses an adaptive rendering pattern to render html depending on user agents. When a bot visits the site, System.Web.UI.Html32TextWriter gets used to render the html and apparently this throws an exception when System.Web.Util.UrlPath.ReduceVirtualPath() gets called internally by the .net framework when using Url Rewriting. To workaround this problem I set up .browser files for major search engine bots like googlebot and Yahoo! Slurp to force them to use System.Web.UI.HtmlTextWriter. Alternatively, you can have a base page override the CreateHtmlTextWriter as following:
protected override HtmlTextWriter CreateHtmlTextWriter(TextWriter tw)
{
return new XhtmlTextWriter(base.CreateHtmlTextWriter(tw));
}
Thankfully this has solved the mystery exceptions I was receiving by email! The moral of the story is to always send an email when an unhandled exception gets thrown on the live environment. Had I not did this, bots would have never indexed the site.
