I just spent the last week banging my head against a wall. I came away with no physical bruises, though my metaphysical cranium hurts a bit.
It is not the first time this has happened to me, and I am sure it won’t be the last. And I’m willing to bet that something similar has happened to you, Dear Reader.
The issue cropped up with our new server environment. We added a second web server and a load balancer to split traffic between the two machines. It will be a great environment for both performance and for redundancy.
We ran into a strange issue after running with the load balancer for a few days. It seemed that if we made a change to the website, the two servers got out of sync and started throwing errors. The error mentioned that it was looking for a particular dynamically built assembly file.
The issue happened quite frequently in our old environment, and typically it just took a refresh of the page to get it running perfectly until the next update. That didn’t work in the new environment. We troubleshooted it for a while, but we finally made the decision to stop load balancing and go back to one server until we can figure out the issue. As soon as we went back to a single server, the issue cleared up.
To me, that meant it was an issue with having more than one server in the mix. I began to work with our hosting partner to find a fix while I did a lot of googling for answers. After a week of trying a lot of different things, including making our site work as a precompiled site instead of dynamically compiled, we were no closer to the answer. I was really confident that precompiling the site would fix it and when it did not work, I was out of ideas.
Enter a second pair of eyes. Eric Patterson had just come free from a different project, so he helped me look deeper into the issue. Armed with the knowledge of all I had tried in the past week, he took a closer look at the Stack Trace for the error we had. Stuck deep down in the stack trace, he kept seeing this:
DBauer.Web.UI.WebControls.DynamicControlsPlaceholder.RestoreChildStructure
That is a control that we use to put our “Page Parts” onto the site in the correct order. For me, that was not a clue because I knew it was used on every page. For Eric, who had no such mental constraints, it was a place to start looking.
He quickly found an article that seemed to fit perfectly:
1) Error only in a web farm (which is what we have after introducing the second server) 2) Error with finding the temporary Assembly name
3) Using the DBauer Dynamic Controls Placeholder.
That article has a pretty easy fix to the issue that will cause the Dynamic Controls Placeholder to use a relative path to the User Control and not a fixed path. If you need the source code, it can be found here. I made the change to the source code, compiled it, and added it to our website.
The error disappeared!
There’s something to be said for walking through a problem and eliminating possibilities. That needs to be done. But when you find the frequency and power of your head banging into that wall increasing, go grab someone else to help. Their fresh perspective can often find you the answer quickly.
Aaron Brander is the VP of Technology for MINDSCAPE at Hanon McKendry.
