Stefan Barlow

ASP.NET 2.0 event log error workaround when using network shares

stefan — Thu, 31 May 2007 19:37:00 GMT

After considerable research by our team and Microsoft, we found an un-documented issue with ASP.NET 2.0. The issue manifests itself with tons of error messages in the event logs:

Unable to get the private bytes memory limit for the W3WP process. The
ASP.NET cache will be unable to limit its memory use, which may lead to a
process restart. Error: 0x80070005

The only support documentation we could find was related to Application Center 2000 or using IISCFG to replicate the IIS Metabase settings between sites in a load balanced cluster. Neither of these applied in our situation, and none of the suggested workarounds actually solved the issue.

What we did find was an issue when using the "Connect As..." option for storing web site files on a network share. It turns out that this account must be included in the IIS_WGP Group as well, plus give it "Logon as a Service" privileges. Here is a complete list of the steps we had to take on each server to resolve the issue:

aspnet_regiis –ga “Network Service”
cscript metaacl.vbs IIS://Localhost/W3SVC/AppPools IIS_WPG RE
Add “Connect As” account to IIS_WPG group.
Allow "Connect As" accont to logon as a service.

All Support Resources Now Under One Site / Login

stefan — Wed, 25 Apr 2007 05:05:00 GMT

Picking the right tools

Over the last few months, we have been implementing a lot of support ticket, development tracking, community building, and documentation systems. We picked what we found to be the best systems for their feature sets, compatibility, and customization abilities. There are no perfect, do it all software packages or services for our requirements, so we had to find the products that were as close as possible, and who had the least amount of unacceptable "features" and behaviors.

The primary systems we have decided on are:

AxoSoft OnTime 2007 - Support tickets, defect tracking, feature request tracking
Community Server - Forums, blogs, portal framework
Absolute FAQ Manager .NET - Knowledge Base management
Adobe Robohelp - Documentation management

Pulling it all together

Having separate systems for all of these pieces left us pretty scattered while we were solidifying our decisions. Separate logins for the forums and support tickets caused a lot of confusion and frustration for our customers, not to mention having so many different places to look for information!

This week we have consolidated these systems under Community Server at myamericommerce.com. We developed a Single Sign-on system to provide one login account for both the help desk and community areas, and integrated the Support Requests, Requested Features, and Reported Bugs directly into Community Server.

The search box at the top has been replaced with our own custom search system which checks both knowledge base items and forum posts.

The knowledge base and documentation have been added as tabs as well, keeping our customers with consistent navigation for all of our support areas.

We have a lot of additional tweaks, integrations, and features to add to these systems over the coming months, and we will continue to strive for the best customer support possible.

.NET 1.0 / 1.1 / 2.0 Compatibility Issues

stefan — Thu, 05 Apr 2007 03:33:00 GMT

I've read extensively about the backwards-compatibility of .NET versions, and that there are but a few breaks between the versions. After our major migration of several hundred thousand lines of code over 26 projects from 1.1 to 2.0, I’d have to say for the most part that is true.

However, there is definitely some sort of issue with referencing assemblies compiled to the older versions. Particularly 3^rd party components, many of which we don’t have the ability to re-compile with our solution.

All of our 3^rd party components are strongly named and live in the GAC with the rest of our assemblies. Everything is versioned, and most are referenced by name in the web.configs.

At seemingly random intervals, *some* of the 3^rd party components, all of which were compiled against 1.0 and 1.1, on one out of many sites that reside in a shared app pool, will throw the following exception:

System.IO.FileLoadException: The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

The ONLY thing that will make this error go away for this site is recycling the entire app pool. Touching the web.config for the site doesn’t work. Moving the site to another app pool, recycling it, and moving it back does not work. Only recycling the original app pool will do the trick.

Note that absolutely no change to any of the sites or assemblies referenced explicitly by the code and config files has been made to cause this issue, and it only tends to happen with one site (sometimes out of 100) in an app pool at a time.

Looking at the Fusion logs doesn’t show much. My postings to the MSDN newsgroups elicited a lot of the common issues and questions, all of which have been ruled out for weeks.

As we are able to replace each of these older component with 2.0 versions, loading those components no longer throw the exception. We have about 3 remaining components that there seem to be no 2.0 versions of available, but the majority have been replaced.

UPDATE & RESOLUTION TO THIS ISSUE (posted 5/21/2007)

After much deliberation, we found that due to ASP.NET 2.0’s new abililty to share assemblies across app domains, when we re-added the 3^rd party components to the GAC in our usual rollout, even though they were identical to the existing versions on the servers, it caused their complied images to become invalid. However, since the version numbers are the same, ASP.NET does not JIT re-compile the assemblies. This causes the assembly manifest definition to become invalid. The errors were still very random, which didn't make much sense at all. Go figure! J

Slower initial compile times with 2.0

A side issue we found was the initial startup time of these 1.1 and 1.0 assemblies after a site update was incredibly long, sometimes 60 seconds each or more for the assemblies to be loaded the first time.

So, the only thing we can suggest for others in this situation is that 1.0 and 1.1 assemblies cannot be counted on to run quickly under the 2.0 framework, at least when referenced from 2.0 code.

AxoSoft OnTime 2007

stefan — Mon, 02 Apr 2007 16:57:00 GMT

We started using AxoSoft OnTime 2007 for development / defect / support incident tracking last week, and so far it is working out great. We researched and tweaked the application for many hours before making the decision to switch from Ready Desk for the support ticket management.

The application is very easy to use and quite configurable. I believe the Customer Portal section used for the support incident tracking is brand new in the recently released 2007 version, so we look forward to a lot of improvements and growth of that area. There are a few week points, such as email management and CRM, but so far we've worked around these issues.

Overall it is a great product and I know that it will give us a much better view of our development tasks and schedule than we had before. After a few more weeks with the app, I'll write a follow-up.

Who is Stefan?

stefan — Sat, 31 Mar 2007 03:53:00 GMT

That's a great question. I'm glad you asked! Since you clicked on the link, you must be at least a LITTLE interested?

I the proud father of Oliva and Isaiah, and grateful husband of Tanisha. I enjoy travel, adventure, and coercing technology to do my bidding.

During normal business hours (25 hours per day in the ecommerce world), I am a Co-Founder and the Systems Architect of AmeriCommerce. A fancy title, which essentially means I oversee the development and server teams to bring you cool, profit lifting features in a reliable and secure environment to operate your online business.

My IT career began in 1981 at the ripe age of 5 years old, when my parents gave my older brother and me a Texas Instruments TI-99/4A. It was a hypnotizing device that forever altered my life. Many obsolete platforms and incredible amounts of money later (during the 80's some of these pieces of junk cost almost as much as a car), as a teenager I got hooked on the pre-web internet and creating and operating BBSes (Bulletin Board Systems).

In 1990, while other kids my age were doing whatever 14 year olds do, I was designing ANSI UI and customizing commercial software for an internationally known BBS systems, trying to turn a profit from subscription based members. I began building HTML sites in 1994, moving into PHP, ASP, and then C# and ASP.NET. Terrifically exciting, I know!

Joining with Charlie Cooper at Americom in 2001 as IT Director, we formed the foundation of what was soon to become AmeriCommerce. The last 6 years we have seen amazing growth of the software, infrastructure, and talent behind AmeriCommerce. I greatly look forward to watching this company continue to grow and to always provide innovative and powerful tools for online store owners.