The mystery of the disappearing IIS session state

At the company I work for we are running a few MVC.NET web applications. One of the applications started
behaving funny halfway through 2015. Users were complaining that they would be randomly logged out from the
application. The funny thing is that in QA, the application works as expected, but in production, the application misbehaves.

We store some user information in IIS (in-process) session state. So the first thing I checked was that the
session does not expire unexpectedly. The fist, obvious thing to check is for unexpected app pool recycling.
But after running some test, I could confirm that the application pool was not recycling, even though the
session state gets lost.

Most of the posts on the Internet point to app pool recycling if you search for “session state disappearing”.
But this is not what was happening. After some more googling, the other thing which pops up a lot is about
load balancing and session state. If you use load balancing, you cannot use in-process session state. But
we do not have multiple IIS servers and there is no load balancing infrastructure in place.

At this point I was getting really desperate, because after making a few changes, the application was still
losing its session state randomly, and users were starting to doubt the reliability of the application.

After some more thought, and a lots of googling and reading I came across the following post.
As it turns out, IIS has a built-in load balancing feature called “Web Garden”, which splits the web work
across more than 1 process. This is controlled by the Maximum Worker Process setting on the Application
Pool used by your web application.

At this moment everything started to make sense, and I checked the PROD IIS server, and it was configured to
use 3 worker processes, but our QA IIS server was configured to use only 1 worker process. This
explains why we did not see the same problems in the QA environment. After setting the PROD IIS server’s max
worker processes to 1, there was no more strange behaviour.

[title style=”2″]Lessons learned[/title]

Lesson 1 : Even if you don’t have load balancing infrastructure in place, load balancing might be going
on because it can be configured in IIS, even if you only have 1 IIS server.

If you use load-balancing, you cannot use in-process session state.

During the time we tried to figure out what was going on, there were a few things that I tried.

First, I tried to switch from using in-process session state to out-of-process session state. It
should just be as easy as adding some lines to the web.config, right. WRONG! Switching from in-process
session state to out-of-process session state has the additional requirement that all objects stored in the
session state must also be serializable.

As it turns out, we stored quite a complex data graph in session (which had cyclic references), and trying
to change the entire data model to be serializable was too big of a task for the amount of time we had.

Lesson 2 : It is a good idea to make sure all your objects you might want to store in session state can be serialized from the start of writing the application, just for in-case.

Then, if you later need to switch to out-of-process session state, you should just be able to change the web.config, without needing to change some of the source code. Switching to out-of-process session state might be harder later on in the life cycle of the application.