Thread starvation in WCF Send Port

I recently ran into an intermittent problem where we were receiving the following exception:
System.InvalidOperationException: There were not enough free threads in the ThreadPool to complete the operation.
in conjunction with a stack trace indicating we were well and truly down in the WCF ServiceModel code base. This resulted in numerous suspended messages, and a messy Event Log.

A little investigation showed that it occurred due to a bulk update of the source system, which propagated a whole bunch of messages (around 1000 in a 5 minute period), via WCF, to a web service on the destination system. This resulted in a flood of messages from BizTalk to the WCF Send Adapter, which happily allocates each message to be sent to the back of it’s own processing queue. Each pending message gets allocated a work item thread.

The destination system only exposed a ASMX-style web service, so we were restricted to using basicHttpBinding.  Windows, and by inference, BizTalk, limits the default number of outbound HTTP connections to 2.  So were were pumping a whole lot of messages to WCF, as quickly as BizTalk could, but WCF could only dispose of, at maximum, 2 at a time.  Within a short amount of time, the process thread pool is exhausted, and errors resultant.

My solution was two-fold:

Increase the ‘maxconnections’ property for outbound Http connections.

This can either be applied at the machine level, or at the app config level.  I chose to avoid unwanted side effects by only modifying the BTSNTSvc64.exe.config, adding the following snippet within the <configuration></configuration> root:

<system.net>
  <connectionManagement>
    <add address="http://name.of.server.here" maxconnection="12" />
  </connectionManagement>
</system.net>

Again, I took the conservative option, and only altered the maxconnection property for those intended for that specific server.  I dislike side-effects…

Increase the maximum number of threads allocated to the Host Instance

Fortunately, the BizTalk architecture was set up in such a way that this specific application had it’s own set of Host Instances.  This allowed me to increase the number of Maximum Engine Threads, without too much risk of starving other applications and processes on the server.

This is done within the Host Settings Dashboard, accessed by right-clicking on the BizTalk Group and selecting ‘Settings…” (sorry, BizTalk 2010+ only, folks).  You get a screen similar to the one below:

Host Engine Threads

So, I doubled the number of threads, which in conjunction with the increased throughput, above, should ensure we don’t run into this issue again.