Monday, June 5, 2017

There were not enough free threads in the ThreadPool to complete the operation

During stress test , with over 1000 files dropped at receive location at once, found 73 suspended instances, upon checking found the Thread starvation error


error-There were not enough free threads in the ThreadPool to complete the operation



There were not enough free threads in the ThreadPool to complete the operation

Why it happened


It is clear from error that  the number of threads available were not sufficient to cater the load of messages pumped in.

What to do


1. The first thought would be to create a separate Host for Receive/Orchestration/Send so that each process has sufficient threads/resources for processing - but here it doesn't seem that significant as this server has only one application deployed with simple orchestration calling web service and collecting response.


Note: It is best practice to have dedicated host for Receiving, Sending, Orchestration and Tracking.


2. So the next was to increase threads, you will find following in host settings


Maximum engine threads: it denotes the maximum number of messaging engine threads per CPU.



By Default it is set to 20, I set it to 30 - restarted host instance and tested again with over 1000 messages. No errors now.

You can change the below values from Host Instance Settings in Admin Console(Platform Settings -> Host  -> Select the Host -> Settings) . You have to restart the Host Instance afterwards
host maximum engine thread setting


This option specifies the maximum number of threads that can be used by the End Point Manager (EPM) - the Messaging engine. The EPM doesn't use all threads instead it starts with the number of threads equivalent to 10% of this value and  as load increases adds threads up to the specified value. As load is reduced, the number of threads allocated is reduced.

Out of curiosity tried setting the number of threads to 100 - but it didn't allow, the limit is 50



maximum engine thread limit


 3. Each host instance has its own set of resources such as memory, handles, and threads in the .NET thread pool.Thus it is important to allocate enough threads to the .NET thread pool associated with an instance of a BizTalk host to prevent thread starvation.

.NET CLR: Use this to update the number of Windows threads available in the .NET thread pool associated with an instance of a BizTalk host.You can change the below values from Host Instance Settings in Admin Console(Platform Settings -> Host Instances -> Select the Host Instance -> Settings) . You have to restart the Host Instance afterwards-

Net clr host settings

Worker threads min/max values 
By Default it is   5/25
Recommended    25/100


The values specified for the thread pools are per CPU. For example, setting MaxWorkerThreads to 100 has an effective value of 200 on a 2 CPU computer


Out of curiosity tried setting the number of threads for Max Worker/IO Threads to 5000 - but it didn't allow, limit  is 500/1000 respectively


maximum limit of IO thread


maximum limit of worker thread


Worker threads are used to handle queued work items and I/O threads are dedicated callback threads associated with an I/O completion port to handle a completed asynchronous I/O request.

So if you see(point2 and 3) EPM(Messaging Engine) has its own thread pool which we can configure using attribute Maximum Engine thread(Maximum number of messaging engine threads per CPU) whereas Orchestration and send adapter share the worker threads - XLANG (Orchestration Engine) does most of its work, including accessing the BizTalk databases to send and receive messages, on .NET thread-pool worker threads and also the send adapters(SOAP/HTTP based).

4.Fourth thing to look would be maxconnection - Controls the maximum number of outgoing HTTP connections that you can initiate from a client to a specific IP address. This can be a bottleneck if BizTalk Server makes a large number of HTTP, SOAP, or WSE requests.

Increasing the maxconnection attribute results in increased thread pool and processor utilization. With the increase in the maxconnection value, a higher number of I/O threads will be available to make outbound concurrent calls to the Web service.

By Default it is   2
Recommended     12 * Number of CPUs 


To increase the number of concurrent connections, you can add/modify the entry in the BizTalk Server configuration file, BTSNTSvc.exe.config or BTSNTSvc64.exe.config for 64-bit hosts, on each BizTalk server. You can increase/limit this for the specific servers being called by adding additional entry for it 
       <system.net>
          <connectionManagement>
              <add address=" http://www.xyz.com" maxconnection="2" />
              <add address="*" maxconnection="20" />
          </connectionManagement>
       </system.net>

 or you can have one entry set for all, like below . Maximum value is 20.


maxconnection in BizTalk config


Note:  Although increasing maxconnection would help BizTalk to push more messages out, the capacity of the downstream system should be taken into account.


Every system has a breaking point, doing above steps will help to understand the limits of your server by playing  around with various combination of loads and configuration. 

Final step - If above steps are not sufficient, then its time to scale up/out your server.

No comments:

Post a Comment