I have a WCF duplex service (session instancing with net.tcp binding) that can potentially do a lot of database access. When it goes into its high-db-access routine, it become unresponsive to any WCF service call.
I've checked the CPU utilization and, while high, it is not 100%. This is on a 4 core box.
It seems that no actual requests are getting handled by WCF. Not only are no service operations called, but I have some security custom behaviors further down the WCF stack that are not getting invoked either. This makes me think that somehow WCF is getting starved of threads and so is not able to allocate new threads for access.
I don't think this is a throttling problem, since I can make this happen when there are a few current sessions with the service (less than 5).
The DB operations in question are thousands of single row inserts. This is done with a single entity framework context that is disposed and reconstructed every 500 calls to reduce the memory accumulation of entity frameworks internal caches. The target database is SQL Express 2008 R2.
There are a fair number of worker threads in this service. All are instantiated by the Task Parallel Library... some as regular short-lived tasks (which uses the CLR threadpool), others are long-running tasks (get their own CLR thread), but non of these are I/O threads (unless the CLR is doing some magic that I don't know about). The DB writes are happening on a long-running Task.
Are there any WCF or debugging diagnostic tools that can report or visualize the current state of WCF threads, worker threads, I/O threads and WCF throttling?
Environment Summary
WCF service is hosted inside a Windows service
WCF service uses a net.tcp duplex binding
WCF service is session based (required by duplex contract)
There is only a single instance of the Windows service, but multiple instances of the WCF service object are instantiated as required by WCF.
WCF service operations quickly delegate work to background threads that are persistent inside the Windows service. All WCF operations quickly return and then trickle additional results generated by the background threads down the WCF duplex callback channel.
All DB access is done on background worker threads. No WCF threads are used for long-running work.
WCF service has a single custom security behavior. There are no other behaviors (such as reliable messaging, etc)
If you're using session instancing, then each each client connection has a host service. If the host is engaged in accessing the database, then I'd say it's blocking & won't be able to handle any other calls from its client until it's done. You might be able to change the way it works to be asynchronous. Call a start method which kicks off the database activity on a worker thread and returns immediately. Either have a progress method to check the status, or since you're using duplex, have the host signal the client when it's done.
Or is the wcf service not accepting new clients?
Related
I have a WCF (on top of IIS) which will be hosted on a shared hosting environment, so I don't have access to window services or permissions for installations.
This WCF would have a long running computation (it is a spatial interpolation), so my question is about which architecture to use in order to not affect performance, in particular I don't want to grab threads from the ASP.NET tread pool for such long task.
I know that a possible solution would be a window service for multi-threading computation and MSMQ for communicating between the WCf and the window service, but as I said I don't have the possibility to install a service.
Anybody could suggest a solution?
thanks in advance
You could simple use an asynchronous/one-way method on your WCF service and call this.
We use a similar method to upload data and kick off the import process. The client will then poll using another WCF method and when the initial process has finished, update the relevant data to indicate that and return it back to the client in the poll.
We know about relation between Process and Thread.
Thread comes under Process, we can say Process is a container and Thread is an element of a container.
But what about Service ?
I can say Process and Thread having same genre.
Can we say the same thing for Services?
I found Window Services and Android Services having similarity, say in Android if we want to play Media then we have to get getSystemService(Context.AUDIO_SERVICE) likewise in Windows (8) if you stop Windows Audio (audiosrv.dll) services from services.msc then Media will not play.
What is Service?
Windows
A service is an application type that runs in the system background without a user interface and is similar to a UNIX daemon process.
Android A service is a component which runs in the background, without direct interaction with the user.
A service runs by default in the same process in the main thread as the application.
Services which run in the process of the application are sometimes called local services.
With above definition we can say apparently that Service is also a Process (i am not sure, please make me correct)
Let me start with the statement - Service is not a process. It is an activity without GUI
If you start a thread, it runs parallel with your main activity thread. But a Service is not guaranteed to always run in a new thread. So, you cannot call a service similar to Thread.
A Service is not a separate process. The Service object itself does not imply it is running in its own process; unless otherwise specified, it runs in the same process as the application it is part of.
A Service is not a thread. It is not a means itself to do work off of the main thread (to avoid Application Not Responding errors).
So When to use of a service in your application ?
If your application tells the system about something it wants to be doing in the background (even when the user is not directly interacting with the application). This corresponds to calls to Context.startService(), which ask the system to schedule work for the service, to be run until the service or someone else explicitly stop it.
Reason - Your application together with all its global variables will not be wiped out as long as there is a Service still running. So if the user is not interacting with your application and some other application in foreground needs more memory and if the OS triggers a low memory warning and your activity is destroyed, still your application is not completely lost as the service is running.
A facility for an application to expose some of its functionality to other applications. This corresponds to calls to Context.bindService(), which allows a long-standing connection to be made to the service in order to interact with it.
How the Application priority is defined based on service ?
If the service is currently executing code in its onCreate(), onStartCommand(), or onDestroy() methods, then the hosting process will be a foreground process to ensure this code can execute without being killed.
If the service has been started, then its hosting process is considered to be less important than any processes that are currently visible to the user on-screen, but more important than any process not visible.
If there are clients bound to the service, then the service's hosting process is never less important than the most important client.
A started service can use the startForeground(int, Notification) API to put the service in a foreground state, where the system considers it to be something the user is actively aware of and thus not a candidate for killing when low on memory.
I have a windows service which wakes up everyday at a particular time and finds around 100k transactions that it needs to process. It will spawn 25 threads, which look at the bucket of transactions that need to be processed and will make a call to a WCF service.
This WCF service will do some internal processing and make a synchronous call to an external service (which we have mocked and written an emulator for the sake of volume testing). Using this setup for shorter runs for around 10k transactions we were able to achieve a TPS of around 10.
I scaled this setup to have three load balanced servers running our WCF services and two other servers running the emulator, also we increased the number of threads on the windows service to 75 threads. With this new setup we expected an increase in performance, but the TPS is still at 10.
I have performance monitor running on all five machines. The three loaded balanced servers which have the WCF service are showing an "Outstanding Calls" of around 25 constantly in "ServiceModelService" category for the WCF service. But the two servers which have the emulators running show only around 9 "Outstanding Calls" constantly for the mocked out service. This same emulator was showing around 20 "Outstanding Calls" when it was running on a single server.
My questions are:
Why is there no increase in TPS in the three load balanced machines setup?
Where is the bottleneck in this system?
The target is to get to a TPS of around 30 with the three loaded balanced servers running the WCF service.
Note: I have increased the maxconnection limit in the web config on the WCF service and windows service to 200 which increased the TPS from around 6 to the current value of 10.
Edit: More information, if each of the load balanced server has 25 outstanding calls, shouldn't the mocked external service have 3*25=75 outstanding calls?
By maxconnection limit I meant:
<system.net>
<connectionManagement>
<add address="*" maxconnection="200" />
</connectionManagement>
</system.net>
You probably can't give us enough information to diagnose the problem accurately. However, what you describe gives me enough to recommend some places to look.
First, you probably shouldn't be spawning 25 (or more) threads in your Windows service. Rather, you can have a single thread looking at the "bucket of transactions," and make asynchronous calls to the WCF service. You can control the number of concurrent transactions using a Semaphore. Something like:
Semaphore _transactionSemaphore = new Semaphore(25, 25);
while (transactionCount > 0)
{
transactionSemaphore.WaitOne(); // wait for a free spot
var transaction = getTransaction();
DoAsyncWcfCall(transaction);
}
And the async completed event (see the above link about asynchronous calls) releases the semaphore:
void AsyncCompletedEvent(...)
{
// do after-call processing
// and then release the semaphore
transactionSemaphore.Release();
}
When transactionCount gets to 0, you have to wait for all outstanding calls to complete. You do that by repeatedly waiting on the semaphore: 25 times. That is:
for (int i = 0; i < 25; ++i)
{
transactionSemaphore.WaitOne();
}
If your main thread has gobbled up the semaphore, then you know that there can't be any outstanding calls.
You can extend that concurrent transaction count to 50 or 75, or whatever value you like. Your hardware being able to handle it, of course.
The difference here is that asynchronous service calls use I/O completion ports rather than individual threads. Allocating a thread that just sits there and waits is very wasteful. With the I/O completion port, the only time a thread gets involved is when the call completes--in the async completed method. Those threads are allocated automatically by the thread pool.
If the service is constantly showing 25 outstanding calls, then the total of outstanding calls for all of the servers better not be more than that. If the WCF services are showing more outstanding transactions than the Windows service is showing, then you have a problem. Come to think of it, if the service is showing more outstanding calls than do the load balanced servers, you also have a problem. If the two don't match, then somebody's dropping something: either the Windows service thinks that it has outstanding calls that the WCF services think are filled, or vice-versa.
It's hard to say where the bottleneck is. If you're experiencing high CPU usage on the Windows service machine, that's probably your bottleneck. You say that the WCF services call yet another external service. That external service could be the bottleneck. Depending on how you mocked it, the mock could be the bottleneck. Have you determined how much time that external service takes? That the services running the mocked service seem to have lower throughput than the WCF service that's talking to the real service makes me think there's a problem with performance of your mock.
I suppose it's possible that your WCF services aren't properly cleaning up resources, and they are spending an inordinate amount of time in garbage collection. Have you verified that you're using the server garbage collector? I think that's the default for WCF services, but you need to check.
Given the information you've provided, I consider those the most likely possible bottlenecks.
One other thing. It's incredibly wasteful to have a Windows service that does nothing but sit there and once a day "wakes up" to process some transactions. You should make that a console application and schedule a task to run it once per day. You can use the Task Scheduler GUI, or you can schedule it with the schtasks command. See Programs are not cats.
Another benefit of making your program a console app rather than a service is that it's a whole lot easier to debug a console app.
The question is pretty much in the title, but I will elaborate.
I have a Silverlight application that acts as a slightly extended user interface.
The main part of my program will run on a server to keep the shared database coherent.
This is where my question comes in: Will two clients calling a WCF service each get a thread inside that service OR will they get a full AppDomain each?
The difference is that if the first is the case they can share the DB easily, but in the second scenario they cannot - as I understand it.
EDIT: This is because the DB makes use of the Identity Map pattern [Fowler] where objects used are saved in physical memory (static singleton variable) - multiple AppDomains would mess that up.
(I asked my university teacher and searched quite a bit before asking this, seemingly, simple question)
The threading model for WCF services is determined by the ConcurrencyMode you configure for your service: http://msdn.microsoft.com/en-us/library/system.servicemodel.concurrencymode.aspx.
Regarding AppDomains - that depends entirely on how you're hosting your service. If you're running a ServiceHost of your own, manually, there will always be exactly 1 AppDomain on the server side, unless you decide to start managing and spinning up your own.
If you're hosting inside IIS...it's up to IIS how it handles requests. It may reuse 1 AppDomain, it may spin up multiple AppDomains (unless you override the setting in the web.config to permit only 1 AppDomain per worker process), or it may spin up multiple physical worker processes (which inherently implies multiple AppDomains) if you have web garden mode enabled.
All this said, I'm not sure exactly why this would affect your data access strategy. Multiple threads or AppDomains should have no problem sharing a DB.
Your help will give me a much better understanding on Windows azure. Thanks in advance.
I understand the worker role like a method(The Run() method in worker role), that is doing some time-consuming opperation. This allows me to have multiple instances of this worker role, to speed things up if traffic surges.
1) can I have a worker role that does more than on thing like:
- a method that creates a PDF;
- a method that creates a chart;
-a method that parses som HTML;
-a method that processes an image
I could easily do all this with 4 worker roles(one RUN() method for each functionality above), but this is very expensive. Can I place this 4 things as methods on the same worker role?
2) What is the disadvantage of hosting a WCF service on a worker role?
3) What is the disadvantage of talking between instances in the same Azure solution trough WCF insteed of queues? Is this slower?
The interface to a worker role is simple... you implement a Run() method that never returns, and we call it. What you do in there can be anything... you can spin up 100 threads doing different things, you can use Process.Start() to launch separate processes, you can start a web server, etc.
I don't know if there are disadvantages to hosting WCF in a worker role. In a web role, you could use IIS as the host, which may help with scalability of the service as compared to running your own host.
One common pattern is to have a worker role pulling work from a queue. The advantage of the queue is that it guarantees each message gets delivered at least once (so you don't lose work). It also distributes the load, because each worker can pull a message from a queue when it's ready for more work. If you use WCF or some other synchronous communication instead, you have to handle how to distribute the work and how to recover from errors (without losing work) yourself. It's certainly possible (and actually probably faster than a roundtrip to a queue), but it's harder to build a reliable, scalable service this way.
Just adding to Smarx - be careful how you slice your functionality - you pay for every worker role you spin up, so unless you have a particularly hotspot in your workflow, and you're already running multiple workers at maximum capacity, you're better off creating a single worker role that can run as multiple instances.