I have a simple StatelessService and I want to knew when it is being closed down so I can perform some quick clean up. But it never seems to call OnCloseAsync.
When the service is running and I use the 'Restart' command on the running node via the Service Fabric Explorer, it removes the services and restarts the node. But it never calls the OnCloseAsync override, even though it is knowingly being closed down.
Nor does it signal the cancellationToken that is passed into the RunAsync method. So there is no indication that the service is being shutdown. Are there any circumstances when it does call OnCloseAsync, because I cannot see much point in it at the moment.
I wonder the reasoning behind issueing the restart command, what behavior do you expect?
It does explain however the behavior you see. From the docs (Keep in mind that a restart is just a combined stop and start)
Stopping a node puts it into a stopped state where it is not a member of the cluster and cannot host services, thus simulating a down node. This is useful for injecting faults into the system to test your application.
Now, if we take a look at the lifecycle we read this:
After CloseAsync() finishes on each listener and RunAsync() also finishes, the service's StatelessService.OnCloseAsync() method is called, if present. OnCloseAsync is called when the stateless service instance is going to be gracefully shut down.
So, the basic problem is that you service is not gracefully shutdown. The restart command kills the process and no cancellation will be issued.
Related
I am troubleshooting an issue where a service dependency is created in the Program.cs and passed into the Service Class. (for more context this is a stateless service, but my question applies for both) This services RunAsync method uses the CancellationToken supplied to determine if the service is still running. If the token gets cancelled then it calls dispose on the dependency. The symptom that I am diagnosing is that on start up sometimes the dependency is not initialized. I am pretty sure I read in the docs somewhere that the host process in some scenarios may be reused and not torn down when a service instance is torn down, but I can't seem to find it now.
Does the Host process outlive, and rehost new service instances in Service Fabric?
As far as I get it, if you have any replica around the process won't shut down. If there are no replicas left, the process will be closed after a grace interval.
See these discussions for more information - Processes keep running after service is deleted and Processes still keep running after Service Fabric App is removed.
Context
I've added configuration validation to some of the modules that compose my Node.js application. When they are starting, each one checks if it is properly configured and have access to the resources it needs (e.g. can write to a directory). If it detects that something is wrong it sends a SIGINT to itself (process.pid) so the application is gracefully shutdown (I close the http server, close possible connections to Redis and so on). I want the operator to realize there is a configuration and/or environment problem and fix it before starting the application.
I use pm2 to start/stop/reload the application and I like the fact pm2 will automatically restart it in case it crashes later on, but I don't want it to restart my application in the above scenario because the root cause won't be eliminated by simply restarting the app, so pm2 will keep restarting it up to max_restarts (defaults to 10 in pm2).
Question
How can I prevent pm2 from keeping restarting my application when it is aborted during startup?
I know pm2 has the --wait-ready option, but given we are talking about multiple modules with asynchronous startup logic, I find very hard to determine where/when to process.send('ready').
Possible solution
I'm considering making all my modules to emit an internal "ready" event and wire the whole thing chaining the "ready" events to finally be able to send the "ready" to pm2, but I would like to ask first if that would be a little bit of over engineering.
Thanks,
Roger
We know about relation between Process and Thread.
Thread comes under Process, we can say Process is a container and Thread is an element of a container.
But what about Service ?
I can say Process and Thread having same genre.
Can we say the same thing for Services?
I found Window Services and Android Services having similarity, say in Android if we want to play Media then we have to get getSystemService(Context.AUDIO_SERVICE) likewise in Windows (8) if you stop Windows Audio (audiosrv.dll) services from services.msc then Media will not play.
What is Service?
Windows
A service is an application type that runs in the system background without a user interface and is similar to a UNIX daemon process.
Android A service is a component which runs in the background, without direct interaction with the user.
A service runs by default in the same process in the main thread as the application.
Services which run in the process of the application are sometimes called local services.
With above definition we can say apparently that Service is also a Process (i am not sure, please make me correct)
Let me start with the statement - Service is not a process. It is an activity without GUI
If you start a thread, it runs parallel with your main activity thread. But a Service is not guaranteed to always run in a new thread. So, you cannot call a service similar to Thread.
A Service is not a separate process. The Service object itself does not imply it is running in its own process; unless otherwise specified, it runs in the same process as the application it is part of.
A Service is not a thread. It is not a means itself to do work off of the main thread (to avoid Application Not Responding errors).
So When to use of a service in your application ?
If your application tells the system about something it wants to be doing in the background (even when the user is not directly interacting with the application). This corresponds to calls to Context.startService(), which ask the system to schedule work for the service, to be run until the service or someone else explicitly stop it.
Reason - Your application together with all its global variables will not be wiped out as long as there is a Service still running. So if the user is not interacting with your application and some other application in foreground needs more memory and if the OS triggers a low memory warning and your activity is destroyed, still your application is not completely lost as the service is running.
A facility for an application to expose some of its functionality to other applications. This corresponds to calls to Context.bindService(), which allows a long-standing connection to be made to the service in order to interact with it.
How the Application priority is defined based on service ?
If the service is currently executing code in its onCreate(), onStartCommand(), or onDestroy() methods, then the hosting process will be a foreground process to ensure this code can execute without being killed.
If the service has been started, then its hosting process is considered to be less important than any processes that are currently visible to the user on-screen, but more important than any process not visible.
If there are clients bound to the service, then the service's hosting process is never less important than the most important client.
A started service can use the startForeground(int, Notification) API to put the service in a foreground state, where the system considers it to be something the user is actively aware of and thus not a candidate for killing when low on memory.
I am a newbie and trying to figure out how process monitoring works with JXcore. I saw the documentation but need few steps in order to make my server application starting multithreaded and monitored properly.
Thanks in advance!
I'll try to explain it to you. There is no shame to be a newbie! :)
JXcore offers you two types of application monitoring.
1) One of them is Process Monitor and this is a process, which runs as separate instance. Your applications may subscribe to it for being monitored. Monitor verifies them in regular intervals, and if it finds that your application is gone it tries to relaunch it. For example, if your application servers http and should be online all the time - Process Monitor will ensure, that it is really running.
The fastest way to start to monitor your application is to:
launch the monitor: > jx monitor start
launch your application with automatic subscription to the monitor: > jx monitor run app.js
After that, when your application crashes, Process Monitor will restart it. You can test it by just killing your application's process.
Process monitor also gives you information about currently monitored processes. You can browse to see the list of them:
http://127.0.0.1:17777/json
2) Second type of a monitoring feature is process and thread recovery. With Process Recovery you can achieve the same as with the Process Monitoring, so there is no reason to use them both at the same time.
Another scenario could be:
Let's say you have a multithreaded application and only to recovering it's threads is enough.
Your application is launched with a command:
jx mt-keep:3 app.js
which means, that you run it with 3 threads.
To enable Thread Recovery is enough to subscribe to process.on('restart') event like this:
process.on('restart', function (cb) {
process.release();
cb();
});
Remember, to call cb() callback. As you probably saw it in the docs, the thread will not restart until you invoke this callback. Before restart, you may back-up things etc.
Basically that's it. Feel free to play with it!
I need to test how my code will handle the failure of a web role instance in a development environment.
How do I terminate one of the instances? I can't see any option in the UI for this. Seems like a strange ommission
Update
The issue is relating to a distributed cache layer (I know that azure offers their own)
I want to be able to test how the system reacts to a missing or additional node etc
Prehaps my real question is
how up to date is RoleEnvironment.CurrentRoleInstance.Role.Instances
The need to simulate ungraceful exits in the dev emulator usually is done because you are doing something in your web role that is stateful or long running. That is generally discouraged, but sometimes is unavoidable.
I suspect the best way to simulate the a failure is to kill processes. If you open task manager (or better Process Explorer), you will see "WatDebugger" hosting either "WaIISHost" or "WaWorkerHost". If you kill this process, I think it will simulate a failure.
Honestly, it is easier to test this one in the cloud however. You can RDP into one of the instances and kill the 'WaAppAgent' process. That will kill your RoleEntryPoint and fabric controller agent. That will be a true ungraceful failure.
By failure, do you mean becoming unavailable? It should be seamless because the next request would simply be handled by one of the other instances. As long as there is one instance available Azure will route calls to that instance.
This is the nature of a high-available system, requests are handled by the available instances. This is why you have multiple instances in the first place, to handle requests in the case of failure in one or more instances.
This is why you need to always be watchful of how your application handles state. State needs to be maintained outside of the instance, either in queues or in a database. This ensures that any process can pickup a piece of work and execute against it.
There is another question dealing with Session State that should help: How does Microsoft Azure handle Session State?
By terminate an instance do you mean reducing instance count and see which one gets killed? I like Ryan's view about ungraceful exits, but if it's forced kill by the fabric it'll be a different ball game.