I have a node.js service running in Azure as worker role. By default the process is restarted every time there is a topology change, e.g. instance count is increased via Azure portal. How can I prevent this restart?
MSDN documentation pointed to handling Azure's "Changing" event. Azure Node SDK's support for cancelling was added here and here.
The code to use the API would be something like
azure.RoleEnvironment.on(ServiceRuntimeConstants.CHANGING, function (changes) {
changes.cancel();
});
From logs I know the handler is called, but restarts still took place afterwards. Am I using the API incorrectly or is this the wrong approach?
When using the Role Changing event in .NET when you cancel the event you are actually asking the system to restart. The docs for the Role Changing event on MSDN say this:
By using the Cancel property, you can ensure that the instance
proceeds through an orderly shutdown sequence and is taken offline
before the configuration change is applied. During the shutdown
process, Windows Azure raises the Stopping event, and then runs any
code in the OnStop method.
The idea of the Role Changing event is that if you can modify the configuration at runtime without requiring a restart you do so. By cancelling the changing event you are in essence saying, "I can't change this at runtime, so restart gracefully and pick up the new changes then."
I've not tried this with Node, but try not cancelling it.
Related
I created an Azure Function App with a Node runtime, which works properly on local and manually created cloud environments.
But when it becomes deployed via Azure Pipelines, it writes a message via context.log and seems working but finally it raises Timeout error.
Timeout value of 00:05:00 exceeded by function 'Functions.<...>' (Id: '<...>'). Initiating cancellation.
I guess, that there is some blocking Node expression because of misconfiguration, but there is no further context logged by Application Insights.
There is a way to handle the cancelation event within your Function App to provide some Node runtime information (e.g. via SIGINT callbacks)?
I've tried to reproduce this issue but failed. But I got some similar question here and noticed we can set functionTimeout value in host.json file. How about trying it.
I have an Azure function running on a timer every few minutes that after a varied amount of time of running will begin to fail every time it runs because of an external API and hitting the restart button manually in the azure portal fixes the problem and the job works again.
Is there a way to either get an azure function to restart itself or have something externally restart an azure function via a web hook or API request or running on a timer
I have tried using Azures API Management service which can be used to restart other kinds of app services in azure but it turns out there is no functionality in the API to request a restart of an azure function, Also looked into power shell and it seems to be the same problem you can restart different app services but not azure functions
i have tried working with the API
https://learn.microsoft.com/en-us/rest/api/azure/
Example API request where you can list functions within an azure function
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/sites/{name}/functions?api-version=2016-08-01
but there is no functionality to restart an azure function from what i have researched
Basically i want to Restart the Azure function as if i was to hit this button
Azure functions manual stop/start and restart buttons in azure portal
because there is a case where the job gets into a bad state every time it runs because of an external API i have no control over and hitting restart manually gets the job going again
Another way to restart your function is by using the "watchDirectories" setting in the host.json file. If your host.json looks like this:
{
"version": "2.0",
"watchDirectories": [ "Toggle" ]
}
You could toggle a restart by using following statement in a function:
System.IO.File.WriteAllText("D:/home/site/wwwroot/Toggle/restart.conf", DateTime.Now.ToString());
Looking at the logs, the function reloads as it has detected the file change in the directory:
Watched directory change of type 'Changed' detected for 'D:\home\site\wwwroot\Toggle\restart.conf'
Host configuration has changed. Signaling restart
Azure functions by their nature are called upon an event. That may be a timer, a trigger or invocation like a HTTP event. They cannot be restarted per se, i.e. if you a function throws and exception, you cannot find the specific instance and re-run it using the out of the box functionality.
However, you can engineer your way to a more reliable solution:
Replay the event that invoked the function (i.e. kick it off again)
For non-sensitive data, log the payload of the function and create a another function that can be called on demand to re-run it. I.e. you create a proxy to "re-invoke" the function.
Harden your code by implementing a retry policy. See Polly.
Add a service bus in to your architecture. Have a simple function to write the call payload to a message bus payload. Have another function to pick up the payload and process it more extensively where there may be unreliable integrations etc). That way if the call fails you can abandon and dead letter failures for later reprocessing.
Consider using Durable Function Extensions and leveraging the durable patterns, these can help make your functions code more robust and manage state.
Why don't you try below ARM API. Since Azure function also fall under App service category, sometimes this may be helpful,
https://learn.microsoft.com/en-us/rest/api/appservice/webapps/restart
I've got an Azure Function app in production on an event hub trigger, it's low throughput with the function typically only being triggered once daily. It's running on an S1 plan at the moment and has a few other functions such as timer triggered and HTTP triggered.
It's been running fine but today it stopped being triggered by new messages until I restarted the app. All other functions were working just fine and responding to their associated triggers.
I've look through App Insights and there are no reported errors or issues, it's just not doing anything.
Has anyone else had this issue or know of what may be causing it?
First of all - is your App Service has Always On enabled?
Second thing - have you tried to test your trigger locally, so you can be sure, that there are no issues with your Event Hub?
Personally, I faced such issues when Event Host Processor implemented in EventHubTrigger was losing a lease because of additional processor introduced. It is also possible, that since it faces a low throughput, it lost a lease and for some reason was not able to renew it:
As an instance of EventProcessorHost starts it will acquire as many
leases as possible and begin reading events. As the leases draw near
expiration EventProcessorHost will attempt to renew them by placing a
reservation. If the lease is available for renewal the processor
continues reading, but if it is not the reader is closed and
CloseAsync is called - this is a good time to perform any final
cleanup for that partition.
https://blogs.msdn.microsoft.com/servicebus/2015/01/21/event-processor-host-best-practices-part-2/
Nonetheless, it is worth to contact the support to make sure there were no other issues.
in my service deployment i have two roles.. a web role and a worker role.. in my on_start() method of webrole im enumerating the instances of the worker roles and creating a tcp connection on some internal end point. but very often it fails because the instances of the worker role havent started yet.
so the question is that can i know wether the instances have started or can i wait for instances of the worker role to start in some way?
herez the code
public override bool OnStart()
{
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
ConnectionStatics.ConnectRouterToWorkers();
Messaging.KeepReadingMessages.Start();
return base.OnStart();
}
I'd recommend building retry logic into your loop so when its unable to establish the connection, it just sleep and retries it again later. Not only will this address your startup issue, but it will help you address changes to the service topology as instances are added/removed by the fabric controller (which can happen for numerous reasons).
Taken a step further, you might be able to leverage the RoleEnvironmentChanging and RoleEnvironmentChanged events to provide notification of when an instances is added/dropped. But I haven't leveraged this personally and can't say with any certain how these methods may or may not reflect the "ready state" of particular instances.
Another option would be to have the worker roles put a message in a queue when they are started up. Then you could just check the queue and wait for it to post a message there.
I see how Azure checks the status of my worker role periodically, but how?
There is no method in RoleEntryPoint to do that, and I'm taking a look on Microsoft.WindowsAzure.ServiceRuntime's classes with ILSpy but I don't see anything relevant.
Here's a blog post that describes how the Windows Azure Fabric Controller monitors instance health.
Aside from that, the controller calls a StatusCheck event, every 15 seconds, that you can handle. If you want to pull yourself out of the load balancer (maybe based on some internal data your instance has), you just call SetBusy() on the RoleEnvironmentStatusCheckEventArgs object. This takes you out of the load balancer until the next check.
I think the mechanism is the same as the one used for WebRoles
the Azure RoleEnvironment performs a StatusCheck - see http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.serviceruntime.roleenvironment.statuscheck.aspx
If you want to tell the service you are busy then call SetBusy() when this event fires