At the risk of asking a stupid simple question:
I have a console application that uses servicestack framework to listen to a redis queue. Eventually I want to publish it up as a continuous azure web job.
I've seen examples that use:
host.RunAndBlock();
However I have no absolutely zero need to use the azure webjobs SDK and so just wondering is there anything 'wrong' with just using:
Console.ReadKey();
To keep the console program running.
I assume you're talking about a continuous WebJob.
I don't think Console.ReadKey() will work. The call is likely to just blow up in the sandbox that WebJobs run in.
If you are able to do the redis queue listening on the main thread, that would be simplest.
If now, just use an infinite sleep loop in your main. e.g.
for (;;)
{
System.Threading.Thread.Sleep(5000);
}
Related
Context
In an ASP.NET Core application I would like to execute an operation which takes say 5 seconds (like sending email). I do know async/await and its purpose in ASP.NET Core, however I do not want to wait the end of the operation, instead I would like to return back to the to the client immediately.
Issue
So it is kinda Fire and Forget either homebrew, either Hangfire's BackgroundJob.Enqueue<IEmailSender>(x => x.Send("hangfire#example.com"));
Suppose I have some more complex method with injected ILogger and other stuff and I would like to Fire and Forget that method. In the method there are error handling and logging.(note: not necessary with Hangfire, the issue is agnostic to how the background worker is implemented). My problem is that method will run completely out of context, probably nothing will work inside, no HttpContext (I mean HttpContextAccessor will give null etc) so no User, no Session etc.
Question
How to correctly solve say this particular email sending problem? No one wants wait with the response 5 seconds, and the same time no one wants to throw and email, and not even logging if the send operation returned with error...
How to correctly solve say this particular email sending problem?
This is a specific instance of the "run a background job from my web app" problem.
there is no universal solution
There is - or at least, a universal pattern; it's just that many developers try to avoid it because it's not easy.
I describe it pretty fully in my blog post series on the basic distributed architecture. I think one important thing to acknowledge is that since your background work (sending an email) is done outside of an HTTP request, it really should be done outside of your web app process. Once you accept that, the rest of the solution falls into place:
You need a durable storage queue for the work. Hangfire uses your database; I tend to prefer cloud queues like Azure Storage Queues.
This means you'll need to copy all the data over that you will need, since it needs to be serialized into that queue. The same restriction applies to Hangfire, it's just not obvious because Hangfire runs in the same web application process.
You need a background process to execute your work queue. I tend to prefer Azure Functions, but another common approach is to run an ASP.NET Core Worker Service as a Win32 service or Linux daemon. Hangfire has its own ad-hoc in-process thread. Running an ASP.NET Core hosted service in-process would also work, though that has some of the same drawbacks as Hangfire since it also runs in the web application process.
Finally, your work queue processor application has its own service injection, and you can code it to create a dependency scope per work queue item if desired.
IMO, this is a normal threshold that's reached as your web application "grows up". It's more complex than a simple web app: now you have a web app, a durable queue, and a background processor. So your deployment becomes more complex, you need to think about things like versioning your worker queue schema so you can upgrade without downtime (something Hangfire can't handle well), etc. And some devs really balk at this because it's more complex when "all" they want to do is send an email without waiting for it, but the fact is that this is the necessary step upwards when a baby web app becomes distributed.
I have a webjob in Azure, hosted on an App Service that is not used for anything. I am currently deploying my webjob from Visual Studio, but this will change in the future as it's not in production. It's a .NET Core 3.1 application webjob that compiles to an EXE, but that shouldn't matter to this question (and I'm aware of Azure Functions, but that is also not a part of my question).
The webjob is a continous webjob triggered by a queue. I have set it up to run 10 batches simultaneously. I have looked online for answers, but I have found unclear answers.
My question is: Let's say I have 3 jobs running. Then I deploy a new version of the EXE file. This seems to work without problems. But what happens to the jobs that are running? Will they continue running to the end? Or will it fail and stop? I haven't quite managed to sort that out and I wanted to ask here in case someone have helpful experience on this.
My queue related config is like this, if that's helpful:
.ConfigureWebJobs(b =>
{
b.AddAzureStorageCoreServices();
b.AddAzureStorage(a =>
{
a.BatchSize = 10;
a.NewBatchThreshold = 5;
a.MaxDequeueCount = 1;
a.MaxPollingInterval = TimeSpan.FromSeconds(20);
});
})
Thank you!
But what happens to the jobs that are running? Will they continue running to the end? Or will it fail and stop?
When a WebJob that uses the WebJobs SDK picks up a message from a queue, it acquires it with a 10 minutes lease. If the job process dies while processing the message, the lease expires after 10 minutes and the message goes back in the queue. If the WebJob is restarted, it will pick that message again. The message is only deleted if the function completes successfully.
Therefore, if the job dies and restarts immediately, like in the case of a redeploy, it might take up to 10 minutes to pick again the message. Also, because of this, it is recommended to either save state yourself or make the function idempotent.
In the WebJobs Dashboard you will see two invocations for the same message. One of them will be marked as Never Finished because the function execution never completed.
Unfortunately, there is no out of the box solution to prevent jobs from running during deploy. You would have to create your own logic that notifies (through a queue message?) that a deploy is about the start and then aborts the host. The host abort will wait for any existing function to stop and will prevent new ones from starting. However, this is a very tricky situation if you have multiple instances of the webjob because only one of them will get the notification.
I am attempting to connect my application to multiple IRC channels to read incoming chat messages and send them to my users. New channels may be added or existing channels may be removed at any time during the day and the application must pick up on this in near real-time. I am currently using Microsoft Azure for my infrastructure and am using App Services for client-facing compute and Azure Functions on the App Service plan for background tasks (Not the consumption billing model).
My current implementation is in C#/.NET Core 3.1 and uses a TcpClient over an SslStream to watch each channel. I then use a StreamReader and await reader.ReadLineAsync() to watch for new messages. The problem I am running into is that neither App Services or Azure Functions seems to be an appropriate place to host a watcher like this.
At first, I tried hosting it in the Azure Function app as this clearly seems like a task for a background worker, however Azure Functions inherently want to be triggered by a specific event, run some code, and then end. In my implementation, the call to await reader.ReadLineAsync() halts processing until a message is received. In other words, the call running the watcher needs to run in perpetuity, which seems to go against the grain of an Azure Function. In my attempt, the Azure Function service eventually crashes, the host unloads, all functions on the service cease and then restart a few minutes later when the host reloads. I am unable to find any way to tell what is causing the crash. This is clearly not the solution I want. If I could find an IrcMessageTrigger Azure Function trigger, this would probably be the best option.
Theoretically, I could host the watcher in my App Service, however when I scale out I would run into a problem due to having multiple servers connecting to each channel at once. New messages would be sent to each server and my users would receive duplicates. I could probably deal with this, but the solution would probably be hacky and I feel like the real solution would be to architect it better in the first place.
Anyone have an idea? I am open to changing the code or using a different Azure service (assuming it isn't too expensive) but I will be sticking with C# and .NET Core on Azure infrastructure for this project.
Below is a part of my watcher code to provide some context.
while (client.Connected)
{
//This line will halt execution until a message is received
var data = await reader.ReadLineAsync();
if (data == null)
{
continue;
}
var dataArray = data.Split(' ');
if (dataArray[0] == "PING")
{
await writer.WriteLineAsync("PONG");
await writer.FlushAsync();
continue;
}
if (dataArray.Length > 1)
{
switch (dataArray[1])
{
case "PRIVMSG":
HandlePrivateMessage(data, dataArray);
break;
}
}
}
Thanks in advance!
Results are preliminary, but it appears that the correct approach is to use Azure WebJobs running continuously to accomplish what I am trying to achieve. I did not consider WebJobs initially because they are older technology than Azure Functions and essentially do the same work at a lower level of abstraction. In this case, however, WebJobs appear to handle a use case that Functions are not intended to support.
To learn more about WebJobs (including continuous WebJobs) and what they are capable of, see the Microsoft documentation
I have an Azure function running on a timer every few minutes that after a varied amount of time of running will begin to fail every time it runs because of an external API and hitting the restart button manually in the azure portal fixes the problem and the job works again.
Is there a way to either get an azure function to restart itself or have something externally restart an azure function via a web hook or API request or running on a timer
I have tried using Azures API Management service which can be used to restart other kinds of app services in azure but it turns out there is no functionality in the API to request a restart of an azure function, Also looked into power shell and it seems to be the same problem you can restart different app services but not azure functions
i have tried working with the API
https://learn.microsoft.com/en-us/rest/api/azure/
Example API request where you can list functions within an azure function
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/sites/{name}/functions?api-version=2016-08-01
but there is no functionality to restart an azure function from what i have researched
Basically i want to Restart the Azure function as if i was to hit this button
Azure functions manual stop/start and restart buttons in azure portal
because there is a case where the job gets into a bad state every time it runs because of an external API i have no control over and hitting restart manually gets the job going again
Another way to restart your function is by using the "watchDirectories" setting in the host.json file. If your host.json looks like this:
{
"version": "2.0",
"watchDirectories": [ "Toggle" ]
}
You could toggle a restart by using following statement in a function:
System.IO.File.WriteAllText("D:/home/site/wwwroot/Toggle/restart.conf", DateTime.Now.ToString());
Looking at the logs, the function reloads as it has detected the file change in the directory:
Watched directory change of type 'Changed' detected for 'D:\home\site\wwwroot\Toggle\restart.conf'
Host configuration has changed. Signaling restart
Azure functions by their nature are called upon an event. That may be a timer, a trigger or invocation like a HTTP event. They cannot be restarted per se, i.e. if you a function throws and exception, you cannot find the specific instance and re-run it using the out of the box functionality.
However, you can engineer your way to a more reliable solution:
Replay the event that invoked the function (i.e. kick it off again)
For non-sensitive data, log the payload of the function and create a another function that can be called on demand to re-run it. I.e. you create a proxy to "re-invoke" the function.
Harden your code by implementing a retry policy. See Polly.
Add a service bus in to your architecture. Have a simple function to write the call payload to a message bus payload. Have another function to pick up the payload and process it more extensively where there may be unreliable integrations etc). That way if the call fails you can abandon and dead letter failures for later reprocessing.
Consider using Durable Function Extensions and leveraging the durable patterns, these can help make your functions code more robust and manage state.
Why don't you try below ARM API. Since Azure function also fall under App service category, sometimes this may be helpful,
https://learn.microsoft.com/en-us/rest/api/appservice/webapps/restart
I'm trying to implement some analytics logic in my WCF REST web service but I don't want to damage performance while I do so.
I was thinking of starting a new thread that would communicate with the analytics service (Mixpanel) while the main thread does the actual work but I'm not sure this accomplishes what I want to do.
My assumption is that the web service would return a response as soon as its main thread is done while the secondary thread runs on its own and may run longer without the client waiting any extra time.
Is that an accurate assumption?
Some testing showed that my assumption was accurate.