I am using a Mobile Services in Windows Azure. I use the new Scheduler available for a Mobile Service. The Scheduler I called SendOut.
I am running a pretty simple script that will insert a message to a queue. The entire script:
function SendOut() {
var azure = require('azure');
var queueService = azure.createQueueService("mailsoutscheduler", "[The key to the storage]");
queueService.createQueueIfNotExists("mailsout", function(error){ });
queueService.createMessage("mailsout", "SendOut", function(error){});
}
It works fine when I try to run the script once. It it scheduled to run every 5 minutes. And it usually goes fine. However sometimes I receive this error:
An unhandled exception occurred. Error: One of your scripts caused the
service to become unresponsive and the service was restarted. This is
commonly caused by a script executing an infinite loop or a long,
blocking operation. The service was restarted after the script
continuously executed for longer than 1000 milliseconds.
at EventEmitter. (C:\DWASFiles\Sites\VogSendOut\VirtualDirectory0\site\wwwroot\runtime\server.js:84:17)
at EventEmitter.emit (events.js:88:20)
I cannot figure out why I get this error - or how to solve it.
Could it be because it's running in the FREE Mobile Service Tier?
I don't think it's due to the FREE mobile subscription.
try to add a
try{}
catch{} block
and use console.log() to log if an error occured. It could help you to resolve you problem.
Related
I am new to Azure development and developed a function app
I published my function app to Azure portal. It is working fine on my development machine but on portal it's throwing following exception (some times)
The operation 'ScanLogs' with id '2eggec6de-54f5-4t34-5423-afffce5c6a43' did not complete in '00:02:00'.
I couldn't find solution to this error. Can somebody help me to understand what this error is about and why we get this?
following is timeout specified in host.json in prod.
Depending on the version of function you are running check here for the exact syntax in host.json - https://learn.microsoft.com/en-us/azure/azure-functions/functions-host-json-v1
The default timeout for a function on the consumption plan is 2 minutes, if you need it to run longer then change/add the functiontimeout value, i.e.
"functionTimeout": "00:05:00",
in host.json.
NB: On a Consumption plan this can't be more than 10 minutes so if you need it to run longer either find a way to break up your function into smaller chunks and maybe use a Durable function fan=out fan-in pattern or change it to run on a dedicated App Service Plan where it can run for as long as you like but obviously you'll have to pay to have the server running 24/7
I have a node server running on Google Cloud Run. Now I want to enable stackdriver tracing. When I run the service locally, I am able to get the traces in the GCP. However, when I run the service as Google Cloud Run, I am getting an an error:
"#google-cloud/trace-agent ERROR TraceWriter#publish: Received error with status code 403 while publishing traces to cloudtrace.googleapis.com: Error: The request is missing a valid API key."
I made sure that the service account has tracing agent role.
First line in my app.js
require('#google-cloud/trace-agent').start();
running locally I am using .env file containing
GOOGLE_APPLICATION_CREDENTIALS=<path to credentials.json>
According to https://github.com/googleapis/cloud-trace-nodejs These values are auto-detected if the application is running on Google Cloud Platform so, I don't have this credentials on the gcp image
There are two challenges to using this library with Cloud Run:
Despite the note about auto-detection, Cloud Run is an exception. It is not yet autodetected. This can be addressed for now with some explicit configuration.
Because Cloud Run services only have resources until they respond to a request, queued up trace data may not be sent before CPU resources are withdrawn. This can be addressed for now by configuring the trace agent to flush ASAP
const tracer = require('#google-cloud/trace-agent').start({
serviceContext: {
service: process.env.K_SERVICE || "unknown-service",
version: process.env.K_REVISION || "unknown-revision"
},
flushDelaySeconds: 1,
});
On a quick review I couldn't see how to trigger the trace flush, but the shorter timeout should help avoid some delays in seeing the trace data appear in Stackdriver.
EDIT: While nice in theory, in practice there's still significant race conditions with CPU withdrawal. Filed https://github.com/googleapis/cloud-trace-nodejs/issues/1161 to see if we can find a more consistent solution.
I am trying to diagnose this error when my timer function runs. I have not found much help on this on google search
UnscheduledInvocationReason: IsPastDue, OriginalSchedule: 2019-06-13T15:13:00.0000000-07:00
It seems like the process just stops when this error comes.
Anyone have any insight on this?
The IsPastDue flag is passed to your azure function to indicate if the timer was overdue or not. A timer function can run late in some scenarios like the app service was restarted, in this case it is still invoked but the IsPastDue flag will be set to true to give your function a chance to react.
These links are helpful
Timer trigger for Azure Functions
Also, it seems that it was an issue and it is submitted on github but that was on 2017
TimerTrigger can miss IsPastDue
Got this error locally.
[2021-06-16T14:58:22.779Z] Executing 'Functions.TimerTrigger'
(Reason='Timer fired at 2021-06-16T16:58:22.7688953+02:00',
Id=adbaee54-8a3e-4983-a7e4-a73f69153e5e) [2021-06-16T14:58:22.780Z]
Trigger Details: UnscheduledInvocationReason: IsPastDue,
OriginalSchedule: 2021-06-16T16:37:00.0000000+02:00
[2021-06-16T14:59:22.614Z] Starting worker process failed
[2021-06-16T14:59:22.615Z] The operation has timed out.
Solved by clearing Blob Emulator locally:
delete
FROM [AzureStorageEmulatorDb510].[dbo].[Blob]
I have some code written to communicate with an azure service bus. It sends messages to a queue. It's in a project targeting .net standard 2.0
When I run it from a .net core terminal app it runs fine. But, when the same code is called from a .net framework 4.7.2 project then the first attempt to send a message results in the following exception after 30 to 90 seconds:
"The remote party closed the WebSocket connection without completing the close handshake."
But any further messages will be sent without problem.
// This is using Microsoft.Azure.ServiceBus, if that makes any difference...
MessageSender MessageSender = new MessageSender(ConnectionString, SendQueueName;
try
{
await MessageSender.SendAsync(new Message(Encoding.UTF8.GetBytes("Test that won't work")));
}
catch(Exception e)
{
// Error will be caught here:
// "The remote party closed the WebSocket connection without completing the close handshake."
}
await MessageSender.SendAsync(new Message(Encoding.UTF8.GetBytes("Test that will work")));
Does anybody know why the first call fails? And how to make it not fail? Or fail quicker? I've tried changing the OperationTimeout and RetryPolicy but they don'e seem to have any effect.
These first connections are via port 5671/56712, which Trend antivirus intercepts. Once these have timed out then the framework falls back to using 443, which works fine.
We tried turning Trend off and running testing the connection and its pretty much instantaneous.
I am running an azure webjobs SDK console application (continuous) with the recommended setup:
public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)
The azure queue I am running against has ~6000 messages in it and I am running the web-job locally, as a console application.
The problem I'm having is that the processing randomly stops after processing between zero and ~30 messages. The console stays open, but no more console messages are displayed.
For example, it might just process 2 messages:
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
And then, nothing. There doesn't seem to be anything wrong with my internet connection and I can't trace the issues down to any particular messages.
Has anyone else had issues with this SDK?
Update:
I made sure that I was using the right versions of all of the dependencies by removing the nuget packages and then re-running install-package Microsoft.Axure.Webjobs. I am now using webjobs version 1.1.0 which has pulled in version 4.3 of azure storage.
As recommended by Matthew, I have pulled down the source code for azure webjobs to determine where the process is freezing up. Once the freez-up occurs, I pause execution and checked the running threads for what I believe is the culprit within Microsoft.Azure.WebJobs.Host.CompositeTraceWriter
protected virtual void InvokeTextWriter(TraceEvent traceEvent)
{
if (_innerTextWriter != null)
{
string message = traceEvent.Message;
if (!string.IsNullOrEmpty(message) &&
message.EndsWith("\r\n", StringComparison.OrdinalIgnoreCase))
{
// remove any terminating return+line feed, since we're
// calling WriteLine below
message = message.Substring(0, message.Length - 2);
}
_innerTextWriter.WriteLine(message);
if (traceEvent.Exception != null)
{
_innerTextWriter.WriteLine(traceEvent.Exception.ToDetails());
}
}
}
The line it freezes on is line 66 : _innerTextWriter.WriteLine(message);
_innerTextWriter is an instance of System.IO.TextWriter.SyncTextWriter
Is it possible there is some deadlock issue with this class or the way it is being used?
Some notes:
I am running in the debugger, so in this case I believe the textwriter is forwarding to the console internally
I have my batchsize set to 1 via config.Queues.BatchSize = 1;, not sure if that could matter
I'm currently working on setting up an environment on another computer so that I can see if it is reproducible somewhere other than this machine (surface book).
Update
The issue was me not understanding how the new windows 10 command prompt works. Any time you click on the command window, it goes into "select" mode which completely pauses execution of the process.
Basically: https://superuser.com/questions/419717/windows-command-prompt-freezing-randomly?newreg=ece53f5584254346be68f85d1fd2f18d
You can tell it is in this state because it will prefix the window title with the word "Select":
You have to press enter or click again to get it going once again.
So, two final comments:
1) What an incredibly confusing and un-intuitive behavior for a command window!
2) I hope some admin will come take pity on the shame I have brought upon myself and my family by deleting this question.
To get rid of this strange behavior, you can disable QuickEdit mode:
Strange. When it is in this stuck state, can you try adding a new queue message to the queue and see if that triggers? Are you sure your function isn't hanging internally? What version of the SDK are you using? You might also try upgrading to v1.1.0 which we just released last week. If there are really a bunch of messages in the queue waiting to be processed, I can't think of anything that would cause this. The queue listener in the SDK should chug along, reading batches of messages in parallel and dispatching them to your function. Have you changed any of the JobHostConfiguration.Queues configuration knobs? You haven't force updated the version of the Azure SDK have you to something higher than the WebJobs SDK supports?
Another option if you can't figure this out might be to clone the SDK, build it and debug it locally. The repo is here. The main queue processing loop is here.