The remote server returned an error: (404) Not Found While deleting message from queue - azure

We are using Azure Queue for our printing job but when deleting message from queue by queue.DeleteMessage(message), the method throws below exception.
The remote server returned an error: (404) Not Found
Above exception was handled but still looking for workaround.
Can anyone please suggest how to fix it.
Thanks,
Sneh

According to this article, we can find that:
After a client retrieves a message with the Get Messages operation,
the client is expected to process and delete the message. To delete
the message, you must have two items of data returned in the response
body of the Get Messages operation:
The message ID, an opaque GUID value that identifies the message in the queue.
A valid pop receipt, an opaque value that indicates that the message has been retrieved.
If a message with a matching pop receipt is not found, the service returns error code 404 (Not Found). And Pop receipts remain valid until one of the following events occurs:
The message has expired.
The message has been deleted using the last pop receipt received
either from Get Messages or Update Message.
The invisibility timeout has elapsed and the message has been
dequeued by a Get Messages request. When the invisibility timeout
elapses, the message becomes visible again. If it is retrieved by
another Get Messages request, the returned pop receipt can be used
to delete or update the message.
The message has been updated with a new visibility timeout. When the
message is updated, a new pop receipt will be returned.

I ran into this issue today and the root cause was ownership issues between two different queues. We had setup two queues, one to hold our message awaiting processing and one for messages that had errored out. The problem came with the logic of how the message was moved between queues.
If our processing failed, we would perform the following logic:
_errorQueue.AddMessage(msg);
_queue.DeleteMessage(msg);
The DeleteMessage would also return a (404) Not Found because the msg had been moved to the errorQueue. There were two solutions that I found to this issue:
1. Switch Logic
If you switch the logic than the msg will be deleted before being added to the errorQueue which will avoid the ownership swap.
_queue.DeleteMessage(msg);
_errorQueue.AddMessage(msg);
2. Insert Copy of Message
Solution #1 has the potential to lose the message if something happens between deletion and insertion (small chance but a chance nonetheless). The solution I went with inserted a copy of the msg with the same payload so it didn't run into this ownership issue because it was a different object.
_errorQueue.AddMessage(new CloudQueueMessage(msg.AsString));
_queue.DeleteMessage(msg);
Debugging Tip
One useful tip I encountered while debugging it making sure the exception your catching isn't the default Exception. Catch the StorageException instead to get access to Azure Storage related error information.
try
{
_queue.DeleteMessage(msg);
}
catch (StorageException ex) //use this instead of base Exception
{
var info = ex.RequestInformation; //has useful information
}
If can provide more information to help you debug your real issue.

Related

How do we check a message that caused a SessionLockLostException error in the service bus?

I am getting a few sessionlostlock exceptions while sending some messages to our SB triggered function app. We've set up alerts to check whenever such an error occurs. Although these messages do get retried and there no messages are sent to the DLQ. How do I find out which messages gave these exceptions in the first place.
How do I find out which messages gave these exceptions in the first place?
According to documentation:
You can use MessagingException.Detail Property to get the detail information on the messaging exception.
public Microsoft.ServiceBus.Messaging.MessagingExceptionDetail Detail { get; }
References: Cause and resolution of SessionLockLostException, Class SessionLockLostException and SessionLockLostException(String, Exception)

Run a function when a ClientEmitter has an error

I'm using an event handler, where each event code is in it's own files. I'm attaching the events to the client, so when that file has the event emitted, it will run that code:
// looping through all event files
client.on(file.split('.')[0], require(`events/${file}`).bind(null, client, Util);
If the event file was message.js, it would be similar to:
client.on('message', require('events/message.js').bind(null, client, Util);
So when the message event is emitted, it runs the message.js file, passing along the client and Util classes.
I also have a function that is attached to the client called report. It basically reports when there is an error. I would like it so whenever any event from the client has an error, it will run that function.
I've done this slightly with the commands: command.run(...).catch(error => client.report(error)).
Is there a similar way to do this, instead of having to put a try-catch around all code in all the event files?
Try this way
client.on('error', require('events/report.js').bind(null, client, Util);
Error handling should be context driven. This means your bot's response to the error should be dependent on what it was doing, in what channel, etc - both for debugging and for the end user's information on what happened. You'll miss out on all of the context by letting errors just travel all the way up into an uncaught exception, and without the ability to create an error message, the user will just see the bot not respond and think it's down or the command is broken.
My suggestion: Create helper methods for your most common error producing functions that wrap them with error handling. I did this mostly for sending messages, as there's a myriad of things that could cause a message send to fail out of your control and the handling consists of generating an error message and attempting to send it in the channel or DM it to the user if that fails.

Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - inequivalent arg 'x-max-priority' for queue 'xyz'

While running the app in consumer mode, my application is frequently crashing with an error Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - inequivalent arg 'x-max-priority' for queue 'xyz' in vhost '/': received the value '10' of type 'byte' but current is none". Does anyone know the fix for this issue?
#here Deleting the queue 'xyz' worked for me.
It appears that you're making a call to a server that requires one or more preconditions to be met before it will return a result. Since this is a remote server call, you'd need to check the API documentation to determine what precondition(s) is/are required, or if there is no API documentation, contact the manager of the server in question.
See 406 HTTP "Not Acceptable" Status Code for more about this particular error.
While the exception of assertQueue still exists, but I found that I don't need to run assertqueue before consuming .. so for now I just commented the assertQueue call. and consuming is working.
I don't know why all examples of ameplib run assertQueue before anything?!
406 ERROR also may occur if the queueNamed channel is acknowledeged more than once, Make sure channel ack type is set to manual, {noAck: false}. this closes the queueChannel which causes the consumer/publisher to not be able to access the queueChannel.
Note: Make sure you ack the queue in the same order it is received by the system. If your handling many publishers and consumers in single functionality also check if you have pre acked the channel.

How to stop an Azure WebJobs queue message from being deleted from an Azure Queue?

I'm using Azure WebJobs to poll a queue and then process the message.
Part of the message processing includes a hit to 3rd party HTTP endpoint. (e.g. a Weather api or some Stock market api).
Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??
If I continue .. then I assume the message will be deleted by the WebJobs SDK.
How can I:
1) Say to the SDK - please don't delete this message (so it will be retried automatically at the next queue poll and when the message is visible again).
2) Set the invisibility time value, when the SDK pops a message off the queue for processing.
Thanks!
Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??
The Webjobs SDK behaves like this: If your method throws an uncaught exception, the message is returned to the Queue with its dequeueCount property +1. Else, if all is well, the message is considered successfully processed and is deleted from the Queue - i.e. queue.DeleteMessage(retrievedMessage);
So don't gracefully catch the HTTP 500, throw an exception so the SDK gets the hint.
If I continue .. then I assume the message will be deleted by the WebJobs SDK.
From https://github.com/Azure/azure-content/blob/master/articles/app-service-web/websites-dotnet-webjobs-sdk-get-started.md#contosoadswebjob---functionscs---generatethumbnail-method:
If the method fails before completing, the queue message is not deleted; after a 10-minute lease expires, the message is released to be picked up again and processed. This sequence won't be repeated indefinitely if a message always causes an exception. After 5 unsuccessful attempts to process a message, the message is moved to a queue named {queuename}-poison. The maximum number of attempts is configurable.
If you really dislike the hardcoded 10-minute visibility timeout (the time the message stays hidden from consumers), you can change it. See this answer by #mathewc:
From https://stackoverflow.com/a/34093943/4148708:
In the latest v1.1.0 release, you can now control the visibility timeout by registering your own custom QueueProcessor instances via JobHostConfiguration.Queues.QueueProcessorFactory. This allows you to control advanced message processing behavior globally or per queue/function.
https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/MiscOperations/CustomQueueProcessorFactory.cs#L63
protected override async Task ReleaseMessageAsync(CloudQueueMessage message, FunctionResult result, TimeSpan visibilityTimeout, CancellationToken cancellationToken)
{
// demonstrates how visibility timeout for failed messages can be customized
// the logic here could implement exponential backoff, etc.
visibilityTimeout = TimeSpan.FromSeconds(message.DequeueCount);
await base.ReleaseMessageAsync(message, result, visibilityTimeout, cancellationToken);
}

SoapHttpClientProtocol automatically retry after exception?

I am just curious about this. I am making a change in this project, that is using NetSuite web service, and sometimes it throws a SoapException at random, "Only one request may be made against a session at a time".
private NetSuiteService _service;
SessionResponse response = _service.login(passport); //if SoapException, retries??
Status status = response.status;
Reference.cs:
public partial class NetSuiteService :
System.Web.Services.Protocols.SoapHttpClientProtocol
My question is: If I am in debug mode, I trace this, and I hit F5, and it seems to automatically retry after exception is thrown (the code keeps running, with no try catch block implemented, no while loop) until successful (status.isSuccess == true). When I run it in release mode, as a windows service, the log shows it stops running after exception is thrown.
How is this possible? Is it better to catch this exception in a try catch block and retry?
NS Server refuses a request if its already processing one from the same user.
If you want to make sure that your request succeeds than you have to catch this exception and retry.
This was not the experience I had. We thought this related to netsuite sessions but turned out to be nothing to do with that at all and in fact was not even hitting netsuite (according to netsuite log)​​. Turned out we were trying to execute too many commands in a single request and it totally refused to send it to netsuite. Never seen this error before, may be it is a new thing with the new version!

Resources