For a few weeks there is an issue with Azure Notification Hub - we have a backend service that registers devices & schedules notifications. When one of the notifications is triggered, then we send it to Azure based on the device type (create the payload and all of that).
The response we get from azure is positive, the notification has been enqueued and is ready to be sent. However, it seems that Azure does not send them all. No error is thrown but only a small percentage is sent. It also does not seem to depend on the device type.
There are no logs to read - or none tha I can find and I cannot pin point the issue.
We have an endpoint that triggers the sending of a notification - we did a test run with 30 notifications (similar and different payloads for the same device) and it all went well.
Not sure how to troubleshoot this or what might cause this. I understood from the documentation that the device might be inactive and then the notification is deleted from the queue after a while. But it seems that the said device receives 1 out of 3 notifications (sent at the same time).
Any feedback would be appreciated.
Edit update:
After upgrading to the Standard plan, we managed to get some more information and it looks like Azure gives back a "Success" with a count of "1" but the device does not receive the notification - the problem is, this happens randomly. It's not always the same device that does or does not receive the notification.
Sorry this isn't a super helpful answer, but could you please open a support ticket in the Azure Portal for us to investigate your particular case? This does not seem like expected behavior so the team will have to look into the logs for your namespace and hub.
One immediate thought - there is de-duplication logic when sending notifications. So within a period of time (something like 15 minutes) if the same notification payload is being sent to the same device, Notification Hubs will automatically drop that duplicate notification.
Related
We have integrated MS Teams with Azure DevOps using below:
https://learn.microsoft.com/en-us/azure/devops/pipelines/integrations/microsoft-teams?view=azure-devops
Pipelines have been subscribed to send notifications on failure. However they stop working randomly. We are unable to identify where the problem is. Could someone guide me on how to troubleshoot the issue?
Here is a troubleshoot advice:
Please go to your Project Settings -> General/Service hooks and check the sevice hook to Microsoft Teams.
If you find that the notification was failed to send, it will usually provide an error message for further investigation.
If you find that it doesn't send notifications when the pipeline fails, it is recommended to check your connection settings. And identify the scope of the problem, such as whether this problem is related to a particular pipeline.
If you find that the notification has been successfully sent but the Microsoft Teams have not received it, you can further check if the intermittent problem is time related. For example, whether the problem can be repeated over a certain period of time and whether there is a significant delay between the successful notification sent time and the actual delivery time.
Question
Does MsGraph give a lower priority to change notifications when running a subscribed webhook locally? If not, I'm open to hearing any theories as to why this extra delay (from MsGraph to the webhook) could be happening on the local machine, but not when running it in their cloud.
Some background info
When running an Azure Function webhook locally through ngrok (reverse proxy), there's considerable delay in MsGraph sending change notifications to the webhook. Once ngrok receives a notification, it's passed on almost instantly to the correct API call. Calling the ngrok url manually has an near-instant response time. The problem is in Microsoft Graph not sending the notification updates at all (or tens of minutes to hours later).
Forging my own change notifications and feeding them to an Azure Queue for now, but I'd rather just receive live updates to properly test the different paths.
I'm subscribing to the 'messages' change notifications.
Edit: One could work around the delay by forging notifications and manually feeding them to a queue.
I need the Notification Message Id that is supposed to be returned from the azure notification hub when sending the message.
I am sending the message from the app backend using the SendGcmNativeNotificationAsync method.
NotificationOutcome outcome = await hub.SendGcmNativeNotificationAsync(notif, tags);
After sending the notification, the outcome.NotificationId property is empty or null. How can I get the Notification Id?
Although the outcome.Result[0].Outcome received has value
"The Notification was successfully sent to the Push Notification System"
but the notifications are not delivered to the registered device.
Also I get a tracking id in outcome.trackingid how can I use it to track the message.
I am using free version of azure portal.
Please suggest
Thanks in advance!
It took me quite some time to figure this out, because it is not clear documented, but the answer is simple: NotificationOutcome.NotificationId is always null when Azure pricing tier is not S1 (Standard).
To figure out the cause of your problem, temporary switch to the S1 pricing tier. Then you will get the NotificationId and Azure will register Per Message Telemetry.
In theory, you should be able to get the results by calling GetNotificationOutcomeDetailsAsync with the NotificationId to get more error info. It can take up to 30 minutes (!) before this call succeeds (if you call it too soon you get a MessagingEntityNotFoundException).
Unfortunately, the result State is mostly Completed, even though the message had a wrong or expired device ID.
The results in the Metrics shown in Azure for your Notification Hub are correct, but I did not find a reliable way yet to retrieve the results on a Per Message base.
Don't forget to switch back to the F1 (Free) pricing tier!
I am currently working on an application built in VS using a Xamarin Unified IOS project. one requirement of the application is the utilisation of push notifications. The issue is that our test device/application registers in the first instance without any issues, we can see the registration events in azure portals, and the azure service bus. I have also confirmed the initial registration using the 'Service Bus Explorer' application and the server explorer feature in VS.
The issue arises when sending the test push notification. According to all the logs I have access too the message is successfully pushed to apples APNS server and I can't see any failures being returned as part of the PNS request. However once this completed the device registration is removed and the notification never arrives on the test device.
My understanding is that the azure application hub handles PNS rejections itself in order to tidy up device registrations. This in and of itself is not a terrible idea, but due to a lack of logging, it's very hard to determine the root cause. I have done a ridiculous amount of googling for this one, and a lot of people suggest an issue with the apple certificates that are used. I have deleted, recreated, resigned, verified, both APNS certs and Provisioning profiles until I ran out of combinations. None of that has resolved the issue.
To make the issue more confusing, if I use a service such as 'http://pushtry.com' or 'http://pushwatch.com' and upload the certificate and device token, it will allow me to successfully push a notification to the device.
Am I missing something glaringly obvious? or is the notification hub absurdly difficult to setup and debug for APNS?
Any help you can provide will be appreciated, I would publish some code, but as I know that the device registers the correct device token, I don't see it necessary at this point, but please feel free to tell me otherwise. Many thanks.
What's also important to note, if you are using Azure Notification Hubs APNS, is that it will unregister devices if the application mode is not set to in production. Once you have deployed to the app store, you need to switch the "application mode" in your Azure Notification Hub from sandbox to production. Otherwise your device registrations will be unregistered.
I'm using the same token/key for both sandbox and production with no issues.
After many man hours and through luck rather than any kind of logical process, we determined that 'Azure Notification Hubs' have an in built validator for the apple device token that's passed up to the APNS. So even though there are multiple examples and answers out there that suggest submitting a device token without spaces or symbols to the notification hub, this is in fact the wrong approach. The device token should always be submitted to the 'Azure Notification Hub' in an 'as is' state, without any validating from within the Xamarin.IOS application. Although this seems like such a simple thing to overlook, there is a lot of contradictory evidence out there on google, and Azure's documentation says very little about the format of submitted device tokens. Anyway hopefully this will help someone else someday.
-- Edit --
Device Token format from Apple: < XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX >
Format that many online resources suggest it is passed to the notification hub as: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
without spaces or the '<' and '>'
However it should be passed through to the notification hub as it is received from Apple.
On the pricing page for Azure Notification Hubs:
https://azure.microsoft.com/en-us/pricing/details/notification-hubs/
It states that "Standard namespaces have access to Per Message Telemetry and Push Notification Services Feedback". I am using a Standard namespace.
The app is sending template based notifications to an iOS app from my back end using the C# Notification Hub client. For example:
NotificationOutcome result = await hub.SendTemplateNotificationAsync(params, tags);
The result.State is coming back as Enqueued, but some notifications are not being delivered and I am trying to figure out why. (There are no bad registrations in the hub - I have already checked that.)
In the Azure portal, I can see aggregate metrics of messages received by the hub, errors, etc. - but I cannot seem to view the specific results of a push message to determine why it failed to reach the device, or even if it was successfully handed off to APNS.
How can I see the results of an attempted push? Is this even possible with Azure Notification Hubs?
Did you try Fiddler+https://msdn.microsoft.com/en-us/library/azure/mt608135.aspx ?
Also, as far as i know, Apple does not guarantee the delivery of all notifications.
1) Is Apple's push notification service reliable?
More of that,
2) https://azure.microsoft.com/en-us/documentation/articles/notification-hubs-faq/
"A PNS does not guarantee any SLA for delivering notifications; however, typically a vast majority of push notifications are delivered to target devices within a few minutes (usually within the limits of 10 minutes) from the time they are sent to our platform. "