We receive large feed files(excel/csv) of ~5gb size every night, need to read and have to post one record after another from the file to azure service bus topic. Is it possible using Azure Data Factory, if yes, how? or any other better azure based solution available? please suggest.
Data Factory doesn't support Azure Service Bus topic. So it's impossible to using Azure Data Factory
For more details, please see : Supported data stores and formats.
I think Azure Function may can help you achieve it.
You can reference :
Azure Blob storage bindings for Azure Functions.
This article explains how to work with Azure Blob storage bindings in Azure Functions. Azure Functions supports trigger, input, and output bindings for blobs. The article includes a section for each binding: Blob trigger, Blob input binding,Blob output binding.
Azure Service Bus bindings for Azure Functions. This article explains how to work with Azure Service Bus bindings in Azure Functions. Azure Functions supports trigger and output bindings for Service Bus queues and topics
Here is a blog about Copy data from Azure Service Bus Topic to a Storage Account inside Azure.
We don't know how much time it will take. Azure Functions has two kinds of pricing plans:Consumption plan and App Service plan.
Azure Functions in a Consumption plan are limited to 10 minutes for a single execution. In the Premium plan, the run duration defaults to 30 minutes to prevent runaway executions. However, you can modify the host.json configuration to make this unbounded for Premium plan apps.
For more details, please reference: Azure Functions Premium plan.
Maybe Azure Function is not fit for this long process, but can help you execute this long process.
Hope this helps.
Related
I have been using the Azure Function App consumption plan with Azure Blob Storage V1 for some time, with no problem.
Two months ago, we changed over to Azure blob Storage V2 and all of a sudden we noticed a drastic increase in costs.
After some investigation, we found that there are several 10's of thousands of API calls for
GetBlob,
GetBlobProperties,
GetBlobServiceProperties,
ListBlobs,
PurBlob,
RenewBlobLease.
My functions do not use this particular storage account for any storage, as I am using an external account for tables.
Has anyone experienced such an issue or know why these API calls are being called?
Maybe this storage is also the AzureWebJobsStorage of your function app?
The files of your function app will be stored in the AzureWebJobsStorage.
You can follow this to check the source of the request:
How to view Blob Storage logs in Azure App Insights?
I need to move data received in my Service Bus Topic to a Storage Account inside Azure.
I believe Azure Function is one of the good ways to achieve this.
Please share suitable example if you have one.
Regards,
Surya
Fire up an Azure Functions project, create a function with a Service Bus trigger binding (if you're in the portal or Visual Studio then it'll offer you a template) and add a storage output binding for blob, queue or table as appropriate. Then in the function code just copy the relevant data from your trigger parameter to your output parameter.
I'm trying to figure out if Azure Blob Storage has similar functionality to Amazon S3. An S3 bucket can be configured in a way, that when new object is created, bucket sends message to SQS. I'm wondering if Azure Blob Storage is able to do the same with Azure Service Bus (which is kind of similar to SQS, correct?).
The only resource I've found so far, which mentions something similar is https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview, but there is no Azure Service Bus on the right side. I know I can use Functions as a proxy, but I'm interested in direct connection.
Any ideas?
Service bus(I think you compare service bus with SQS and SNS in AWS) don't have the ability to subscripe to Blob storage events. Event Grid(the link that you reffered to has Service bus support on the roadmap but no date is confirmed.
I think your best choice is Azure Functions(or Logic app if you don't want to write code) that has a blob Storage trigger to catch events and do action X.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-storage-blob-triggered-function.
Or wait a litte for event grid but you still get that "proxy" part.
One option is to use logic apps/ event grid and you can add trigger directly from azure blob storage (https://azure.microsoft.com/it-it/blog/azure-service-bus-now-integrates-with-azure-event-grid/) . Another option would be to add blob trigger with azure functions and write the code to do whatever action which you are looking for .
I deployed WorkerRole to Azure Cloud Service (classic) in new portal. With this, I also created Azure Storage account for queue.
Try to add AutoScale rule, the storage account is not listed. Tried to select Other Resource and put Resource Identifier of storage, there's no Metric name listed.
Is it by design that classic Cloud Service and new Storage account not working together?
Storage account data (e.g. blobs, queues, containers, tables) are accessible simply with account name + key. Any app can work with them.
However, to manage/enumerate available storage accounts, there are Classic-created and ARM-created accounts, each with different API's.
The original Azure Service Management (ASM) API doesn't know anything about ARM resources. There's a fairly good chance that, since you're deploying to a Classic cloud service, it's using ASM only and will not be able to enumerate ARM-created storage accounts.
If you create a Classic storage account (which has zero difference in functionality), you should be able to see it as an option for auto-scale.
I have a bit more details on the differences in this answer.
At this time, it is not possible to autoscale anything based on a new "v2" storage account. It has nothing to do with the fact that you are using the classic Azure Cloud Service. I am having the same issue with using Azure App Services. In the end, I just created a classic storage account to use for the autoscaling. There is no difference in how you interact with the different types of storage accounts.
I am trying to implement azure storage metrics code in my role but i am checking if there is easy way to get azure storage metric data about my files usage. my code is stable and i do not want to change code again.
Actually if you already have Windows Azure role running, then you don't need to make any changes to your code and you still can get Windows Azure Blob storage Metrics data.
I have written a blog about it last time as Collecting Windows Azure Storage REST API level metrics data without a single line of programming, just by using tools
Please try above and see if this works for you.
Storage analytics is disabled by default, so any operations against your storage up til now has not been logged for analysis.
You may choose to enable analytics at any time, for both logging (detailed access information for every single object) and metrics (hourly rollups). Further, you may choose which specific storage service to track (blobs, tables, queues) and which operations to track (read, write, delete). Once analytics are enabled, you may access the resulting analytics data from any app (as long as you have the storage account name + key).
Persistent Systems just published a blog post on enabling storage analytics for Java apps. The same principles may be applied to a .net app (and the sdk's are very similar).
Additionally, Full Scale 180 published a sample app encapsulating storage analytics (based on REST API, as it was written before SDK v1.6 came out).