I am reaching out to you gather best practices around ingestion of data from various possible API's into a Blob Storage. I am considering to interface with all the various possible API's using Data Factory and here are the possible set of API's which I have currently:
Ingesting from an API service within the same Resource Group in Azure Cloud
Ingesting from an API service from a different resource group but within the same Azure subscription
Ingestion from an API service from a different Azure Subscription / VNET
Ingestion from an API service available publicly such as Twitter, Facebook
Ingestion from an API service which is available on-premises
Any other possible API services
My questions around the above API services are:
a) What are the specific security related settings I need to take care in order to interface with the above API's (Managed Identity, Service Principal etc.)
b) When to use which security setting ?
c) Along with Azure Data Factory, is there any other Azure service which can be leveraged for the above ingestion from the API's
d) What are the specifics of Runtimes / Linked services which I should be taking care about e) Any specifics around AAD resource and Authentication type
Just wanted to add that for #5 , you will have to use SHIR ( Self hosted IR ) . Please read about this here .
for c) Along with Azure Data Factory, is there any other Azure service which can be leveraged for the above ingestion from the API's
Logic Apps has connectors backed in for many existing API's
Related
May I ask what is the security protocol (Https/TCPIP etc) applied in the following scenarios in Azure? I need these details to write my design document.
Between Azure Services
Azure Data Factory interacting with Azure Storage
Azure Databricks interacting with Azure Storage
Azure Python SDK connecting to Storage Account (Is it TCP/IP ?)
If there is any support page in MS Azure, please direct me there.
Inside the Azure data centers used TLS/SSL for communication between
services and you can read about it "Encryption of data in transit"
section on this page.
The main SDK implementations are wrappers around the REST API and
Python SDK is one of them.
Does anyone know if there is a cost associated to the Azure API egress calls. We are calling the Azure API from external sources outside azure to extract data from the Azure resources. Wanted to know if there would be a cost of pulling the data from Azure via API.
We are not using Azure API management, this is a custom solution which is pulling data from Azure.
Thanks very much for the help.
Cheers
I am trying to access logs in Azure blob storage. Here I saw I can use both Resources API and Storage Services API to access them.
Can somebody please explain the advantages and disadvantages of using one over another.
That APIs have different purpose:
Azure Storage Resource Provider REST API we should use for managing storage like a resource. You can see all your resources in this link: Azure resources
Azure storage Service REST API we use for managing like a service. If you take a look that have a different method to manage.
Also, you can give to user different roles for user. And you have a to control who can do and what.
I'm trying to monitor all our resources in a single place and after reading the msdn pages on monitoring web apps and cloud services and azure sql databases i can't seem to understand how to query azure (thorugh the SDK or powershell) to give me the same data i can see in the azure monitoring page.
is there some programattic way to get this data?
I'm afraid there isn't an easy answer to this. There isn't one API where you can fetch all the data: it's mostly service specific.
A couple of pointers:
Metrics for web sites, web/worker roles and VMs can be accessed through the metrics API. See here: https://convective.wordpress.com/2014/06/22/using-azure-monitoring-service-with-azure-virtual-machines/
Metrics for SQL Database are available through the sys.resource_stats and sys.dm_db_resource_stats DMVs.
Windows Azure Diagnostics (for web/worker roles) performance counters can be fetched though the table storage API, in the wadperformancecounterstable table.
There are probably API's for other services as well. I've built a tool myself that fetches data from a couple of these services, and can both plot it and expose it through a unified API, see: https://github.com/WadGraphEs/AzurePlot
I am trying the read the performance metrics of vms in the azure cloud using the rest api. As per the this link ( azure documentation ) http://www.windowsazure.com/en-us/documentation/articles/cloud-services-how-to-monitor/#accessverbose
but when i try to fetch the tables that are available for a particular from a storage account i am not getting empty table list.
Please correct me what the ways we can capture vm performance metrics using the rest api to integrate with 3rd party system management providers.