If I want to host a self hosted integration runtime to process data on my local network, and just use ADF a data manipulation tool rather than something that submits data to the cloud, will it ever be sent up to Azure to be processed?
It appears to be that the data would solely be processed on the nodes of the Self Hosted Integration Runtime, but wanted to confirm.
Reference: https://learn.microsoft.com/en-us/azure/data-factory/create-self-hosted-integration-runtime?tabs=data-factory
Azure Data Factory including Azure Integration Runtime and Self-hosted
Integration Runtime does not store any temporary data, cache data or
logs except for linked service credentials for cloud data stores,
which are encrypted by using certificates.
The Self-Hosted IR is Hybrid Scenario. You can implement additional security features like Storing credentials in Key-Vault, using different ports, different data encryption techniques (IPSec VPN or Azure ExpressRoute) during transit.
Refer: Security considerations for data movement in Azure Data Factory
Related
is it possible to use Azure Data Factory on-premise without letting the data run through the cloud? I know Talend got a prodcut, where the data is transfered only on our machines and not in the cloud.
Read documentation on Microsoft.com but didnt find any useful information
You may use a self-hosted integration runtime to transfer data entirely through your on-premises infrastructure, as long as both the data source and sink are on-premises.
However, the control flow will still happen through the cloud, even if the data itself never leaves your data center. For this reason, setting up a self-hosted integration runtime will still require outbound network access from your infrastructure to Azure.
Check out this piece of documentation for more information: https://learn.microsoft.com/en-us/azure/data-factory/create-self-hosted-integration-runtime?tabs=data-factory#command-flow-and-data-flow
May I ask what is the security protocol (Https/TCPIP etc) applied in the following scenarios in Azure? I need these details to write my design document.
Between Azure Services
Azure Data Factory interacting with Azure Storage
Azure Databricks interacting with Azure Storage
Azure Python SDK connecting to Storage Account (Is it TCP/IP ?)
If there is any support page in MS Azure, please direct me there.
Inside the Azure data centers used TLS/SSL for communication between
services and you can read about it "Encryption of data in transit"
section on this page.
The main SDK implementations are wrappers around the REST API and
Python SDK is one of them.
I'm following the instructions to set up App Insights to spool to SQL using Azure Stream Analytics, but I'm trying to deviate slightly to use an on-premise SQL server (that the web application already uses) over VPN.
At the point of adding the output, this is failing with:
Is it the case that IP addresses are not supported, or is it something more fundamental than that?
You are probably looking for answers directly to your question, which Jean-Sébastien answers succinctly. But an alternative architecture, if you haven't considered it already...
You could stream to a transient Azure SQL Database or Blob storage (likely cheaper depending on your workload), and then use Azure Data Factory tunnelled via a Self-Hosted Data Factory Integration Runtime to "send" the data back to on-premise SQL.
Data Factory V2 also has blob triggers, so rather than needing a schedule it could pickup any new blobs in micro batches.
I say "send" in quotation marks as the Integration Runtime actually creates an outgoing connection to from on-premise to Azure, yet gives the capability for push-like data transfer.
If data factory proves useful, here is a guide creating copy pipelines: https://learn.microsoft.com/en-us/azure/data-factory/tutorial-hybrid-copy-portal
Albeit this guide is for on-prem sql to blob, but it gives you a stronger starting point.
At this time only Azure SQL Databases are supported in Azure Stream Analytics.
Sorry for the inconvenience.
Thanks,
JS (Azure Stream Analytics)
I write an application using cocos2d-x. Now I want to store some data in the Windows Azure Storage and get the data sometime, how can I do that?
As written, it's difficult to answer such a broad question. Having said that: I'll do my best to give you an objective answer describing Azure's storage options from a service perspective.
Azure Mobile Services. This lets you have a CRUD interface to storage, and is build to provide a REST-based API, which fronts storage. It defaults to SQL Database, but you can easily override this by creating your own custom API and using server-side JavaScript / Node.js to read/write to any storage system.
Azure blobs/tables/queues. This is the collective set of Azure large-scale storage, with up to 200TB per account namespace. You can access storage directly from your game, or through your own service tier - that's up to you. You need to worry about security, as you don't want to have your blobs exposed as public unless you want to. Fortunately you may use something called a Shared Access Signature to grant access to your app, while keeping these resources private to the rest of the world.
SQL Database. Azure provides database-as-a-service, largely compatible with SQL Server. As long as you have a proper connection string, it's just like having a local database.
3rd-party hosted solutions. There are companies that host data services in Azure, such as ClearDB (MySQL) and MongoLab (MongoDB).
One other option: Custom database solutions. If you're not using a built-in or 3rd-party storage service, you can always install a database server within a Virtual Machine. You're now managing the server, but this would give you ultimate choice.
I have a web role that talks to Azure Storage, Azure Shared Cache Service and Azure SQL Databases. It is only ever the web roles that communicate with these storage mediums, and never the client browser. The Azure Table Storage contains sensitive data, but the cache and SQL databases do not.
Question is, if all data access goes over plain HTTP, is there a risk that someone can intercept my packets, and read my storage key? If so, who can sniff these packets - just Microsoft employees, or do I need to worry about other Azure tenants that might have effected a jailbreak?
A few things to consider:
If your webrole and storage accounts are in the same data center, then the traffic is contained within data center. In that case, going of HTTP would not create any problems IMO. However if the webrole and storage accounts are in different data centers, then definitely make use of HTTPS.
Since you never send your storage account key with your requests to storage, you can be assured on that part. What you do is sign the requests using your key (or the storage client library does) and send that signature as a part of your requests. I don't think one would be able to reverse engineer that signature to get your storage account key.
HTH.
In addition to the previous answers, you should also take a look at the official security whitepaper: Windows Azure Security Overview. It talks about how isolation and packet filter secure the communication in the datacenter.