I Googled a bit but question remains: Can Azure notebooks call Azure Functions?
Seems Azure Function dont play well with Data Bricks/Notebooks?
I can think of a way to integrate Azure Functions <--> Data Bricks/Notebooks
My Azure Functions can persist calculation result in SQL server for example. Notebooks simply read from it.
I am however unsure if my approach is most appropriate.
Many thanks
You can use Azure Data Factory to orchestrate your pipeline; ADF can trigger both Databricks Notebooks as well as Function Apps. You can also pass outputs from the Databricks Notebook into the Function App.
Related
I have a requirement to parse a lot of small files and load them into a database in a flattened structure. I prefer to use ADF V2 and SQL Database to accomplish it. The file parsing logic is already available using Python script and I wanted to orchestrate it in ADF. I could see an option of using Python Notebook connector to Azure Databricks in ADF v2. May I ask if I will be able to just run a plain Python script in Azure Databricks through ADF? If I do so, will I just run the script in Databricks cluster's driver only and might not utilize the cluster's full capacity. I am also thinking of calling Azure functions as well. Please advise which one is more appropriate in this case.
Just provide some ideas for your reference.
Firstly, you are talking about Notebook and Databricks which means ADF's own copy activity and Data Flow can't meet your needs, since as i know, ADF could meet just simple flatten feature! If you miss that,please try that first.
Secondly,if you do have more requirements beyond ADF features, why not just leave it?Because Notebook and Databricks don't have to be used with ADF,why you want to pay more cost then? For Notebook, you have to install packages by yourself,such as pysql or pyodbc. For Azure Databricks,you could mount azure blob storage and access those files as File System.In addition,i suppose you don't need many workers for cluster,so just configure it as 2 for max.
Databricks is more suitable for managing as a job i think.
Azure Function also could be an option.You could create a blob trigger and load the files into one container. Surely,you have to learn the basic of azure function if you are not familiar with it.However,Azure Function could be more economical.
I've got some image processing code that I need to run in Azure. It's perfect for an Azure Function, but unfortunately requires a component with a complex installation procedure and therefore will need to run in a VM.
However, I'd like to make it behave much like an Azure Function, and trigger whenever new items arrive in blob storage.
My question is: Does Azure provide me with any handy way of doing this, or do I have to write code that polls the blob storage looking for new items?
Have a look at Azure WebJobs SDK. It shares API model with Functions, but you can host it in any .NET application. Blob Trigger.
I need to upload WEB API response files into Azure Data Lake.
Then I have to dump those files into Azure SQL tables.
Above both processes must be scheduled to execute on hourly basis.
Should I use Azure Web Jobs or Azure Function.
Azure Data Factory is probably the better mechanism to drive this recurring hourly pipeline. More details here.
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-scheduling-and-execution
If you are running Azure Functions on Consumption plan, the function call must complete within 5 minutes, which might be not enough for big data sets.
For the rest Functions and Web Jobs are similar for your scenario. Functions are actually running on top of Web Jobs. And if you don't need any advanced features of Functions (e.g. bindings), I would go for a Job.
Does Azure ML only provide output through it's web services?
Is it possible to feed the output to an Azure SQL database?
Is it possible to feed the output to a Redshift database?
Essentially I am looking to know if I can integrate Azure ML Studio with our existing redshift analytics database.
yes you can write to SQL DB in Azure.
you can also use a Python module to make REST calls so in theory you can write to Redshift.
Writing to SQL DB is possible in Azure ML and so is Writing directly to Azure Blob Storage.
However, unlike #Hai, I do not believe you can write to a Redshift DB since it is clearly stated by the "Python Module" documentation from Microsoft that the Python execution is Sandboxed and therefore can not access resources outside the virtual machine it runs on(i.e Internet resources, on-premises resources, ...)
A simple question: Can this be achieved directly? I mean without the Azure blob storage in between (as showed in all the examples)? Can someone provide some code example please.
yes, you can do this directly. In fact, you can do direct copies from any of our supported sources/sinks, you don't have to pass through blob. To go from on-prem SQL Server-->SQL azure, you will need to setup a Data Management Gateway connector on your on-prem server. Then, you use a linked service of type AzureStorage and an output dataset of type AzureSQLTable as the output dataset, instead of AzureBlob as is shown in the example. The exact steps to setup the DMG and the JSON code for the linked services, datasets, and pipelines can be found in our documentation. We are also improving our UI in the near future to make these kinds of copy setups an easy code-free experience.
https://azure.microsoft.com/en-us/documentation/articles/data-factory-sqlserver-connector/