My question ,is there a way to retrieve the data from a SQL Server into an Azure ML pipeline or into the datastore ?
I am currently have a pipeline in Azure machine learning that's taking in data from the datastore and then training a model on it.
Go to Azure ML studio and click on Data -> Datastore and provide required details.
Next create dataset, Set the name and type for your data asset -> Query the database.
Go to Designer, Visualize the data in workflow
For more information refer this link.
Related
I am trying to transform data from Salesforce before loading it to dedicated SQL pool.
When I try to create a dataset from Synapse's Dataflow, I am not able to choose Salesforce a Data store:
Can anyone suggest how to transform data from Salesforce or any other Datasource that is not supported by Dataflow?
As per the Official Documentation, Currently Dataflows does not support Salesforce data as source or sink.
If you want, you can raise the feature request in the Synapse portal.
As an alternate, you can use Copy activity in the Azure Data factory to copy data from Salesforce to Dedicated SQL pool and then you can transform it using Dataflows in synapse from Dedicated SQL DB to Dedicated SQL DB.
Follow the below steps to achieve your requirement:
First create a Data Factory Workspace.
Select the Author hub and a create a pipeline. Now, drag the copy activity from the workspace and select the source. You can see that Salesforce is supported when you select new source dataset. Select it and create a linked service for that.
Now, select the sink dataset and click on Azure Synapse analytics.
Create a linked service for the Dedicated SQL database and select it.
Then, you can select the table in the Dedicated SQL and copy your data by running this.
After this copy, go to Synapse workspace and click on the Source of the Dataflow.
Select the Azure Synapse Analytics in source and click on continue.
Now, click on New to create linked service for the SQL DB. Give the subscription and server name and authenticate with your database.
After the creation of linked service, select it and give your table which is result of the copy in the DB.
Now, go to sink and select Azure Synapse Analytics and create another linked service for it as same above and select the resultant table in DB which you want after transform.
By following the above process, we can achieve the transformation from Salesforce data to Dedicated SQL DB.
Can anyone suggest how to transform data from Salesforce or any other Datasource that is not supported by Dataflow?
You can try this approach for the data stores which are not supported by the Data flows and please refer this to check various data stores supported by Copy activity before doing this process for the other data stores.
I have created an Azure Analysis Service and CosmosDB in same region (West-India).But when I try to add new Model in Azure Analysis Service, I able to find only Sample Data(Adventurous DW) in the data source list. I am unable to find CosmosDB name in the drop down list of data sources.
Microsoft has mentioned that we can connect to Cosmos DB from Azure analysis Services for In-memory model. But I can't even find Cosmos DB in the list.
https://learn.microsoft.com/en-in/azure/analysis-services/analysis-services-datasource
Followed the official tutorial:Add a sample model from the portal,you could only work with sample model which is a completed version of the Adventure Works Internet Sales (1200) sample data model. A sample model is useful for testing model management, connecting with tools and client applications, and querying model data.
But based on the statements in the supported list data source document,the cosmos db needs Tabular 1400 and higher models only.
So,please follow the Adventure Works tutorial to create a tabular model project.Then you could create a connection to your cosmos db account inn Tabular Model Explorer, right-click Data Sources > Import from Data Source.
Key your db infomation:
Also,you could refer to a case related to this:https://social.msdn.microsoft.com/Forums/en-US/9394a10b-f085-4a68-9951-5000a6f799ef/cosmos-db-data-source-how-to-configure-the-key-in-azure-analysis-services?forum=AzureAnalysisServices
Does PowerBI directly connect with Azure's ML workbench?
if yes,How?
Please respond.
Azure ML Workbench is compute tool for building and executing Data Science experiments and analytics, it is not a data store. ML Workbench would typically have data store inputs and outputs (e.g. SQL Server or Azure Data Lake). You would connect PowerBI on to the output (or input if you need) data set in order to visualise the analytics produced in the ML Workbench.
With the recent preview release of 'Data discovery & classification' for Azure SQL databases, has anybody found where this data is stored and if it can be queried directly from the Azure database? I know for on-premise databases if you right click on a database and choose 'Tasks - Classify Data...' anything you enter into that interface is stored as extended properties on the 'table/column'. However, after entering the same data via the interface in the Azure portal, there are no extended property values that I can find in my Azure SQL database. I would really like to be able to query this classification data directly so I can incorporate other metadata about the column such as data type, sample value, collation etc.
For Azure SQL DB, this metadata is stored in new attributes that have been introduced into the SQL Engine to support tagging column sensitivity, which are currently not exposed. We plan to expose them via REST/Powershell/T-SQL as the feature continues rolling out.
Please follow our announcements and the online feature documentation for updates.
Thanks,
Gilad (MSFT)
I have created an azure ml experiment which fetches data from API and updates it in sql azure database. My power bi report picks data from this database and displays the report. The data from the source is changing frequently. So I need something like a checkbox in power bi which when checked will trigger the azure ml experiment and update the database with latest data.
I know that we can schedule it to run in Rstudio pipeline but we are not thinking of this approach as it is not financially viable.
Thanks in Advance.
You could use a direct query connection from Power BI to your Azure SQL instance. Then the reports in power bi will be always up to date with the latest data you have. Then the only question is when to trigger the ML experiment. If this really needs to be on demand (rather than on a schedule) you could do that in a button in your own App. You could embed the report in your app so that you get an end to end update.
You could have a look at the Azure Data Factory (ADF), that will help you build data pipelines in the cloud.
You can use ADF to read the data from the API (refresh your data), batch-wise-score it in Azure Machine Learning, and push it directly to your Azure SQL making PowerBI always seeing the latest data which will be scored.
Take a look at the following blog where they take data through this kind of pipeline. You just have to change that the data doesn't come from Stream Analytics but from your API.
http://blogs.msdn.com/b/data_insights_global_practice/archive/2015/09/16/event-hubs-stream-analytics-azureml-powerbi-end-to-end-demo-part-i-data-ingestion-and-preparation.aspx