I'm trying to find any way to publish my console app (.net) written for azure data factory v2.
But could not find any solution.
More details would be really appreciated but if you mean that you are using the .NET SDK to create ADF V2 objects, my understanding is that there is no such thing as publish compared to the new User Interface in the portal where you create/edit the objects first and then you click on publish.
if you use the library they get automatically uploaded to ADF V2 and you can easily test that now with the new UI.
It would be useful to have a bit more info on your context. You're talking about running a custom activity from an Azure Batch account? What did you try already?
When running a custom activity, you'll have to upload your executable + depedencies to an Azure storage account. Create a blob container and copy the files there. Then you'll have to configure the activity to use this storage account and the point it to the right container.
If you're asking for a deployment like a right-click -> deploy option, it doesn't exist. I've automated my deployments using PowerShell, using Set-AzureStorageBlobContent to write the files to the Storage account.
Related
I have created Azure Data Factory with Copy Activity using C# and Azure SDK.
How can deploy it using CI/CD ?
Any URL or link will help
Data Factory continuous integration and delivery is now possible with directly through the web user interface using ARM Templates or even Git (Github or Azure DevOps).
Just click on "Set up Code Repository" and follow the steps.
Check the following link for more information, including a video demostration: https://aka.ms/azfr/401/02
One idea that I got from Microsoft was that using the same Azure SDK you could deserialize the objects and save down the JSON files following the official directory structure into your local GitHub/Git working directory
In other words you would have to mimic what the UI Save All/Save button does from the portal.
Then using Git bash, you can just commit and push to your working branch (i.e. develop) and from the UI you can just publish (this will create an adf_publish release branch with the ARM objects)
Official reference for CI using VSTS and the UI Publish feature: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment
Unfortunately, CI/CD for ADF is not very intuitive at first glance.
Check out this blog post where I'm describing what/how/why step by step:
Deployment of Azure Data Factory with Azure DevOps
Let me know if you have any questions or concerns and finally - if that works for you.
Good luck!
My resources on how to enable CI/CD using Azure DevOps and Data Factory comes from the Microsoft site below:
Continuous integration and delivery (CI/CD) in Azure Data Factory
I am still new to DevOps and CI/CD, but I do know that other departments had this set up and it looks to be working for them.
Is it possible to trigger an VSTS/TFS build based on the condition if a blob in a storage account is updated?
I tried to create a function app, but with little to no result, I cannot trigger a VSTS/TFS build.
Easiest solution (from my understanding) will be to use a logic app:
Create a logic app that is triggered whenever a blob is added/updated into a specific container.
Queue a VSTS build
NOTE
your VSTS account should have "Third-party application access via OAuth" enabled. (Go to Administration > Control panel > Settings page)
According to the official docs, it is possible to:
...start a function when a new or updated blob is detected. The blob contents are provided as input to the function.
Then, the only thing the Azure function should do it to queue a build in VSTS with the help of REST API. You might also want to check the Getting Started page of VSTS REST API docs - it contains the basic samples to quickly get up to speed.
I have multiple pipelines in adf. How to monitor them using Program, to get the status of all the pipelines. The difficulty I am facing, I am not sure where to find the applicationID and authenticationKey for that DataFactory.
You will have to generate the application ID and application key by creating a new application in Azure active directory. Once created then you have to assign the role to that app.
you can go through following link to get generate the application id and application key: Create azure active directory application
For monitoring, there are various ways to monitor the pipeline by using the the powershell script, .NET SDK or Azure portal.
Please check the following link to monitor pipeline using .NET SDK:Monitor and manage ADF v1
Hope this helps,
Jai
We are trying to upload the artifact to blob storage from TFS build server. AzCopy task needs the azure subscription details, which is not available to us. We need to upload the artifacts to azure blob storage using azure blob storage connection string. Is there a way to upload files to blob storage using connection string only.
Anything you can do from PowerShell you can do from build and release. There is a task named "PowerShell" and one named "Azure PowerShell". If you don't have the Azure subscription details I doubt you will be able to use the "Azure PowerShell" task. However, if you have a PowerShell script you run locally that works you might be able to simply run it as part of your build with the "PowerShell" task.
Option two is have someone that knows the details to create an Azure Service Endpoint for you. They never have to share the details with you to make the connection. Once the connection is created you can use it without having to ever know the details.
Is it possible to automatically publish (static) files from Team Foundation Service to an Azure Cloud Storage Blob during build?
UPDATE:
If it isn't possible to publish directly to a blob, perhaps there is a good workaround to publish indirectly? I also publish another project to an azure website, so one way is to publish the files to that website and then using WebActivator I can move the files to the blob at first activation. But I hope for a better solution to this.
I think you would need to be able to customize the build process in order to do that, and that customization is something I don't believe you can do today with Team Foundation Service. If using Team Foundation Server, then yes.