When deploying a dataflow job with terraform it does not drain the old job.
Is there a possibility to automatically drain it and the deploy?
I think it's not the responsability of Terraform to deploy custom Dataflow jobs (not Dataflow templates).
It's more the responsability of the deployment task proposed by a CI CD pipeline.
In this case there is no possibility to automatically drain and deploy a new version of the job.
You have to develop your own script (Shell for example) to do that and apply your strategy.
For example, a Dataflow job can be drained by a gcloud command :
gcloud dataflow jobs drain JOB_ID
Related
In our requirement we need to connect to ARGO instance and take necessary actions on the jobs.
So we are consuming ARGO java apis to connect to argo instance and performing the actions like suspend,resume,retry....
But i am not able to find apis for stopping/terminating cron jobs.
So how can i stop/terminate the cron jobs ?
There is no direct API for stopping/terminating cron jobs in ARGO. However, you can achieve the same using the ARGO API for getting the list of jobs and then deleting the job using the API for deleting a job.
I'm new to GCP and trying to gain a better understanding of ETL services offered by Google Cloud. I'm trying to do a POC where a dataflow job runs whenever a file arrives at a specified location in GCS, I want to utilize Cloud Composer for the same.
How can I utilize different airflow operators (one to watch for file in GCS and other to trigger dataflow job) in a DAG?
Create a Cloud Function triggered when a file is created / arrives at your bucket. You can find how to deploy such function in the following documentation https://cloud.google.com/functions/docs/calling/storage#deploy
Code your function to trigger your DAG, it's well explained in the following documentation: https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf
Use one of the following airflow operators to run a dataflow job as a task within your DAG
DataflowCreateJavaJobOperator if you are using Java SDK
DataflowCreatePythonJobOperator if you are using Python SDK
More details here : https://airflow.apache.org/docs/apache-airflow-providers-google/2.1.0/operators/cloud/dataflow.html
That was the answer to your question about composer, but you should consider replacing Composer by Cloud Workflows, and rather than using an airflow dataflow operator, you can use Cloud Workflows dataflow connector, and you can code your cloud function to trigger a Cloud Workflows execution. There are many clients library in different languages to do that. Here the link to choose your library: https://cloud.google.com/workflows/docs/reference/libraries
Finally Cloud Workflows pros compared to Composer.
it's serverless: you don't have to bother about machine types, network, ...
you use yaml to describe your workflow in easy and fast way, so you don't have to worry about learning airflow
it's way cheaper than composer
...
I have a published and scheduled pipeline running at regular intervals. Some times, the pipeline may fail (for example if the datastore is offline for maintenance). Is there a way to specify the scheduled pipeline to perform a certain action if the pipeline fails for any reason? Actions could be to send me an email, try to run again in a few hours later or invoke a webhook. As it is now, I have to manually check the status of our production pipeline at regular intervals, and this is sub-optimal for obvious reasons. I could of course instruct every script in my pipeline to perform certain actions if they fail for whatever reason, but it would be cleaner and easier to specify it globally for the pipeline schedule (or the pipeline itself).
Possible sub-optimal solutions could be:
Setting up an Azure Logic App to invoke the pipeline
Setting a cron job or Azure Scheduler
Setting up a second Azure Machine Learning pipeline on a schedule that triggers the pipeline, monitors the output and performs relevant actions if errors are encountered
All the solutions above suffers from being convoluted and not very clean - surely there must exist a simple, clean solution for this problem?
This solution reads from the logs of your pipeline and let's you do something within a Logic App capability, I used it to email the team when a scheduled pipeline failed.
Steps:
Create Event Namespace and Event Hub
Create Service Bus Namespace and Service Bus Queue
Create a Stream Analytics Job using EventHub as Input and Service
Bus Queue as Output
Create Logic App with a trigger to any event coming into the Service
Bus Queue then, add an Outlook 360 send an email (v2) step
Create an Event Subscription inside ML Service that sends filtered
events to the Event Hub
Start Stream Analytics Job
Two fundamental steps while creating the Event subscription:
Subscribe to the 'Run Status Changed' event to get the log when a pipeline fails
Use the advanced filters section to specify which pipeline you want to monitor (change 'deal-UAT' to your specific ml experiment), like this:
It looks like a lot of setup but it's super easy and quick to do, it would look something like this in the end:
In my Azure DevOps release, I need to trigger an Azure Data Factory pipeline and wait for the process to finish.
Is there any way to do this without any special trick in Az DevOps? Currently using vsts-publish-adf in my release.
Thanks
It is feasible, though I am unable to evaluate whether it is a good idea in your situation. Here's the practical answer however:
You could trigger and follow the pipeline run with a Azure CLI Task that runs in your Release stage. Azure CLI has Data Factory-specific commands which begin with az datafactory, so you can use them in both cases.
starting the run with az datafactory pipeline-run
waiting for its completion in a loop, running az datafactory pipeline-run show e.g. once a minute
Another solution could be using a REST API, such as in this example of monitoring the pipeline run
Is there any way to do this without any special trick in Az DevOps?
The direct answer is No cause the third-party task itself doesn't support this scenario by design.
According to comment from the Author liprec: At this moment the task only triggers a pipeline run and is not waiting for that run to complete. He has plans to add such a task to wait and poll the task run. So what you want could be possible in coming days, but for now it's not supported.
You have to use something like Powershell scripts to trigger ADF pipeline run via command-line like Mekki suggests above. Here's another similar PS example.
I am evaluating options for launching arbitrary Python tasks/scripts on temporary cloud VMs that get shut down once the job is complete. I am looking across all cloud providers, but the ideal solution should not be vendor-specific. Here is what I found:
Docker Swarm / Kubernetes / Nomad for spinning up docker containers. All look attractive, but cannot confirm if they can terminate VMs once the task is complete.
Cloud Functions/Lambdas look great, but works only for short-lived (few minutes) tasks. Also, GCP supports only JavaScript.
Spins up/down VMs explicitly from a launch script with vendor specific commands. Straightforward and should work.
AWS Batch, Azure Batch - vendor-specific services for batch jobs
AWS Data Pipeline, Azure Data Factory, Google Dataflow - vendor-specific services for data pipelines
Did I miss any good option? Does any container orchestration service like Docker Swarm support allocation and deallocation of multiple temporary VMs to run a one-shot job?