Unable to create a cron job of my pyspark script using Airflow - apache-spark

Question

I have a pyspark script which is working perfectly fine now what I want to do is that I want to schedule that job for every minute and for that I'm using Apache Airflow, I have created a .py file for airflow which is following:
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
import os
from builtins import range
import airflow
from airflow.models import DAG
from airflow.operators.dummy_operator import DummyOperator
srcDir = os.getcwd() + '/home/user/testing.py'
sparkSubmit = '/home/usr/spark-2.4.0-bin-hadoop2.7/bin/spark-submit'
default_args = {
"owner": "usr",
"depends_on_past": False,
"start_date": datetime(2019, 4, 8),
"email": ["abc#gmail.com"],
"email_on_failure": True,
"email_on_retry": True,
'retries': 5,
'retry_delay': timedelta(minutes=1),
}
dag= DAG('my_airflow',default_args=default_args, schedule_interval='* * * * *')
t1 = BashOperator(
task_id='task1',
bash_command='/home/user/spark-2.4.0-bin-hadoop2.7/bin/spark-submit' + ' ' + srcDir,
dag=dag,
)
But when I run this by python3 air_flow.py it shows nothing neither on console nor on Airflow UI.
I want to know how to make my pyspark script scheduled on every minute by Apache Airflow?
Any help would be really appreciated

Answer 1

Running python3 air_flow.py just parses your file.
To run the file on a schedule, your would first need to Start Airflow Webserver and Airflow Scheduler.
# initialize the database
airflow initdb
# start the web server, default port is 8080
airflow webserver -p 8080
# start the scheduler
airflow scheduler
Then on your browser, visit http://localhost:8080 which will redirect you to Airflow Webserver UI which looks like the following:
Your script would be run automatically every minute. If you want to trigger it manually from the UI, click on the Run button in the right side of your DAG.
Follow the Quick Start Guide: https://airflow.readthedocs.io/en/1.10.2/start.html