I just went through the process of configuring my Airflow setup to be capable of parallel processing by following this article and using this article.
Everything seems to be working fine in the sense that I was able to run all of those commands from the articles without any errors, warnings, or exceptions. I was able to start up the airflow webserver and airflow scheduler and I'm able to go on the UI and view all my DAGs but now none of my DAGs are starting that previously were working. I had this basic example DAG that was working when my executor was set to SequentialExecuter but now that I have it set to LocalExecuter it never runs. All of the tasks in the DAG are colored white on the graph view with no status when the first one should be in the running state while it waits for the S3 file to appear. I've already cleared all of it's PAST, FUTURE, UPSTREAM history on the UI and I have the DAG turned on so that's not the issue. Also, the scheduler is currently running too.
I've tried using this Stackoverflow Post on the same topic as well but to no avail.
Here is the code I have:
from airflow import DAG
from airflow.operators import SimpleHttpOperator, HttpSensor, EmailOperator, S3KeySensor
from datetime import datetime, timedelta
from airflow.operators.bash_operator import BashOperator
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2018, 5, 29),
'email': ['something#here.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 5,
'retry_delay': timedelta(minutes=5)
}
dag = DAG('myDag', default_args=default_args, schedule_interval= '#once')
t1 = BashOperator(
task_id='my_t1_id',
bash_command='echo "Dag Ran Successfully!" >> /home/ec2-user/output.txt',
dag=dag)
sensor = S3KeySensor(
task_id='my_sensor_id',
bucket_key='*',
wildcard_match=True,
bucket_name='foobar',
s3_conn_id='s3://foobar',
timeout=18*60*60,
poke_interval=120,
dag=dag)
t1.set_upstream(sensor)
And if needed here is my airflow.cfg file (note the only lines I changed were executor = LocalExecutor and sql_alchemy_conn = postgresql+psycopg2://postgres:password#localhost/airflow_meta_db
[core]
# The home folder for airflow, default is ~/airflow
airflow_home = /home/ec2-user/airflow
# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /home/ec2-user/airflow/dags
# The folder where airflow should store its log files
# This path must be absolute
base_log_folder = /home/ec2-user/airflow/logs
# Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
# must supply an Airflow connection id that provides access to the storage
# location.
remote_log_conn_id =
encrypt_s3_logs = False
# Logging level
logging_level = INFO
# Logging class
# Specify the class that will specify the logging configuration
# This class has to be on the python classpath
# logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
logging_config_class =
# Log format
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor
#executor = SequentialExecutor
executor = LocalExecutor
# The SqlAlchemy connection string to the metadata database.
# SqlAlchemy supports many different database engine, more information
# their website
#sql_alchemy_conn = sqlite:////home/ec2-user/airflow/airflow.db
sql_alchemy_conn = postgresql+psycopg2://postgres:password#localhost/airflow_meta_db
# The SqlAlchemy pool size is the maximum number of database connections
# in the pool.
sql_alchemy_pool_size = 5
# The SqlAlchemy pool recycle is the number of seconds a connection
# can be idle in the pool before it is invalidated. This config does
# not apply to sqlite.
sql_alchemy_pool_recycle = 3600
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32
# The number of task instances allowed to run concurrently by the scheduler
dag_concurrency = 16
# Are DAGs paused by default at creation
dags_are_paused_at_creation = True
# When not using pools, tasks are run in the "default pool",
# whose size is guided by this config element
non_pooled_task_slot_count = 128
# The maximum number of active DAG runs per DAG
max_active_runs_per_dag = 16
# Whether to load the examples that ship with Airflow. It's good to
# get started, but you probably want to set this to False in a production
# environment
load_examples = True
# Where your Airflow plugins are stored
plugins_folder = /home/ec2-user/airflow/plugins
# Secret key to save connection passwords in the db
fernet_key = ibwZ5uSASmZGphBmwdJ4BIhd1-5WZXMTTgMF9u1_dGM=
# Whether to disable pickling dags
donot_pickle = False
# How long before timing out a python file import while filling the DagBag
dagbag_import_timeout = 30
# The class to use for running task instances in a subprocess
task_runner = BashTaskRunner
# If set, tasks without a `run_as_user` argument will be run with this user
# Can be used to de-elevate a sudo user running Airflow when executing tasks
default_impersonation =
# What security module to use (for example kerberos):
security =
# Turn unit test mode on (overwrites many configuration options with test
# values at runtime)
unit_test_mode = False
# Name of handler to read task instance logs.
# Default to use file task handler.
task_log_reader = file.task
# Whether to enable pickling for xcom (note that this is insecure and allows for
# RCE exploits). This will be deprecated in Airflow 2.0 (be forced to False).
enable_xcom_pickling = True
# When a task is killed forcefully, this is the amount of time in seconds that
# it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED
killed_task_cleanup_time = 60
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8080
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
[operators]
# The default owner assigned to each new operator, unless
# provided explicitly or passed via `default_args`
default_owner = Airflow
default_cpus = 1
default_ram = 512
default_disk = 512
default_gpus = 0
[webserver]
# The base url of your website as airflow cannot guess what domain or
# cname you are using. This is used in automated emails that
# airflow sends to point links to the right web server
base_url = http://localhost:8080
# The ip specified when starting the web server
web_server_host = 0.0.0.0
# The port on which to run the web server
web_server_port = 8080
# Paths to the SSL certificate and key for the web server. When both are
# provided SSL will be enabled. This does not change the web server port.
web_server_ssl_cert =
web_server_ssl_key =
# Number of seconds the gunicorn webserver waits before timing out on a worker
web_server_worker_timeout = 120
# Number of workers to refresh at a time. When set to 0, worker refresh is
# disabled. When nonzero, airflow periodically refreshes webserver workers by
# bringing up new ones and killing old ones.
worker_refresh_batch_size = 1
# Number of seconds to wait before refreshing a batch of workers.
worker_refresh_interval = 30
# Secret key used to run your flask app
secret_key = temporary_key
# Number of workers to run the Gunicorn web server
workers = 4
# The worker class gunicorn should use. Choices include
# sync (default), eventlet, gevent
worker_class = sync
# Log files for the gunicorn webserver. '-' means log to stderr.
access_logfile = -
error_logfile = -
# Expose the configuration file in the web server
expose_config = False
# Set to true to turn on authentication:
# http://pythonhosted.org/airflow/security.html#web-authentication
authenticate = False
# Filter the list of dags by owner name (requires authentication to be enabled)
filter_by_owner = False
# Filtering mode. Choices include user (default) and ldapgroup.
# Ldap group filtering requires using the ldap backend
#
# Note that the ldap server needs the "memberOf" overlay to be set up
# in order to user the ldapgroup mode.
owner_mode = user
# Default DAG view. Valid values are:
# tree, graph, duration, gantt, landing_times
dag_default_view = tree
# Default DAG orientation. Valid values are:
# LR (Left->Right), TB (Top->Bottom), RL (Right->Left), BT (Bottom->Top)
dag_orientation = LR
# Puts the webserver in demonstration mode; blurs the names of Operators for
# privacy.
demo_mode = False
# The amount of time (in secs) webserver will wait for initial handshake
# while fetching logs from other worker machine
log_fetch_timeout_sec = 5
# By default, the webserver shows paused DAGs. Flip this to hide paused
# DAGs by default
hide_paused_dags_by_default = False
# Consistent page size across all listing views in the UI
page_size = 100
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
# Uncomment and set the user/pass settings if you want to use SMTP AUTH
# smtp_user = airflow
# smtp_password = airflow
smtp_port = 25
smtp_mail_from = airflow#example.com
[celery]
# This section only applies if you are using the CeleryExecutor in
# [core] section above
# The app name that will be used by celery
celery_app_name = airflow.executors.celery_executor
# The concurrency that will be used when starting workers with the
# "airflow worker" command. This defines the number of task instances that
# a worker will take, so size up your workers based on the resources on
# your worker box and the nature of your tasks
celeryd_concurrency = 16
# When you start an airflow worker, airflow starts a tiny web server
# subprocess to serve the workers local log files to the airflow main
# web server, who then builds pages and sends them to users. This defines
# the port on which the logs are served. It needs to be unused, and open
# visible from the main web server to connect into the workers.
worker_log_server_port = 8793
# The Celery broker URL. Celery supports RabbitMQ, Redis and experimentally
# a sqlalchemy database. Refer to the Celery documentation for more
# information.
broker_url = sqla+mysql://airflow:airflow#localhost:3306/airflow
# Another key Celery setting
celery_result_backend = db+mysql://airflow:airflow#localhost:3306/airflow
# Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start
# it `airflow flower`. This defines the IP that Celery Flower runs on
flower_host = 0.0.0.0
# This defines the port that Celery Flower runs on
flower_port = 5555
# Default queue that tasks get assigned to and that worker listen on.
default_queue = default
# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
[dask]
# This section only applies if you are using the DaskExecutor in
# [core] section above
# The IP address and port of the Dask cluster's scheduler.
cluster_address = 127.0.0.1:8786
[scheduler]
# Task instances listen for external kill signal (when you clear tasks
# from the CLI or the UI), this defines the frequency at which they should
# listen (in seconds).
job_heartbeat_sec = 5
# The scheduler constantly tries to trigger new tasks (look at the
# scheduler section in the docs for more information). This defines
# how often the scheduler should run (in seconds).
scheduler_heartbeat_sec = 5
# after how much time should the scheduler terminate in seconds
# -1 indicates to run continuously (see also num_runs)
run_duration = -1
# after how much time a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
dag_dir_list_interval = 300
# How often should stats be printed to the logs
print_stats_interval = 30
child_process_log_directory = /home/ec2-user/airflow/logs/scheduler
# Local task jobs periodically heartbeat to the DB. If the job has
# not heartbeat in this many seconds, the scheduler will mark the
# associated task instance as failed and will re-schedule the task.
scheduler_zombie_task_threshold = 300
# Turn off scheduler catchup by setting this to False.
# Default behavior is unchanged and
# Command Line Backfills still work, but the scheduler
# will not do scheduler catchup if this is False,
# however it can be set on a per DAG basis in the
# DAG definition (catchup)
catchup_by_default = True
# This changes the batch size of queries in the scheduling main loop.
# This depends on query length limits and how long you are willing to hold locks.
# 0 for no limit
max_tis_per_query = 0
# Statsd (https://github.com/etsy/statsd) integration settings
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
# The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run.
max_threads = 2
authenticate = False
[ldap]
# set this to ldaps://<your.ldap.server>:<port>
uri =
user_filter = objectClass=*
user_name_attr = uid
group_member_attr = memberOf
superuser_filter =
data_profiler_filter =
bind_user = cn=Manager,dc=example,dc=com
bind_password = insecure
basedn = dc=example,dc=com
cacert = /etc/ca/ldap_ca.crt
search_scope = LEVEL
[mesos]
# Mesos master address which MesosExecutor will connect to.
master = localhost:5050
# The framework name which Airflow scheduler will register itself as on mesos
framework_name = Airflow
# Number of cpu cores required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_cpu = 1
# Memory in MB required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_memory = 256
# Enable framework checkpointing for mesos
# See http://mesos.apache.org/documentation/latest/slave-recovery/
checkpoint = False
# Failover timeout in milliseconds.
# When checkpointing is enabled and this option is set, Mesos waits
# until the configured timeout for
# the MesosExecutor framework to re-register after a failover. Mesos
# shuts down running tasks if the
# MesosExecutor framework fails to re-register within this timeframe.
# failover_timeout = 604800
# Enable framework authentication for mesos
# See http://mesos.apache.org/documentation/latest/configuration/
authenticate = False
# Mesos credentials, if authentication is enabled
# default_principal = admin
# default_secret = admin
[kerberos]
ccache = /tmp/airflow_krb5_ccache
# gets augmented with fqdn
principal = airflow
reinit_frequency = 3600
kinit_path = kinit
keytab = airflow.keytab
[github_enterprise]
api_rev = v3
[admin]
# UI to hide sensitive variable fields when set to True
hide_sensitive_variable_fields = False
airflow scheduler output:
[2018-05-31 21:15:12,056] {jobs.py:1504} INFO -
================================================================================
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
-------------------------------------------------------------- ----- --------- -------------- -------------------
/home/ec2-user/airflow/dags/Test_Dag_Create_EMR.py 1.00s 2018-05-31T21:15:12
/home/ec2-user/airflow/dags/s3_triggered_emr_cluster_dag.py 19214 0.01s 1.00s 2018-05-31T21:15:10
/home/ec2-user/airflow/dags/myDag.py 1.00s 2018-05-31T21:15:11
/home/ec2-user/airflow/dags/s3_sensor_connection_test.py 1.01s 2018-05-31T21:15:11
/home/ec2-user/airflow/dags/three_s3_triggers_then_emr_work.py 19213 0.01s 1.01s 2018-05-31T21:15:10
================================================================================
[2018-05-31 21:15:12,112] {jobs.py:1742} INFO - Processing file /home/ec2-user/airflow/dags/three_s3_triggers_then_emr_work.py for tasks to queue
[2018-05-31 21:15:12,112] {models.py:189} INFO - Filling up the DagBag from /home/ec2-user/airflow/dags/three_s3_triggers_then_emr_work.py
[2018-05-31 21:15:12,118] {jobs.py:1742} INFO - Processing file /home/ec2-user/airflow/dags/s3_triggered_emr_cluster_dag.py for tasks to queue
[2018-05-31 21:15:12,118] {models.py:189} INFO - Filling up the DagBag from /home/ec2-user/airflow/dags/s3_triggered_emr_cluster_dag.py
[2018-05-31 21:15:12,173] {jobs.py:1754} INFO - DAG(s) dict_keys(['example_trigger_controller_dag', 'example_python_operator', 'example_skip_dag', 'test_utils', 'example_xcom', 'example_passing_params_via_test_command', 'latest_only', 'example_trigger_target_dag', 'example_branch_operator', 'example_http_operator', 'example_branch_dop_operator_v3', 'example_subdag_operator', 'example_subdag_operator.section-1', 'example_subdag_operator.section-2', 'latest_only_with_trigger', 'example_bash_operator', 'tutorial', 'example_short_circuit_operator', 's3_triggered_emr_cluster_dag']) retrieved from /home/ec2-user/airflow/dags/s3_triggered_emr_cluster_dag.py
[2018-05-31 21:15:12,173] {jobs.py:1754} INFO - DAG(s) dict_keys(['example_trigger_controller_dag', 'example_python_operator', 'example_skip_dag', 'test_utils', 'example_xcom', 'example_passing_params_via_test_command', 'latest_only', 'example_trigger_target_dag', 'example_branch_operator', 'example_http_operator', 'example_branch_dop_operator_v3', 'example_subdag_operator', 'example_subdag_operator.section-1', 'example_subdag_operator.section-2', 'latest_only_with_trigger', 'example_bash_operator', 'tutorial', 'example_short_circuit_operator', 'three_s3_triggers_then_emr_work']) retrieved from /home/ec2-user/airflow/dags/three_s3_triggers_then_emr_work.py
[2018-05-31 21:15:12,309] {models.py:341} INFO - Finding 'running' jobs without a recent heartbeat
[2018-05-31 21:15:12,309] {models.py:345} INFO - Failing jobs without heartbeat after 2018-05-31 21:10:12.309615
[2018-05-31 21:15:12,311] {models.py:341} INFO - Finding 'running' jobs without a recent heartbeat
[2018-05-31 21:15:12,311] {models.py:345} INFO - Failing jobs without heartbeat after 2018-05-31 21:10:12.311879
[2018-05-31 21:15:12,314] {jobs.py:375} INFO - Processing /home/ec2-user/airflow/dags/three_s3_triggers_then_emr_work.py took 0.267 seconds
[2018-05-31 21:15:12,316] {jobs.py:375} INFO - Processing /home/ec2-user/airflow/dags/s3_triggered_emr_cluster_dag.py took 0.265 seconds
[2018-05-31 21:15:13,057] {jobs.py:1627} INFO - Heartbeating the process manager
[2018-05-31 21:15:13,057] {dag_processing.py:468} INFO - Processor for /home/ec2-user/airflow/dags/three_s3_triggers_then_emr_work.py finished
[2018-05-31 21:15:13,057] {dag_processing.py:468} INFO - Processor for /home/ec2-user/airflow/dags/s3_triggered_emr_cluster_dag.py finished
[2018-05-31 21:15:13,060] {dag_processing.py:537} INFO - Started a process (PID: 19219) to generate tasks for /home/ec2-user/airflow/dags/s3_sensor_connection_test.py
[2018-05-31 21:15:13,062] {dag_processing.py:537} INFO - Started a process (PID: 19220) to generate tasks for /home/ec2-user/airflow/dags/myDag.py
[2018-05-31 21:15:13,063] {jobs.py:1662} INFO - Heartbeating the executor
[2018-05-31 21:15:13,064] {jobs.py:368} INFO - Started process (PID=19219) to work on /home/ec2-user/airflow/dags/s3_sensor_connection_test.py
[2018-05-31 21:15:13,068] {jobs.py:368} INFO - Started process (PID=19220) to work on /home/ec2-user/airflow/dags/myDag.py
[2018-05-31 21:15:13,130] {jobs.py:1742} INFO - Processing file /home/ec2-user/airflow/dags/s3_sensor_connection_test.py for tasks to queue
[2018-05-31 21:15:13,130] {models.py:189} INFO - Filling up the DagBag from /home/ec2-user/airflow/dags/s3_sensor_connection_test.py
[2018-05-31 21:15:13,134] {jobs.py:1742} INFO - Processing file /home/ec2-user/airflow/dags/myDag.py for tasks to queue
[2018-05-31 21:15:13,134] {models.py:189} INFO - Filling up the DagBag from /home/ec2-user/airflow/dags/myDag.py
[2018-05-31 21:15:13,189] {jobs.py:1754} INFO - DAG(s) dict_keys(['example_trigger_controller_dag', 'example_python_operator', 'example_skip_dag', 'test_utils', 'example_xcom', 'example_passing_params_via_test_command', 'latest_only', 'example_trigger_target_dag', 'example_branch_operator', 'example_http_operator', 'example_branch_dop_operator_v3', 'example_subdag_operator', 'example_subdag_operator.section-1', 'example_subdag_operator.section-2', 'latest_only_with_trigger', 'example_bash_operator', 'tutorial', 'example_short_circuit_operator', 'myDag']) retrieved from /home/ec2-user/airflow/dags/myDag.py
[2018-05-31 21:15:13,315] {models.py:341} INFO - Finding 'running' jobs without a recent heartbeat
[2018-05-31 21:15:13,316] {models.py:345} INFO - Failing jobs without heartbeat after 2018-05-31 21:10:13.316206
[2018-05-31 21:15:13,321] {jobs.py:375} INFO - Processing /home/ec2-user/airflow/dags/s3_sensor_connection_test.py took 0.257 seconds
[2018-05-31 21:15:13,333] {models.py:341} INFO - Finding 'running' jobs without a recent heartbeat
[2018-05-31 21:15:13,334] {models.py:345} INFO - Failing jobs without heartbeat after 2018-05-31 21:10:13.334021
[2018-05-31 21:15:13,338] {jobs.py:375} INFO - Processing /home/ec2-user/airflow/dags/myDag.py took 0.270 seconds
[2018-05-31 21:15:14,065] {jobs.py:1627} INFO - Heartbeating the process manager
[2018-05-31 21:15:14,066] {dag_processing.py:468} INFO - Processor for /home/ec2-user/airflow/dags/s3_sensor_connection_test.py finished
[2018-05-31 21:15:14,066] {dag_processing.py:468} INFO - Processor for /home/ec2-user/airflow/dags/myDag.py finished
[2018-05-31 21:15:14,068] {dag_processing.py:537} INFO - Started a process (PID: 19225) to generate tasks for /home/ec2-user/airflow/dags/Test_Dag_Create_EMR.py
[2018-05-31 21:15:14,069] {jobs.py:1662} INFO - Heartbeating the executor
[2018-05-31 21:15:14,072] {jobs.py:368} INFO - Started process (PID=19225) to work on /home/ec2-user/airflow/dags/Test_Dag_Create_EMR.py
[2018-05-31 21:15:14,187] {jobs.py:1742} INFO - Processing file /home/ec2-user/airflow/dags/Test_Dag_Create_EMR.py for tasks to queue
[2018-05-31 21:15:14,188] {models.py:189} INFO - Filling up the DagBag from /home/ec2-user/airflow/dags/Test_Dag_Create_EMR.py
[2018-05-31 21:15:14,239] {jobs.py:1754} INFO - DAG(s) dict_keys(['example_trigger_controller_dag', 'example_python_operator', 'example_skip_dag', 'test_utils', 'example_xcom', 'example_passing_params_via_test_command', 'latest_only', 'example_trigger_target_dag', 'example_branch_operator', 'example_http_operator', 'example_branch_dop_operator_v3', 'example_subdag_operator', 'example_subdag_operator.section-1', 'example_subdag_operator.section-2', 'latest_only_with_trigger', 'example_bash_operator', 'tutorial', 'example_short_circuit_operator', 'kyles_dag']) retrieved from /home/ec2-user/airflow/dags/Test_Dag_Create_EMR.py
[2018-05-31 21:15:14,366] {models.py:341} INFO - Finding 'running' jobs without a recent heartbeat
[2018-05-31 21:15:14,366] {models.py:345} INFO - Failing jobs without heartbeat after 2018-05-31 21:10:14.366593
[2018-05-31 21:15:14,371] {jobs.py:375} INFO - Processing /home/ec2-user/airflow/dags/Test_Dag_Create_EMR.py took 0.299 seconds
[2018-05-31 21:15:15,071] {jobs.py:1627} INFO - Heartbeating the process manager
Note: I don't think it's very relevant for this question but I'm running Airflow on an Amazon EC2-Instance.
I'm not sure which of these steps exactly solved my problem and I'm not sure exactly what the root cause of the problem was but I did this:
I literally just reset everything. First I shut down the webserver and scheduler using kill theirPIDs or ctrl + c if it's open still in the terminal. Then I deleted all the entries under /home/ec2-user/airflow/dags/__pycache__. Then I restarted the postgre database using sudo /etc/init.d/postgresql restart then I ran airflow resetdb. Then I reran airflow webserver and airflow scheduler. I went in the UI and turned on the DAG and voila it went into the running state and then worked successfully. No idea what was going on though.....
I want to emulate SLURM on Ubuntu 16.04. I don't need serious resource management, I just want to test some simple examples. I cannot install SLURM in the usual way, and I am wondering if there are other options. Other things I have tried:
A Docker image. Unfortunately, docker pull agaveapi/slurm; docker run agaveapi/slurm gives me errors:
/usr/lib/python2.6/site-packages/supervisor/options.py:295: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2017-10-29 15:27:45,436 CRIT Supervisor running as root (no user in config file)
2017-10-29 15:27:45,437 INFO supervisord started with pid 1
2017-10-29 15:27:46,439 INFO spawned: 'slurmd' with pid 9
2017-10-29 15:27:46,441 INFO spawned: 'sshd' with pid 10
2017-10-29 15:27:46,443 INFO spawned: 'munge' with pid 11
2017-10-29 15:27:46,443 INFO spawned: 'slurmctld' with pid 12
2017-10-29 15:27:46,452 INFO exited: munge (exit status 0; not expected)
2017-10-29 15:27:46,452 CRIT reaped unknown pid 13)
2017-10-29 15:27:46,530 INFO gave up: munge entered FATAL state, too many start retries too quickly
2017-10-29 15:27:46,531 INFO exited: slurmd (exit status 1; not expected)
2017-10-29 15:27:46,535 INFO gave up: slurmd entered FATAL state, too many start retries too quickly
2017-10-29 15:27:46,536 INFO exited: slurmctld (exit status 0; not expected)
2017-10-29 15:27:47,537 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-29 15:27:47,537 INFO gave up: slurmctld entered FATAL state, too many start retries too quickly
This guide to start a SLURM VM via Vagrant. I tried, but copying over my munge key timed out.
sudo scp /etc/munge/munge.key vagrant#server:/home/vagrant/
ssh: connect to host server port 22: Connection timed out
lost connection
So ... we have an existing cluster here but it runs an older Ubuntu version which does not mesh well with my workstation running 17.04.
So on my workstation, I just made sure I slurmctld (backend) and slurmd installed, and then set up a trivial slurm.conf with
ControlMachine=mybox
# ...
NodeName=DEFAULT CPUs=4 RealMemory=4000 TmpDisk=50000 State=UNKNOWN
NodeName=mybox CPUs=4 RealMemory=16000
after which I restarted slurmcltd and then slurmd. Now all is fine:
root#mybox:/etc/slurm-llnl$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
demo up infinite 1 idle mybox
root#mybox:/etc/slurm-llnl$
This is a degenerate setup, our real one has a mix of dev and prod machine and appropriate partitions. But this should answer your "can backend really be client" question. Also, my machine is not really called mybox but is not really pertinent for the question in either case.
Using Ubuntu 17.04, all stock, with munge to communicate (which is the default anyway).
Edit: To wit:
me#mybox:~$ COLUMNS=90 dpkg -l '*slurm*' | grep ^ii
ii slurm-client 16.05.9-1ubun amd64 SLURM client side commands
ii slurm-wlm-basic- 16.05.9-1ubun amd64 SLURM basic plugins
ii slurmctld 16.05.9-1ubun amd64 SLURM central management daemon
ii slurmd 16.05.9-1ubun amd64 SLURM compute node daemon
me#mybox:~$
I would still prefer to run SLURM natively, but I caved and spun up a Debian 9.2 VM. See here for my efforts to troubleshoot a native installation. The directions here worked smoothly, but I needed to make the following changes to slurm.conf. Below, Debian64 is the hostname, and wlandau is my user name.
ControlMachine=Debian64
SlurmUser=wlandau
NodeName=Debian64
Here is the complete slurm.conf. An analogous slurm.conf did not work on my native Ubuntu 16.04.
# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=Debian64
#ControlAddr=
#BackupController=
#BackupAddr=
#
AuthType=auth/munge
#CheckpointType=checkpoint/none
CryptoType=crypto/munge
#DisableRootJobs=NO
#EnforcePartLimits=NO
#Epilog=
#EpilogSlurmctld=
#FirstJobId=1
#MaxJobId=999999
#GresTypes=
#GroupUpdateForce=0
#GroupUpdateTime=600
#JobCheckpointDir=/var/lib/slurm-llnl/checkpoint
#JobCredentialPrivateKey=
#JobCredentialPublicCertificate=
#JobFileAppend=0
#JobRequeue=1
#JobSubmitPlugins=1
#KillOnBadExit=0
#LaunchType=launch/slurm
#Licenses=foo*4,bar
#MailProg=/usr/bin/mail
#MaxJobCount=5000
#MaxStepCount=40000
#MaxTasksPerNode=128
MpiDefault=none
#MpiParams=ports=#-#
#PluginDir=
#PlugStackConfig=
#PrivateData=jobs
ProctrackType=proctrack/pgid
#Prolog=
#PrologFlags=
#PrologSlurmctld=
#PropagatePrioProcess=0
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#RebootProgram=
ReturnToService=1
#SallocDefaultCommand=
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=wlandau
#SlurmdUser=root
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
#TaskEpilog=
TaskPlugin=task/none
#TaskPluginParam=
#TaskProlog=
#TopologyPlugin=topology/tree
#TmpFS=/tmp
#TrackWCKey=no
#TreeWidth=
#UnkillableStepProgram=
#UsePAM=0
#
#
# TIMERS
#BatchStartTimeout=10
#CompleteWait=0
#EpilogMsgTime=2000
#GetEnvTimeout=2
#HealthCheckInterval=0
#HealthCheckProgram=
InactiveLimit=0
KillWait=30
#MessageTimeout=10
#ResvOverRun=0
MinJobAge=300
#OverTimeLimit=0
SlurmctldTimeout=120
SlurmdTimeout=300
#UnkillableStepTimeout=60
#VSizeFactor=0
Waittime=0
#
#
# SCHEDULING
#DefMemPerCPU=0
FastSchedule=1
#MaxMemPerCPU=0
#SchedulerRootFilter=1
#SchedulerTimeSlice=30
SchedulerType=sched/backfill
SchedulerPort=7321
SelectType=select/linear
#SelectTypeParameters=
#
#
# JOB PRIORITY
#PriorityFlags=
#PriorityType=priority/basic
#PriorityDecayHalfLife=
#PriorityCalcPeriod=
#PriorityFavorSmall=
#PriorityMaxAge=
#PriorityUsageResetPeriod=
#PriorityWeightAge=
#PriorityWeightFairshare=
#PriorityWeightJobSize=
#PriorityWeightPartition=
#PriorityWeightQOS=
#
#
# LOGGING AND ACCOUNTING
#AccountingStorageEnforce=0
#AccountingStorageHost=
#AccountingStorageLoc=
#AccountingStoragePass=
#AccountingStoragePort=
AccountingStorageType=accounting_storage/none
#AccountingStorageUser=
AccountingStoreJobComment=YES
ClusterName=cluster
#DebugFlags=
#JobCompHost=
#JobCompLoc=
#JobCompPass=
#JobCompPort=
JobCompType=jobcomp/none
#JobCompUser=
#JobContainerType=job_container/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
#SlurmSchedLogFile=
#SlurmSchedLogLevel=
#
#
# POWER SAVE SUPPORT FOR IDLE NODES (optional)
#SuspendProgram=
#ResumeProgram=
#SuspendTimeout=
#ResumeTimeout=
#ResumeRate=
#SuspendExcNodes=
#SuspendExcParts=
#SuspendRate=
#SuspendTime=
#
#
# COMPUTE NODES
NodeName=Debian64 CPUs=1 RealMemory=744 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN
PartitionName=debug Nodes=Debian64 Default=YES MaxTime=INFINITE State=UP
My mistake - after 6-8 hours of running programs on Java i get this log hs_err_pid6662.log
and this
[testuser#apus ~]$ sh /home/progr/work/import.sh
/usr/bin/hadoop: fork: retry: Resource temporarily unavailable
/usr/bin/hadoop: fork: retry: Resource temporarily unavailable
/usr/bin/hadoop: fork: retry: Resource temporarily unavailable
/usr/bin/hadoop: fork: retry: Resource temporarily unavailable
/usr/bin/hadoop: fork: Resource temporarily unavailable
Programs run every five minutes and try to import/export from oracle
How to fix this?
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (gcTaskThread.cpp:48), pid=6662,
tid=0x00007f429a675700
#
--------------- T H R E A D ---------------
Current thread (0x00007f4294019000): JavaThread "Unknown thread"
[_thread_in_vm, id=6696, stack(0x00007f429a575000,0x00007f429a676000)]
Stack: [0x00007f429a575000,0x00007f429a676000], sp=0x00007f429a674550,
free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
VM Arguments:
jvm_args: -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.11.1-
1.cdh5.11.1.p0.4/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -
Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.11.1-
1.cdh5.11.1.p0.4/lib/hadoop -Dhadoop.id.str= -
Dhadoop.root.logger=INFO,console -
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/java/jdk1.8.0_102
# JRE version: (8.0_102-b14) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode linux-
amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
Memory: 4k page, physical 24591972k(6051016k free), swap 12369916k(11359436k
free)
I am running programs like sqoop-import,sqoop-export on Java every 5 minutes.
example:
#!/bin/bash
hadoop jar /home/progr/import_sqoop/oracle.jar.
CDH version 5.11.1
java version jdk1.8.0_102
OS:Red Hat Enterprise Linux Server release 6.9 (Santiago)
Mem free:
total used free shared buffers cached
Mem: 24591972 20080336 4511636 132036 334456 2825792
-/+ buffers/cache: 16920088 7671884
Swap: 12369916 1008664 11361252
Host Memory Usage
enter image description here
The maximum heap memory is (by default) limited to 1GB. You need to increase this
JRE version: (8.0_102-b14) (build )
jvm_args: -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.11.1-
1.cdh5.11.1.p0.4/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -
Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.11.1-
1.cdh5.11.1.p0.4/lib/hadoop -Dhadoop.id.str= -
Dhadoop.root.logger=INFO,console -
Try the following for to increase this to 2048MB (or higher if required).
export HADOOP_CLIENT_OPTS="-Xmx2048m ${HADOOP_CLIENT_OPTS}"
Reference:
Pig: Hadoop jobs Fail
https://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201104.mbox/%3C5FFFF0E4-B3BA-420A-ADE3-B422A66E8B11#yahoo-inc.com%3E