I am currently attempting to setup LDAP integration with an existing LDAP server in Airflow. In the past, I have attempted making a cacert (ldap_ca.crt) and have followed this guide and this guide.
When I start up Airflow I am presented with a login screen that does not accept any users on the LDAP server and simply clears the username/password box when attempting to sign in.
This is the current code in my webserver_config.py (I have also tried making edits to airflow.cfg without success):
# The authentication type
# AUTH_OID : Is for OpenID
# AUTH_DB : Is for database
# AUTH_LDAP : Is for LDAP
# AUTH_REMOTE_USER : Is for using REMOTE_USER from web server
# AUTH_OAUTH : Is for OAuth
AUTH_TYPE = AUTH_LDAP
# Uncomment to setup Full admin role name
# AUTH_ROLE_ADMIN = 'Admin'
# Uncomment to setup Public role name, no authentication needed
# AUTH_ROLE_PUBLIC = 'Public'
# Will allow user self registration
AUTH_USER_REGISTRATION = True
# The default user self registration role
AUTH_USER_REGISTRATION_ROLE = "Viewer"
# When using LDAP Auth, setup the ldap server
# AUTH_LDAP_SERVER = "ldap://ldapserver.new"
AUTH_LDAP_SERVER = "ldap://ldap-server-name.org.com:999"
AUTH_LDAP_BIND_USER = "CN=p_biaas,OU=Unix,OU=ServiceAccounts,OU=AAA,OU=AAA,DC=ms,DC=ds,DC=aaa,DC=com"
AUTH_LDAP_BIND_PASSWORD = "password"
#AUTH_LDAP_SEARCH = "CN=Users,DC=ms,DC=ds,DC=aaa,DC=com"
#AUTH_LDAP_SEARCH= "OU=Unix,OU=ServiceAccounts,OU=AAA,OU=AAA,DC=ms,DC=ds,DC=aaa,DC=com"
AUTH_LDAP_SEARCH = "DC=ms,DC=ds,DC=aaa,DC=com"
AUTH_LDAP_UID_FIELD = "sAMAccountName"
#AUTH_LDAP_USE_TLS = False
AUTH_LDAP_FIRSTNAME_FIELD = "givenName"
AUTH_LDAP_LASTTNAME_FIELD = "sn"
I just made a video to set up Airflow 2.0 with LDAP.
I think it will help you a lot :)
Configure AIRFLOW 2.0 with LDAP
The two guides that you followed are for airflow v1.10.1 and v1.10.12. Airflow 2.0 introduces a host of changes to providers (akin to python 2 to python 3).
As a start please refer to the current version of the airflow docs on access control
If you have a working configuration of LDAP in 1.10.12, try upgrading to v 1.10.14 and then installing the backport providers before following the recommended upgrade path.
Airflow has put out a guide on upgrading to airflow 2.0.
I had exact same issue...
Are you using the configuration files generated by the previous version of Airflow?
I had a similar configuration of LDAP (like you have) but it was not working with old configuration files.
Then I generated a brand new config by Airflow 2.0.1, passed in my old LDAP configuration and it worked.
Maybe it is the same issue.
There is a webserver_config.py configuration for Airflow 2.2.2 to connect IBM Bluepages LDAP. It is based on Marc's answer.
The only difference is to set the default role to the Viewer for new users. User with Public role only after login sees a weird page that looks like something going wrong.
import os
from airflow import configuration as conf
from airflow.www.fab_security.manager import AUTH_LDAP
basedir = os.path.abspath(os.path.dirname(__file__))
# The SQLAlchemy connection string.
SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
# Flask-WTF flag for CSRF
CSRF_ENABLED = True
# AUTH_TYPE = AUTH_OAUTH
AUTH_TYPE = AUTH_LDAP
AUTH_LDAP_SERVER = 'ldaps://bluepages.ibm.com:636'
# search configs
AUTH_LDAP_SEARCH = 'ou=bluepages,o=ibm.com'
AUTH_LDAP_UID_FIELD = 'mail'
AUTH_LDAP_ALLOW_SELF_SIGNED = True
# username and password to login IBM Bluepages
AUTH_LDAP_BIND_USER = 'uid=<<ibm user uid>>,c=us,ou=bluepages,o=ibm.com'
AUTH_LDAP_BIND_PASSWORD = '<<ibm user password>>'
# Will allow user self registration
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = 'Viewer'
AUTH_LDAP_FIRSTNAME_FIELD = "givenName"
AUTH_LDAP_LASTNAME_FIELD = "sn"
AUTH_LDAP_EMAIL_FIELD = "mail"
# ----------------------------------------------------
# Theme CONFIG
# ----------------------------------------------------
# Flask App Builder comes up with a number of predefined themes
# that you can use for Apache Airflow.
# http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes
# Please make sure to remove "navbar_color" configuration from airflow.cfg
# in order to fully utilize the theme. (or use that property in conjunction with theme)
# APP_THEME = "bootstrap-theme.css" # default bootstrap
# APP_THEME = "amelia.css"
# APP_THEME = "cerulean.css"
# APP_THEME = "cosmo.css"
# APP_THEME = "cyborg.css"
# APP_THEME = "darkly.css"
# APP_THEME = "flatly.css"
# APP_THEME = "journal.css"
# APP_THEME = "lumen.css"
# APP_THEME = "paper.css"
# APP_THEME = "readable.css"
# APP_THEME = "sandstone.css"
# APP_THEME = "simplex.css"
# APP_THEME = "slate.css"
# APP_THEME = "solar.css"
# APP_THEME = "spacelab.css"
# APP_THEME = "superhero.css"
# APP_THEME = "united.css"
# APP_THEME = "yeti.css"
Related
I am trying to use Python code to connect from GCP Cloud functions to PostgreSQL instance of CloudSQL . I am new to python , so struggling with some error and need help to fix the error
My code is as follows:
# Don't add the "/.s.PGSQL.5432 suffix" because it will already be added back automatically by the library...
import psycopg2
import sqlalchemy
from sqlalchemy import create_engine
engine = sqlalchemy.create_engine('postgresql+psycopg2://postgres:a1b2c3d4e190#X.X.X.X/postgres_dbase')
print("connected")
The function is deployed fine but the URL is failing with below error
Error: could not handle the request
It is showing below error in the log
Details:'Request' object has no attribute '_instantiate_plugins'
It works fine when I execute the code directly
$ python3 open.py
connected
I am using Python 3.7
UPDATE
It looks like I have to set DATABASE_URL but not sure how to set it in CLoud functions . Added below Runtime environmental variable in Cloud functions but no luck
DATABASE_URL="sqlite://"
https://github.com/sqlalchemy/sqlalchemy/issues/5330
Not sure if you are trying to connect from Public IP or from Private. However, here you can find an example about how to perform your concern using Public IP. Your code should looks like
db_user = os.environ["DB_USER"]
db_pass = os.environ["DB_PASS"]
db_name = os.environ["DB_NAME"]
db_socket_dir = os.environ.get("DB_SOCKET_DIR", "/cloudsql")
cloud_sql_connection_name = os.environ["CLOUD_SQL_CONNECTION_NAME"]
pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgres+pg8000://<db_user>:<db_pass>#/<db_name>
# ?unix_sock=<socket_path>/<cloud_sql_instance_name>/.s.PGSQL.5432
sqlalchemy.engine.url.URL(
drivername="postgresql+pg8000",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
database=db_name, # e.g. "my-database-name"
query={
"unix_sock": "{}/{}/.s.PGSQL.5432".format(
db_socket_dir, # e.g. "/cloudsql"
cloud_sql_connection_name) # i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
}
),
**db_config
)
On the other hand, if you are using Private IP, you should have in your code something like this
db_user = os.environ["DB_USER"]
db_pass = os.environ["DB_PASS"]
db_name = os.environ["DB_NAME"]
db_host = os.environ["DB_HOST"]
# Extract host and port from db_host
host_args = db_host.split(":")
db_hostname, db_port = host_args[0], int(host_args[1])
pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgres+pg8000://<db_user>:<db_pass>#<db_host>:<db_port>/<db_name>
sqlalchemy.engine.url.URL(
drivername="postgresql+pg8000",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
host=db_hostname, # e.g. "127.0.0.1"
port=db_port, # e.g. 5432
database=db_name # e.g. "my-database-name"
),
**db_config
)
Make sure you have configured your Cloud Functions depending on if you are using Public IP or Private IP.
This has worked for my connecting Google App Engine to PostgreSQL instance from Google Cloud Platform.
import psycopg2
try:
conn = psycopg2.connect(dbname='your_data_base_name', user='your_data_base_user', password='your_data_base_user_password', host='/cloudsql/project_name:region:database_instance_name')
except psycopg2.Error as e:
print("Error: Could not make connection to the Postgres database")
print(e)
cur = conn.cursor()
I have an airflow version 1.10.9 running in a docker.
It works but it has some errors that I can see using docker logs:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/local/lib/python3.6/dist-packages/airflow/executors/local_executor.py", line 112, in run
key, command = self.task_queue.get()
File "<string>", line 2, in get
File "/usr/lib/python3.6/multiprocessing/managers.py", line 757, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
And also this one (not full trace), which is raised when I click in the succes green mark in the column Recent tasks.
self.fail('no filter named %r' % node.name, node.lineno)
File "/usr/local/lib/python3.6/dist-packages/jinja2/compiler.py", line 309, in fail
raise TemplateAssertionError(msg, lineno, self.name, self.filename)
jinja2.exceptions.TemplateAssertionError: no filter named 'min'
What I don't understand is howit raise an EOFError, but airflow still running. Neither I don't understand what does this error mean.
entrypoint.sh
#!/usr/bin/env bash
TRY_LOOP="20"
: "${AIRFLOW_HOME:="/usr/local/airflow"}"
: "${AIRFLOW__CORE__FERNET_KEY:=${FERNET_KEY:=$(python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)")}}"
: "${AIRFLOW__CORE__EXECUTOR:="LocalExecutor"}"
# Load DAGs examples (default: Yes)
if [[ -z "$AIRFLOW__CORE__LOAD_EXAMPLES" && "${LOAD_EX:=n}" == n ]]; then
AIRFLOW__CORE__LOAD_EXAMPLES=False
fi
export \
AIRFLOW_HOME \
AIRFLOW__CORE__EXECUTOR \
AIRFLOW__CORE__FERNET_KEY \
AIRFLOW__CORE__LOAD_EXAMPLES \
# Install custom python package if requirements.txt is present
if [ -e "/requirements.txt" ]; then
$(command -v pip) install --user -r /requirements.txt
fi
wait_for_port() {
local name="$1" host="$2" port="$3"
local j=0
while ! nc -z "$host" "$port" >/dev/null 2>&1 < /dev/null; do
j=$((j+1))
if [ $j -ge $TRY_LOOP ]; then
echo >&2 "$(date) - $host:$port still not reachable, giving up"
exit 1
fi
echo "$(date) - waiting for $name... $j/$TRY_LOOP"
sleep 5
done
}
# Other executors than SequentialExecutor drive the need for an SQL database, here PostgreSQL is used
if [ "$AIRFLOW__CORE__EXECUTOR" != "SequentialExecutor" ]; then
# Check if the user has provided explicit Airflow configuration concerning the database
if [ -z "$AIRFLOW__CORE__SQL_ALCHEMY_CONN" ]; then
# Default values corresponding to the default compose files
: "${POSTGRES_HOST:="XXX"}"
: "${POSTGRES_PORT:="XXX"}"
: "${POSTGRES_USER:="XXX"}"
: "${POSTGRES_PASSWORD:="XXX"}"
: "${POSTGRES_DB:="XXX"}"
: "${POSTGRES_EXTRAS:-""}"
AIRFLOW__CORE__SQL_ALCHEMY_CONN="postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}#${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}${POSTGRES_EXTRAS}"
export AIRFLOW__CORE__SQL_ALCHEMY_CONN
# Check if the user has provided explicit Airflow configuration for the broker's connection to the database
if [ "$AIRFLOW__CORE__EXECUTOR" = "CeleryExecutor" ]; then
AIRFLOW__CELERY__RESULT_BACKEND="db+postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}#${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}${POSTGRES_EXTRAS}"
export AIRFLOW__CELERY__RESULT_BACKEND
fi
else
if [[ "$AIRFLOW__CORE__EXECUTOR" == "CeleryExecutor" && -z "$AIRFLOW__CELERY__RESULT_BACKEND" ]]; then
>&2 printf '%s\n' "FATAL: if you set AIRFLOW__CORE__SQL_ALCHEMY_CONN manually with CeleryExecutor you must also set AIRFLOW__CELERY__RESULT_BACKEND"
exit 1
fi
# Derive useful variables from the AIRFLOW__ variables provided explicitly by the user
POSTGRES_ENDPOINT=$(echo -n "$AIRFLOW__CORE__SQL_ALCHEMY_CONN" | cut -d '/' -f3 | sed -e 's,.*#,,')
POSTGRES_HOST=$(echo -n "$POSTGRES_ENDPOINT" | cut -d ':' -f1)
POSTGRES_PORT=$(echo -n "$POSTGRES_ENDPOINT" | cut -d ':' -f2)
fi
wait_for_port "Postgres" "$POSTGRES_HOST" "$POSTGRES_PORT"
fi
# CeleryExecutor drives the need for a Celery broker, here Redis is used
if [ "$AIRFLOW__CORE__EXECUTOR" = "CeleryExecutor" ]; then
# Check if the user has provided explicit Airflow configuration concerning the broker
if [ -z "$AIRFLOW__CELERY__BROKER_URL" ]; then
# Default values corresponding to the default compose files
: "${REDIS_PROTO:="redis://"}"
: "${REDIS_HOST:="redis"}"
: "${REDIS_PORT:="6379"}"
: "${REDIS_PASSWORD:=""}"
: "${REDIS_DBNUM:="1"}"
# When Redis is secured by basic auth, it does not handle the username part of basic auth, only a token
if [ -n "$REDIS_PASSWORD" ]; then
REDIS_PREFIX=":${REDIS_PASSWORD}#"
else
REDIS_PREFIX=
fi
AIRFLOW__CELERY__BROKER_URL="${REDIS_PROTO}${REDIS_PREFIX}${REDIS_HOST}:${REDIS_PORT}/${REDIS_DBNUM}"
export AIRFLOW__CELERY__BROKER_URL
else
# Derive useful variables from the AIRFLOW__ variables provided explicitly by the user
REDIS_ENDPOINT=$(echo -n "$AIRFLOW__CELERY__BROKER_URL" | cut -d '/' -f3 | sed -e 's,.*#,,')
REDIS_HOST=$(echo -n "$POSTGRES_ENDPOINT" | cut -d ':' -f1)
REDIS_PORT=$(echo -n "$POSTGRES_ENDPOINT" | cut -d ':' -f2)
fi
wait_for_port "Redis" "$REDIS_HOST" "$REDIS_PORT"
fi
case "$1" in
webserver)
airflow initdb
if [ "$AIRFLOW__CORE__EXECUTOR" = "LocalExecutor" ] || [ "$AIRFLOW__CORE__EXECUTOR" = "SequentialExecutor" ]; then
# With the "Local" and "Sequential" executors it should all run in one container.
airflow scheduler &
fi
exec airflow webserver
;;
worker|scheduler)
# Give the webserver time to run initdb.
sleep 10
exec airflow "$#"
;;
flower)
sleep 10
exec airflow "$#"
;;
version)
exec airflow "$#"
;;
*)
# The command is something like bash, not an airflow subcommand. Just run it in the right environment.
exec "$#"
;;
esac
Airflow.cfg
[core]
# The home folder for airflow, default is ~/airflow
# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /usr/local/airflow/dags
# The folder where airflow should store its log files
# This path must be absolute
base_log_folder = /usr/local/airflow/logs
# Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search.
# Users must supply an Airflow connection id that provides access to the storage
# location. If remote_logging is set to true, see UPDATING.md for additional
# configuration requirements.
remote_logging = False
remote_log_conn_id =
remote_base_log_folder =
encrypt_s3_logs = False
# Logging level
logging_level = INFO
fab_logging_level = WARN
# Logging class
# Specify the class that will specify the logging configuration
# This class has to be on the python classpath
# logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
logging_config_class =
# Log format
# we need to escape the curly braces by adding an additional curly brace
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
# Log filename format
# we need to escape the curly braces by adding an additional curly brace
log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
# Hostname by providing a path to a callable, which will resolve the hostname
hostname_callable = socket:getfqdn
# Default timezone in case supplied date times are naive
# can be utc (default), system, or any IANA timezone string (e.g. Europe/Amsterdam)
default_timezone = utc
# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor
executor = LocalExecutor
# The SqlAlchemy connection string to the metadata database.
# SqlAlchemy supports many different database engine, more information
# their website
sql_alchemy_conn = $AIRFLOW_CONN_PROD_RDS
# If SqlAlchemy should pool database connections.
sql_alchemy_pool_enabled = True
# The SqlAlchemy pool size is the maximum number of database connections
# in the pool. 0 indicates no limit.
sql_alchemy_pool_size = 5
# The SqlAlchemy pool recycle is the number of seconds a connection
# can be idle in the pool before it is invalidated. This config does
# not apply to sqlite. If the number of DB connections is ever exceeded,
# a lower config value will allow the system to recover faster.
sql_alchemy_pool_recycle = 1800
# How many seconds to retry re-establishing a DB connection after
# disconnects. Setting this to 0 disables retries.
sql_alchemy_reconnect_timeout = 300
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32
# The number of task instances allowed to run concurrently by the scheduler
dag_concurrency = 16
# Are DAGs paused by default at creation
dags_are_paused_at_creation = True
# When not using pools, tasks are run in the "default pool",
# whose size is guided by this config element
non_pooled_task_slot_count = 128
# The maximum number of active DAG runs per DAG
max_active_runs_per_dag = 16
# Whether to load the examples that ship with Airflow. It's good to
# get started, but you probably want to set this to False in a production
# environment
load_examples = False
# Where your Airflow plugins are stored
plugins_folder = /usr/local/airflow/plugins
# Secret key to save connection passwords in the db
fernet_key =
# Whether to disable pickling dags
donot_pickle = False
# How long before timing out a python file import while filling the DagBag
dagbag_import_timeout = 30
# The class to use for running task instances in a subprocess
task_runner = BashTaskRunner
# If set, tasks without a `run_as_user` argument will be run with this user
# Can be used to de-elevate a sudo user running Airflow when executing tasks
default_impersonation =
# What security module to use (for example kerberos):
security =
# If set to False enables some unsecure features like Charts and Ad Hoc Queries.
# In 2.0 will default to True.
secure_mode = False
# Turn unit test mode on (overwrites many configuration options with test
# values at runtime)
unit_test_mode = False
# Name of handler to read task instance logs.
# Default to use task handler.
task_log_reader = task
# Whether to enable pickling for xcom (note that this is insecure and allows for
# RCE exploits). This will be deprecated in Airflow 2.0 (be forced to False).
enable_xcom_pickling = True
# When a task is killed forcefully, this is the amount of time in seconds that
# it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED
killed_task_cleanup_time = 60
# Whether to override params with dag_run.conf. If you pass some key-value pairs through `airflow backfill -c` or
# `airflow trigger_dag -c`, the key-value pairs will override the existing ones in params.
dag_run_conf_overrides_params = False
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
# If you set web_server_url_prefix, do NOT forget to append it here, ex:
# endpoint_url = http://localhost:8080/myroot
# So api will look like: http://localhost:8080/myroot/api/experimental/...
endpoint_url = http://localhost:8080
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
[lineage]
# what lineage backend to use
backend =
[atlas]
sasl_enabled = False
host =
port = 21000
username =
password =
[operators]
# The default owner assigned to each new operator, unless
# provided explicitly or passed via `default_args`
default_owner = Airflow
default_cpus = 2
default_ram = 2048
default_disk = 2048
default_gpus = 0
[hive]
# Default mapreduce queue for HiveOperator tasks
default_hive_mapred_queue =
[webserver]
# The base url of your website as airflow cannot guess what domain or
# cname you are using. This is used in automated emails that
# airflow sends to point links to the right web server
base_url = http://localhost:8080
# The ip specified when starting the web server
web_server_host = 0.0.0.0
# The port on which to run the web server
web_server_port = 8080
# Paths to the SSL certificate and key for the web server. When both are
# provided SSL will be enabled. This does not change the web server port.
web_server_ssl_cert =
web_server_ssl_key =
# Number of seconds the webserver waits before killing gunicorn master that doesn't respond
web_server_master_timeout = 120
# Number of seconds the gunicorn webserver waits before timing out on a worker
web_server_worker_timeout = 120
# Number of workers to refresh at a time. When set to 0, worker refresh is
# disabled. When nonzero, airflow periodically refreshes webserver workers by
# bringing up new ones and killing old ones.
worker_refresh_batch_size = 1
# Number of seconds to wait before refreshing a batch of workers.
worker_refresh_interval = 30
# Secret key used to run your flask app
secret_key = temporary_key
# Number of workers to run the Gunicorn web server
workers = 4
# The worker class gunicorn should use. Choices include
# sync (default), eventlet, gevent
worker_class = sync
# Log files for the gunicorn webserver. '-' means log to stderr.
access_logfile = -
error_logfile = -
# Expose the configuration file in the web server
expose_config = False
# Set to true to turn on authentication:
# https://airflow.incubator.apache.org/security.html#web-authentication
authenticate = False
# Filter the list of dags by owner name (requires authentication to be enabled)
filter_by_owner = False
# Filtering mode. Choices include user (default) and ldapgroup.
# Ldap group filtering requires using the ldap backend
#
# Note that the ldap server needs the "memberOf" overlay to be set up
# in order to user the ldapgroup mode.
owner_mode = user
dag_default_view = tree
dag_orientation = LR
demo_mode = False
log_fetch_timeout_sec = 30
hide_paused_dags_by_default = False
page_size = 100
# Use FAB-based webserver with RBAC feature
rbac = False
# Define the color of navigation bar
navbar_color = #007A87
# Default dagrun to show in UI
default_dag_run_display_number = 25
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
smtp_port = 25
smtp_mail_from = airflow#example.com
[celery]
celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_log_server_port = 8793
# The Celery broker URL. Celery supports RabbitMQ, Redis and experimentally
# a sqlalchemy database. Refer to the Celery documentation for more
# information.
# http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings
broker_url =
# The Celery result_backend. When a job finishes, it needs to update the
# metadata of the job. Therefore it will post a message on a message bus,
# or insert it into a database (depending of the backend)
# This status is used by the scheduler to update the state of the task
# The use of a database is highly recommended
# http://docs.celeryproject.org/en/latest/userguide/configuration.html#task-result-backend-settings
result_backend = sqlite:///~/airflow_tutorial/airflow.db
# Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start
# it `airflow flower`. This defines the IP that Celery Flower runs on
flower_host = 0.0.0.0
# The root URL for Flower
# Ex: flower_url_prefix = /flower
flower_url_prefix =
# This defines the port that Celery Flower runs on
flower_port = 5555
# Default queue that tasks get assigned to and that worker listen on.
default_queue = default
# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
# In case of using SSL
ssl_active = False
ssl_key =
ssl_cert =
ssl_cacert =
[celery_broker_transport_options]
# This section is for specifying options which can be passed to the
# underlying celery broker transport. See:
# http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_transport_options
# The visibility timeout defines the number of seconds to wait for the worker
# to acknowledge the task before the message is redelivered to another worker.
# Make sure to increase the visibility timeout to match the time of the longest
# ETA you're planning to use.
#
# visibility_timeout is only supported for Redis and SQS celery brokers.
# See:
# http://docs.celeryproject.org/en/master/userguide/configuration.html#std:setting-broker_transport_options
#
#visibility_timeout = 21600
[dask]
# This section only applies if you are using the DaskExecutor in
# [core] section above
# The IP address and port of the Dask cluster's scheduler.
cluster_address = 127.0.0.1:8786
# TLS/ SSL settings to access a secured Dask scheduler.
tls_ca =
tls_cert =
tls_key =
[scheduler]
# Task instances listen for external kill signal (when you clear tasks
# from the CLI or the UI), this defines the frequency at which they should
# listen (in seconds).
job_heartbeat_sec = 5
# The scheduler constantly tries to trigger new tasks (look at the
# scheduler section in the docs for more information). This defines
# how often the scheduler should run (in seconds).
scheduler_heartbeat_sec = 5
# after how much time should the scheduler terminate in seconds
# -1 indicates to run continuously (see also num_runs)
run_duration = -1
# after how much time a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
# How many seconds to wait between file-parsing loops to prevent the logs from being spammed.
min_file_parsing_loop_time = 1
dag_dir_list_interval = 300
# How often should stats be printed to the logs
print_stats_interval = 30
child_process_log_directory = ~/airflow_tutorial/logs/scheduler
# Local task jobs periodically heartbeat to the DB. If the job has
# not heartbeat in this many seconds, the scheduler will mark the
# associated task instance as failed and will re-schedule the task.
scheduler_zombie_task_threshold = 300
# Turn off scheduler catchup by setting this to False.
# Default behavior is unchanged and
# Command Line Backfills still work, but the scheduler
# will not do scheduler catchup if this is False,
# however it can be set on a per DAG basis in the
# DAG definition (catchup)
catchup_by_default = True
# This changes the batch size of queries in the scheduling main loop.
# If this is too high, SQL query performance may be impacted by one
# or more of the following:
# - reversion to full table scan
# - complexity of query predicate
# - excessive locking
#
# Additionally, you may hit the maximum allowable query length for your db.
#
# Set this to 0 for no limit (not advised)
max_tis_per_query = 512
# Statsd (https://github.com/etsy/statsd) integration settings
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
# The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run.
max_threads = 2
authenticate = False
[ldap]
# set this to ldaps://<your.ldap.server>:<port>
uri =
user_filter = objectClass=*
user_name_attr = uid
group_member_attr = memberOf
superuser_filter =
data_profiler_filter =
bind_user = cn=Manager,dc=example,dc=com
bind_password = insecure
basedn = dc=example,dc=com
cacert = /etc/ca/ldap_ca.crt
search_scope = LEVEL
[mesos]
# Mesos master address which MesosExecutor will connect to.
master = localhost:5050
# The framework name which Airflow scheduler will register itself as on mesos
framework_name = Airflow
# Number of cpu cores required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_cpu = 1
# Memory in MB required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_memory = 256
# Enable framework checkpointing for mesos
# See http://mesos.apache.org/documentation/latest/slave-recovery/
checkpoint = False
# Failover timeout in milliseconds.
# When checkpointing is enabled and this option is set, Mesos waits
# until the configured timeout for
# the MesosExecutor framework to re-register after a failover. Mesos
# shuts down running tasks if the
# MesosExecutor framework fails to re-register within this timeframe.
# failover_timeout = 604800
# Enable framework authentication for mesos
# See http://mesos.apache.org/documentation/latest/configuration/
authenticate = False
# Mesos credentials, if authentication is enabled
# default_principal = admin
# default_secret = admin
# Optional Docker Image to run on slave before running the command
# This image should be accessible from mesos slave i.e mesos slave
# should be able to pull this docker image before executing the command.
# docker_image_slave = puckel/docker-airflow
[kerberos]
ccache = /tmp/airflow_krb5_ccache
# gets augmented with fqdn
principal = airflow
reinit_frequency = 3600
kinit_path = kinit
keytab = airflow.keytab
[github_enterprise]
api_rev = v3
[admin]
# UI to hide sensitive variable fields when set to True
hide_sensitive_variable_fields = True
[elasticsearch]
elasticsearch_host =
# we need to escape the curly braces by adding an additional curly brace
elasticsearch_log_id_template = {dag_id}-{task_id}-{execution_date}-{try_number}
elasticsearch_end_of_log_mark = end_of_log
[kubernetes]
# The repository and tag of the Kubernetes Image for the Worker to Run
worker_container_repository =
worker_container_tag =
# If True (default), worker pods will be deleted upon termination
delete_worker_pods = True
# The Kubernetes namespace where airflow workers should be created. Defaults to `default`
namespace = default
# The name of the Kubernetes ConfigMap Containing the Airflow Configuration (this file)
airflow_configmap =
# For either git sync or volume mounted DAGs, the worker will look in this subpath for DAGs
dags_volume_subpath =
# For DAGs mounted via a volume claim (mutually exclusive with volume claim)
dags_volume_claim =
# For volume mounted logs, the worker will look in this subpath for logs
logs_volume_subpath =
# A shared volume claim for the logs
logs_volume_claim =
# Git credentials and repository for DAGs mounted via Git (mutually exclusive with volume claim)
git_repo =
git_branch =
git_user =
git_password =
git_subpath =
# For cloning DAGs from git repositories into volumes: https://github.com/kubernetes/git-sync
git_sync_container_repository = gcr.io/google-containers/git-sync-amd64
git_sync_container_tag = v2.0.5
git_sync_init_container_name = git-sync-clone
# The name of the Kubernetes service account to be associated with airflow workers, if any.
# Service accounts are required for workers that require access to secrets or cluster resources.
# See the Kubernetes RBAC documentation for more:
# https://kubernetes.io/docs/admin/authorization/rbac/
worker_service_account_name =
# Any image pull secrets to be given to worker pods, If more than one secret is
# required, provide a comma separated list: secret_a,secret_b
image_pull_secrets =
# GCP Service Account Keys to be provided to tasks run on Kubernetes Executors
# Should be supplied in the format: key-name-1:key-path-1,key-name-2:key-path-2
gcp_service_account_keys =
# Use the service account kubernetes gives to pods to connect to kubernetes cluster.
# It's intended for clients that expect to be running inside a pod running on kubernetes.
# It will raise an exception if called from a process not running in a kubernetes environment.
in_cluster = True
[kubernetes_secrets]
# The scheduler mounts the following secrets into your workers as they are launched by the
# scheduler. You may define as many secrets as needed and the kubernetes launcher will parse the
# defined secrets and mount them as secret environment variables in the launched workers.
# Secrets in this section are defined as follows
# <environment_variable_mount> = <kubernetes_secret_object>:<kubernetes_secret_key>
#
# For example if you wanted to mount a kubernetes secret key named `postgres_password` from the
# kubernetes secret object `airflow-secret` as the environment variable `POSTGRES_PASSWORD` into
# your workers you would follow the following format:
# POSTGRES_PASSWORD = airflow-secret:postgres_credentials
#
# Additionally you may override worker airflow settings with the AIRFLOW__<SECTION>__<KEY>
# formatting as supported by airflow normally.
All this setup is running under a c5.large EC2 in AWS
I want to access some charts -which I have saved in Looker- within Databricks. Part of this process is the authentication. I have one Looker auth-script which works but only pulls the tabular results into Databricks which corresponds to a Looker-View. Instead, I want ONLY the charts to be accessed in Databricks which will correspond to a Looker-look or Looker-space. However, when I follow the tutorial on https://discourse.looker.com/t/generating-a-powerpoint-presentation-from-all-looks-in-a-space/8191, I am not able to authenticate with their script. Hopefully, someone can help.
**Working auth-script for Looker-Views**
import looker_tools as tools
api=tools.LookerApi(
api_endpoint="abcd",
client_id=dbutils.secrets.get(scope="looker-api", key="looker_client_id"),
client_secret=dbutils.secrets.get(scope="looker-api",key="looker_client_secret")
)
token = api.login()
**Desired auth-script for Looker-Space/Looks as per tutorial link**
looker_instance = 'your-company.looker.com'
target_space = # 'Period over Period' Space on the Looker instance
client_id = 'xxxxxxxx'
client_secret = 'xxxxxxxx'
# instantiate Auth API
unauthenticated_client = looker_client.ApiClient(configuration=None)
unauthenticated_client.configuration.host = f'https://{looker_instance}:19999/api/3.0/'
unauthenticated_authApi = looker_client.ApiAuthApi(unauthenticated_client)
# authenticate client
token = unauthenticated_authApi.login(client_id=client_id, client_secret=client_secret)
client = looker_client.ApiClient(header_name='Authorization', header_value='token ' + token.access_token)
client.configuration.host = f'https://{looker_instance}:19999/api/3.0/'
I tried translating the code from Current to DESIRED auth-script but the error states the looker_client is not defined!
looker_instance = 'abcd'
target_space = 123
client_id = dbutils.secrets.get(scope="looker-api", key="looker_client_id")
client_secret = dbutils.secrets.get(scope="looker-api",key="looker_client_secret")
# instantiate Auth API
unauthenticated_client = looker_client.ApiClient(configuration=None) --> This line fails!!
unauthenticated_client.configuration.host = f'https://{looker_instance}:19999/api/3.0/'
unauthenticated_authApi = looker_client.ApiAuthApi(unauthenticated_client)
# authenticate client
token = unauthenticated_authApi.login(client_id=client_id, client_secret=client_secret)
client = looker_client.ApiClient(header_name='Authorization', header_value='token ' + token.access_token)
client.configuration.host = f'https://{looker_instance}:19999/api/3.0/'
I hope someone can help on how to define looker_client properly. Thanks.
It looks like this one was resolved here: https://discourse.looker.com/t/generating-a-powerpoint-presentation-from-all-looks-in-a-space/8191/15?u=izzy for those following along at home. There's another issue, but the NameError: name ‘looker_client’ is not defined error was resolved by adding a necessary import:
import looker_client_30 as looker_client
I'm using Mint distro 18.1.
I configure my laptop to join AD domain on Windows 2008R2 Server.
here my configuration:
/etc/krb5.conf
[libdefaults]
default_realm = ACMEAD.COM
clockskew = 300
ticket_lifetime = 60d
forwardable = true
proxiable = true
dns_lookup_realm = true
dns_lookup_kdc = true
krb4_config = /etc/krb.conf
krb4_realms = /etc/krb.realms
kdc_timesync = 1
[realms]
PRIMEURAD.COM = {
kdc = AD.ACME.COM:88
admin_server = AD.ACME.COM:749
default_domain = ACMEAD.COM
ticket_lifetime = 60d
}
[domain_realm]
.kerberos.server = ACMEAD.COM
.acmead.com = ACMEAD.COM
acmead.com = ACMEAD.COM
acmead = ACMEAD.COM
ticket_lifetime = 60d
[appdefaults]
pam = {
ticket_lifetime = 60d
renew_lifetime = 60d
forwardable = true
proxiable = false
retain_after_close = false
minimum_uid = 0
debug = false
}
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/kdc.log
admin_server = FILE:/var/log/kadmind.log
[login]
krb4_convert = true
krb4_get_tickets = false
/etc/samba/smb.conf
[global]
workgroup = primeurad
realm = primeurad.com
netbios name = lap-pc-1976
security = ADS
dns forwarder = 172.16.0.3
idmap config * : backend = tdb
idmap config *:range = 50000-1000000
template homedir = /home/%D/%U
template shell = /bin/bash
winbind use default domain = true
winbind offline logon = yes
winbind nss info = rfc2307
winbind enum users = yes
winbind enum groups = yes
winbind separator = +
winbind cache time = 300
winbind refresh tickets = yes
vfs objects = acl_xattr
map acl inherit = Yes
store dos attributes = Yes
preferred master = no
dns proxy = no
wins server = ad.primeur.com
wins proxy = no
inherit acls = Yes
acl group control = yes
load printers = no
debug level = 3
use sendfile = no
/etc/security/pam_winbind.conf
[global]
debug = no
debug_state = no
try_first_pass = yes
krb5_auth = yes
krb5_ccache_type = FILE
cached_login = yes
silent = no
# mkhomedir = yes
I'm able to login and authenticate my self. I add my PC to the domain with no problem. And I'm also able to login when I'm offline, which is what I want most.
I'm trying to increase the ticket lifetime to 60days, now if I type klist this is what I see
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: user1#ACMEAD.COM
Valid starting Expires Service principal
07/11/2017 12:25:02 07/11/2017 22:25:02 krbtgt/ACMEAD.COM#ACMEAD.COM
renew until 07/18/2017 12:24:59
It seems to me that takes the default of 10h instead of 60 days.
How can I increase it?
The Active Directory domain defaults take precedence here.
Best Practice would be to let the Maximum lifetime for Kerberos service ticket remain at the default of 10 hours. In various technical guides and Active Directory Group Policy, you will see that value written out as 600 minutes which is 10 hours, but shown as 600 minutes instead. I've never known why they did this. If you want to change the value, you will have to open up the Active Directory domain Group Policy Management Console tool (GPMC.msc) and edit the "Default Domain Policy" Group Policy Object. Once that GPO is opened, navigate to the following path, and change 600 minutes to its 60 day equivalent which would be 86400.
Computer Configuration\Windows Settings\Security Settings\Account Policies\Kerberos Policy\Maximum lifetime for service ticket
Reference: Maximum lifetime for service ticket
Note that changing this this would considered a security risk, as it gives potential hackers that much more time to potentially decrypt the service ticket and use for themselves. Just google "silver ticket attack". This is why why 10 hours is set as the default. It is also the default for all major Identity Management implementations using Kerberos, not just Active Directory. It is considered a trade-off between security and usability. You also asked "I see I have to increase also the kerberos principal but not sure how to do it". What did you mean by that? Did you mean "service principal name"? Or the Ticket Granting Ticket? Or the user account? What do you mean by "increasing it?" If you meant about increasing other Kerberos ticket lifetimes, such as the Ticket Granting Ticket, AKA "user ticket", then you can also modify them in the same area of the GPO mentioned above. In that GPO, the Ticket Granting Ticket (TGT), is written as "user ticket". It has the same lifetime of 10 hours. The screenshot below is from my lab, showing everything at the defaults:
EDIT/UPDATE:
To allow for offline logins to an AD domain-joined Windows PC when it is not connected to the network, or in case a domain controller is not available, you will have to allow for what is known as "cached credentials". This allows for the PC to re-use the service ticket and not be prompted to go get a new one. You can either do this on a one-off basis for a single machine, or domain-wide via a GPO. Both methods are described below:
For a single machine, just edit the Registry
For a domain-wide method, use a GPO
Note to the above: This setting would be ignored on a Linux OS joined to AD, since there is no Registry on Linux. In short, you cannot allow for cached (offline) logon to an AD domain for Linux laptops - that is a Windows OS-only feature.
I am using the Omnibus GitLab CE system with LDAP authentication.
Because of LDAP authentication, anyone in my company can sign in to GitLab and a new GitLab user account associated with this user is created (according to my understanding).
I want to modify it so that by default this new user (who can automatically sign in based on his LDAP credentials) cannot create new projects.
Then, I as the admin, will probably handle most new project creation.
I might give the Create Project permission to a few special users.
In newer versions of GitLab >= v7.8 …
This is not a setting in config/gitlab.yml but rather in the GUI for admins.
Simply navigate to https://___[your GitLab URL]___/admin/application_settings/general#js-account-settings, and set Default projects limit to 0.
You can then access individual users's project limit at https://___[your GitLab URL]___/admin/users.
See GitLab's update docs for more settings changed between v7.7 and v7.8.
git diff origin/7-7-stable:config/gitlab.yml.example origin/7-8-stable:config/gitlab.yml.example
For all new users:
Refer to Nick Merrill answer.
For all existing users:
This is the best and quick method to make changes to projects limits:
$ gitlab-rails runner "User.where(projects_limit: 10).each { |u| u.projects_limit = 0; u.save }"
( Update: This applies to versions <= 7.7:)
The default permissions are set in gitlab.yml
In omnibus, that is /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
Look for
## User settings
default_projects_limit: 10
# default_can_create_group: false # default: true
Setting default_projects_limit to zero, and default_can_create_group to false may be what you want.
Then an admin can change the limits for individual users.
Update:
This setting was included in the admin GUI in version 7.8 (see answer by #Nick M). At least with Omnibus on Centos7 an upgrade retains the setting.
Note that the setting default_can_create_group is still in gitlab.yml.
Here's my quick-and-dirty Python script which you can use in case you already have some users created and want to change all your existing users to make them unable to create projects on their own:
#!/usr/bin/env python
import requests
import json
gitlab_url = "https://<your_gitlab_host_and_domain>/api/v3"
headers = {'PRIVATE-TOKEN': '<private_token_of_a_user_with_admin_rights>'}
def set_user_projects_limit_to_zero (user):
user_id = str(user['id'])
put = requests.put(gitlab_url + "/users/" + user_id + "?projects_limit=0", headers=headers)
if put.status_code != 200:
print "!!! change failed with user id=%s, status code=%s" % (user_id, put.status_code)
exit(1)
else:
print "user with id=%s changed!" % user_id
users_processed = 0
page_no = 1
total_pages = 1
print "processing 1st page of users..."
while page_no <= total_pages:
users = requests.get(gitlab_url + "/users?page=" + str(page_no), headers=headers)
total_pages = int(users.headers['X-Total-Pages'])
for user in users.json():
set_user_projects_limit_to_zero(user)
users_processed = users_processed + 1
print "processed page %s/%s..." % (page_no, total_pages)
page_no = page_no + 1
print "no of processed users=%s" % users_processed
Tested & working with GitLab CE 8.4.1 052b38d, YMMV.