I'm trying to follow this article: https://airflow.apache.org/docs/apache-airflow/1.10.6/howto/write-logs.html
so Airflow will start writing logs to blob storage but the problem is I do not now how to configure Airflow to do that. In my case, Airflow is running on Kubernetes Cluster and deployment is done via Helm chart.
I tried to log into webserver Pod but #airflow user is not authorized to create any files in AIRFLOW_HOME directory. I was trying to use sudo but I can't find password (I'm not even sure if it works airflow is not in sudoers anyway )
Should I do all of this in docker image and just restart Airflow?
I am not too familiar with Helm Chart setups but maybe it is worth a try to add the variables for remote logging in the values.yaml file like this:
config:
logging:
remote_logging=True
log_conn_id=<their AWS conn id>
remote_base_log_folder=s3://bucket-name/logs
Plus define a normal Airflow connection either via an ENV variable in the Dockerfile or via the UI and provide that as the AWS conn id.
If that does not work my next attempt would be to use ENV variables for all of the settings in the Dockerfile:
# allow remote logging and provide a connection ID
ENV AIRFLOW__LOGGING__REMOTE_LOGGING=True
ENV AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=${AMAZONS3_CON_ID}
# specify the location of your remote logs using your bucket name
ENV AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3://${S3BUCKET_NAME}/logs
# optional: serverside encryption for S3 logs
ENV AIRFLOW__LOGGING__ENCRYPT_S3_LOGS=True
Also if you are on pre-2 Airflow consider upgrading if you can, it is worth it imho. :)
Related
Is there a way for AWS credentials passed as environment variables to the docker run command to be put to use for getting the caller identity details while the container is running?
This is the docker run command being executed in the application
docker run -e AWS_ACCESS_KEY={user_credentials["AccessKeyId"]} -e AWS_SECRET_ACCESS_KEY={user_credentials["SecretAccessKey"]} -e AWS_SESSION_TOKEN={user_credentials["SessionToken"]} image_name --rm'
The answer is actually simple, but definitely something I was not aware of.
Initialized an STS client with the given credentials and then made a call to to get the caller identity details. Retrieved the credentials using the OS module. The scope of my application is very limited, hence using the credentials to get the user account details. This is what worked for me.
sts_client = boto3.client('sts', aws_access_key_id=os.environ['AccessKeyId'],
aws_secret_access_key=os.environ['SecretAccessKey'],
aws_session_token=os.environ['SessionToken'])
I'm using an Azure App Service to run my docker image. Running my docker container requires using a couple of environment variables. Locally, when I run my container I just do something like:
docker run -e CREDENTIAL -e USERNAME myapp
However, in the Azure App Service, after defining CREDENTIAL and USERNAME as Application Settings, I'm unsure how to pass these to the container. I see from the logs that on startup Azure passes some of its own environment variables, but if I add a startup command with my environment variables, it tacks it on at the end of the one generated by Azure creating an invalid command. How can I pass mine to the container?
As I understand you want to set environment variables in that docker container with -e option.
You don't need to use startup command for that. Pass these variables as application settings:
Application Settings are exposed as environment variables for access by your application at runtime.
Documentation
Background
I have a node app that essentially needs to use aws-sdk to access S3 bucket and perform other operations. In my local machine I have a .aws config file that sits in ~/.aws/config. I want to run this node app in a docker container (so that I can deploy it to EC2 later).
The problem
How do I configure my docker container to use the aws config file so that the node app running in it can use aws-sdk?
Clarification
I do not want to use IAM. I specifically want to use aws config file which has the secret access key etc.
You can do what AWS is doing when they explain how to use their containers on local machines. For example, for local AWS Glue they simply share the ~/.aws/ with the docker container using:
-v ~/.aws:/root/.aws:ro
Obviously you would have to adjust the paths above to match your local and docker setup.
The other way is to pass the AWS credentials using docker environment variables.
We have a Jenkins virtual machine on GCE which deals with deployments, including the ones we do to GKE. We've tried to deploy a project which we have not touched for some time. The deployment failed when calling
kubectl set image deployment my-deployment my-deployment=gcr.io/my-project/my-project:version-tag
getting this error:
Error from server (Forbidden): deployments.extensions "my-deployment" is forbidden: User "client" cannot get resource "deployments" in API group "extensions" in the namespace "default"
The weird thing is, if I log in to the machine, use my Linux user + my gcloud user, I can deploy fine. But when switching to the jenkins user using su - jenkins and then authorizing gcloud with my user I get this same error that our deploy account gets.
Please advise how to fix.
It seems related to cluster RBAC configurations. Did you enable the RBAC fo Google Groups? In this case you should follow the instructions in the documentation above or disable it.
Otherwise, ss Raman Sailopal stated, you can try this:
with your regular user run kubectl config get-contexts to retrieve your current context
copy from /home/Linux user/.kube/config to /home/jenkins/.kube/config
change user to jenkins and be sure you're using the same context by running kubectl config get-contexts and kubectl config set-context ...
try your rights with:
# Check to see if I can create deployments in any namespace
kubectl auth can-i create deployments
# Check to see if I can list deployments in my current namespace
kubectl auth can-i list deployments.extensions
I followed the guide to getting Kubernetes running in Azure here:
http://kubernetes.io/docs/getting-started-guides/coreos/azure/
In order to create pods, etc., the guide has you ssh into the master node kube-00 in the cloud service and run kubectl commands there:
ssh -F ./output/kube_randomid_ssh_conf kube-00
Once in you can run the following:
kubectl get nodes
kubectl create -f ~/guestbook-example/
Is it possible to run these kubectl commands without logging to the master node, e.g., how can I set up kubectl to connect to the cluster hosted in Azure from my development machine instead of ssh'ing into the node this way?
I tried creating a context, user and cluster in the config but the values I tried using did not work.
Edit
For some more background the tutorial creates the azure cluster using a script using the Azure CLI. It ends up looking like this:
Resource Group: kube-randomid
- Cloud Service: kube-randomid
- VM: etcd-00
- VM: etcd-01
- VM: etcd-02
- VM: kube-00
- VM: kube-01
- VM: kube-02
It creates a Virtual Network that all of these VM's live in. As far as I can tell all of the machines in the cloud service share a single virtual IP.
The kubectl command line tool is just a wrapper to execute remote HTTPS API REST calls on the kubernetes cluster. If you want to be able to do so from your own machine you need to open the correct port (443) on your master node and pass along some parameters to the kubectl tool as specified in this tutorial:
https://coreos.com/kubernetes/docs/latest/configure-kubectl.html