winston logs to EFK Stack on GKE - logstash

I'm writing my nodejs logs to a file using winston js and storing them on /tmp/logs volume on my host node on GKE. Now, I'm trying to add the EFK Stack to store them and visualize them but my problem is that I'm new to the whole logging process and I'm facing troubles understanding the configuration of fluentd. I've tried to configure the stack on my local cluster with the help of this tutorial and did visualize the stdout and stderr logs of my containers but those of my winston files are not shown.
To do so:
How should I tell fluentd to read *.log files that I'm storing under /tmp/logs on the cluster?
What is the best way to install EFK stack on GKE cluster (ECK....) since I already have Elasticsearch and Kibana installed for production purposes. I'm thinking of isolating the new ones under a logging namespace but I also found that I can install the monitoring stack on a GCE machine directly.
In terms of resource management, does the EFK stack require a lot of RAM, CPU...?
NB: GCP used to have Elastic gke logging on the marketplace that I wanted to try but it's not there anymore.

fluentd config can be entered into ConfigMap, for example:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
data:
fluentd.conf: |
<source>
type tail
format none
path /tmp/logs/1.log
pos_file /tmp/logs/1.log.pos
tag count.format1
</source>
<source>
type tail
format none
path /tmp/logs/2.log
pos_file /tmp/logs/2.log.pos
tag count.format2
</source>
<match **>
type google_cloud
</match>
There are also complete tutorials on configureing EFK on GKE here and here.

Related

Node container unable to locate Hashicorp Vault secrets file on startup on AWS EKS 1.24

We have a small collection of Kubernetes pods which run react/next.js UIs in a node 16 alpine container (node:16.18.1-alpine3.15 to be precise). All of this runs in AWS EKS 1.23. We make use of annotations on these pods in order to inject secrets from Hashicorp Vault at start up. The annotations pull the desired secrets from Vault and write these to a file on the pod. Example of said annotations below :
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-init-first: "true"
vault.hashicorp.com/agent-pre-populate-only: "true"
vault.hashicorp.com/role: "onejourney-ui"
vault.hashicorp.com/agent-inject-secret-config: "secret/data/onejourney-ui"
vault.hashicorp.com/agent-inject-template-config: |
{{- with secret "secret/data/onejourney-ui" -}}
export AUTH0_CLIENT_ID="{{ .Data.data.auth0_client_id }}"
export SENTRY_DSN="{{ .Data.data.sentry_admin_dsn }}"
{{- end }}
When the pod starts up, we source this file (which is created by default at /vault/secrets/config) to set environment variables and then delete the file. We do that with the following pod arguments in our helm chart :
node:
args:
- /bin/sh
- -c
- source /vault/secrets/config; rm -rf /vault/secrets/config; yarn start-admin;
We recently upgraded some of AWS EKS clusters from 1.23 to 1.24. After doing so, we noted that our node applications were failing to start and entering a crash loop. Looking in the logs of these containers, the problem seemed to be that the pod was unable to locate the secrets file anymore.
Interestingly, the Vault init container completed successfully and shows that the file was successfully created...
Out of curiosity, I removed the node args to source the file which allowed the container to start successfully, but I found when execing into the pod, the file WAS infact present and had the content I was expecting. The file also had the correct owner and permissions as we see in a good working instance in EKS 1.23.
We have other containers (php-fpm) which consume secrets in the same manner however these were not affected on 1.24, only node containers were affected. There were no namespace, pod or deployment annotations I saw added which would have been a possible cause. After rolling the cluster back down to EKS 1.23, the deployment worked as expected.
I'm left scratching my head as to why the pod is unable to source that file on 1.24. Any suggestions on what to check or a possible cause would be greatly appreciated.

How to see more realtime logs when using spark-submit?

I am using 'spark-submit' with configuration file and package option and it is taking very long time to run..
How to 'TURN-ON' more logging (in realtime) so that can see where is the bottleneck (e.g. maybe a request to specific server is being made where I do not have access etc.)..
I would ideally want to see everything - from which libraries are being loaded to which request and to which server is being made.
Thanks.
In most cases, you can see all relevant information either on the Spark UI for currently running jobs (usually, this service is reachable at port 4040 of your driver) or (if your system has one) on the Spark History Server.
You can use the below parameters when you are using Spark on YARN.
--driver-java-options "-Dlog4j.error=true" --verbose
Or
You can always do the below to get logs from YARN
Use the following command format to view all logs of a particular type for a running application:
yarn logs -applicationId <Application ID> -log_files <log_file_type>
For example, to view only the stderr error logs:
yarn logs -applicationId <Application ID> -log_files stderr
The -logFiles option also supports Java regular expressions, so the following format would return all types of log files:
yarn logs -applicationId <Application ID> -log_files .*

Installing Istio in Kubernetes with automatic sidecar injection: istio-inializer.yaml Validation Failure

I'm trying to install Istio with automatic sidecar injection into Kubernetes. My environment consists of three masters and two nodes and was built on Azure using the Azure Container Service marketplace product.
Following the documentation located here, I have so far enabled RBAC and DynamicAdmissionControl. I have accomplished this by modifying /etc/kubernetes/istio-inializer.yaml on the Kubernetes Master by adding the following content outlined in red and then restarting the Kubernetes Master using the Unix command, reboot.
The next step in the documentation is to apply the yaml using kubectl. I assume that the documentation intends for the user to clone the Istio repository and cd into it before this step but that is unmentioned.
git clone https://github.com/istio/istio.git
cd istio
kubectl apply -f install/kubernetes/istio-initializer.yaml
After which the following error occurs:
user#hostname:~/istio$ kubectl apply -f install/kubernetes/istio-initializer.yaml
configmap "istio-inject" configured
serviceaccount "istio-initializer-service-account" configured
error: error validating "install/kubernetes/istio-initializer.yaml": error validating data: found invalid field initializers for v1.ObjectMeta; if you choose to ignore these errors, turn validation off with --validate=false
If I attempt to execute kubectl apply with the mentioned flag, validate=false, then this error is generated instead:
user#hostname:~/istio$ kubectl apply -f install/kubernetes/istio-initializer.yaml --validate=false
configmap "istio-inject" configured
serviceaccount "istio-initializer-service-account" configured
deployment "istio-initializer" configured
error: unable to recognize "install/kubernetes/istio-initializer.yaml": no matches for admissionregistration.k8s.io/, Kind=InitializerConfiguration
I'm not sure where to go from here. The problem appears to be related to the admissionregistration.k8s.io/v1alpha1 block in the yaml but I'm unsure what specifically is incorrect in this block.
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: InitializerConfiguration
metadata:
name: istio-sidecar
initializers:
- name: sidecar.initializer.istio.io
rules:
- apiGroups:
- "*"
apiVersions:
- "*"
resources:
- deployments
- statefulsets
- jobs
- daemonsets
Installed version of Kubernetes:
user#hostname:~/istio$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:21:54Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:21:54Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
I suspect this is a versioning mismatch. As a follow up question, is it possible to deploy a version of kubernetes >= 1.7.4 to Azure using ACS?
I'm fairly new to working with Kubernetes so if anyone could help I would greatly appreciate it. Thank you for your time.
Seems to be a versioning problem as the alpha feature is supported for k8s version> 1.7 as mentioned here (https://kubernetes.io/docs/admin/extensible-admission-controllers/#what-are-initializers).
1.7 introduces two alpha features, Initializers and External Admission
Webhooks, that address these limitations. These features allow admission
controllers to be developed out-of-tree and configured at runtime.
And it is possible to deploy a version of kubernetes >= 1.7.4 to Azure. Note sure about the deployed version using the portal. But if you use acs-egnine to generate the ARM template, it is possible to deploy a cluster with version 1.7.5.
You can refer here for the procedures https://github.com/Azure/acs-engine. Basically it involves three steps. First, you should create the json file by referring to the clusterDefinition section. To use version 1.7.5, you should specify the attribute "orchestratorRelaease" to be "1.7" and also enable the RBAC by specifying the attribute "enableRbac" to be true. Second, use the acs engine (version >= 0.6.0) to parse the json file to ARM template (azuredeploy.json & azuredeploy.parameters.json should be created). Lastly, use the command "New-AzureRmResourceGroupDeployment" in powershell to deploy the cluster to Azure.
Hope this helps :)

How to send node.js logs to Cloudwatch Logs from Elastic Beanstalk Docker application?

Amazon offers these readymade files for sending Tomcat/Apache/nginx logs to Cloudwatch Logs, which work great.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html
However for my purposes they only send nginx logs, which isn't really sufficient and unfortunately they also provide zero documentation on the file format. What I'm trying to achieve is to send node.js logs from my Docker application to Cloudwatch (since autoscaling makes instances come and go).
So having files like /var/log/eb-docker/containers/eb-current-app/add839a3b599-stdouterr.log to appear in Cloudwatch.
So, what I have tried so far is adapt the webrequests config from the link above:
##############################################################################
## Sends docker logs to CloudWatch Logs
##############################################################################
Mappings:
CWLogs:
ApplicationLogGroup:
LogFile: "/var/log/eb-docker/containers/eb-current-app/*-stdouterr.log"
TimestampFormat: "%Y-%m-%d %H:%M:%S"
Outputs:
ApplicationLogGroup:
Description: "The name of the Cloudwatch Logs Log Group created for this environments web server access logs. You can specify this by setting the value for the environment variable: WebRequestCWLogGroup. Please note: if you update this value, then you will need to go and clear out the old cloudwatch logs group and delete it through Cloudwatch Logs."
Value: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0ApplicationLogGroup"}
Resources :
AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0ApplicationLogGroup: ## Must have prefix: AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0
Type: "AWS::Logs::LogGroup"
DependsOn: AWSEBBeanstalkMetadata
DeletionPolicy: Retain ## this is required
Properties:
LogGroupName:
"Fn::GetOptionSetting":
Namespace: "aws:elasticbeanstalk:application:environment"
OptionName: ApplicationLogGroup
DefaultValue: {"Fn::Join":["-", [{ "Ref":"AWSEBEnvironmentName" }, "stdouterr"]]}
RetentionInDays: 14
## Register the files/log groups for monitoring
AWSEBAutoScalingGroup:
Metadata:
"AWS::CloudFormation::Init":
CWLogsAgentConfigSetup:
files:
## any .conf file put into /tmp/cwlogs/conf.d will be added to the cwlogs config (see cwl-agent.config)
"/tmp/cwlogs/conf.d/stdouterr.conf":
content : |
[stdouterr]
file = `{"Fn::FindInMap":["CWLogs", "ApplicationLogGroup", "LogFile"]}`
log_group_name = `{ "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0ApplicationLogGroup" }`
log_stream_name = {instance_id}
datetime_format = `{"Fn::FindInMap":["CWLogs", "ApplicationLogGroup", "TimestampFormat"]}`
mode : "000400"
owner : root
group : root
Unfortunately this doesn't seem to work. :/
Also, does anyone have any idea if logs appear at all if fe. the timestamp format is wrong? Specially important since by default exceptions don't really have timestamps, so the actual errors would just disappear.
My application log lines currently look like this:
2016-07-05 09:11:31 ::1 - GET / 200 (5.107 ms)
You can use this link to setup cloudwatch agents on your beanstalk instances (if you haven't already) - http://serebrov.github.io/html/2015-05-20-cloudwatch-setup.html.
Next - try to send the files in /var/lib/docker/containers//.json to collect your docker logs. It's where the containers stdout and stderr is written to.

Tag AWS beanstalk deployment using .config file in .ebextensions

I added a scripts.config file to .ebextensions at the root of my Node app deployed in beanstalk.I did not see the tags for the EC2 instances in the console. Nor did I see any mention of 1_add_tags in beanstalk logs. What did I do wrong and how do I find out if the commands in the script.config were called at all!
The config file in .ebextensions is as follows ....
01_add_tags:
command: ec2-create-tags $(ec2-metadata -i | cut -d ' ' -f2) --tag Environment=Production --tag Name=Proxy-Server --tag Application=something
env:
EC2_HOME: /opt/aws/apitools/ec2
EC2_URL: https://ec2.ap-southeast-2.ama...
JAVA_HOME: /usr/lib/jvm/jre
PATH: /bin:/usr/bin:/opt/aws/bin/
Cheers,
Prabin
Amazon's answer to the problem. (This worked for me) ...
You can utilise the ebextensions to execute certain commands on instance boot.
Supposing that you want to implement this on Linux based containers. I have formulated a sample config file for you and attached to this case.
Please follow below guidelines :
In the AWS Management console, check the IAM Role/Instance profile used by beanstalk. By default it uses "aws-elasticbeanstalk-ec2-role". Add permissions for this role to create new tags (ec2:CreateTags).
If you do not have ".ebextensions" folder at the root of your application or the "WEB-INF" folder, then create the folder.
Modify the key value pairs in the config file. Multiple pairs are separated by a space.
A sample snippet is as below:
{
"container_commands": {
"01_add_tags": {
"command": "aws ec2 create-tags --resources $(GET http://169.254.169.254/latest/meta-data/instance-id) --tags Key=ClientName,Value=testClient Key=NewTag,Value=new-value --region us-east-1"
}
}
}
Add the modified config file in the ".ebextensions" folder.
Upload this version to beanstalk. It should launch new instances and execute the config file.
Please give it sometime, preferably till the instances pass EC2 instance status checks. Refresh the page for the additional tags to be displayed.
Please note that we are using "Container_commands" instead of "Command" used in the blog.
Container Commands run after the application and web server have been set up and the application version file has been extracted, but before the application version is deployed. This is important as these commands have access to environment variables such as your AWS security credentials set by the instance-profile.
I would recommend you to go through the restrictions for AWS Resources tagging mentioned at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions
I would like to highlight that maximum number of tags per resource is 10.
Also check the table for tagging support for certain resource. For example, currently tagging is not supported for ELB.
I had the similar problem where I tried to install libjpeg using the ./ebextensions/foo.config file. I tried everything but was never able to find a good solution.
I was able to solve it though, by setting up a completely new Elastic Beanstalk Application and then deploying my same version on the new instance instead. When I did this everything was installed perfectly and working fine.
Check out my answers here:
https://stackoverflow.com/a/23109410/2335675
https://stackoverflow.com/a/23131959/2335675
Hope this fixes your issues as well.

Resources