On a self-hosted ELK environment I am aware of how to see errors in the Logstash pipeline to detect log records being thrown away.
But how can I find these errors on a ELK environment hosted by elastic.co?
There is no Logstash in the elastic cloud service hosted by elastic.co, only elasticsearch and kibana instances are available.
If you need logstash in your deployment you will need to host it elsewhere, on-premises or other cloud services, and configure your instance to ship data to your cloud service.
Related
I have a Nodejs server which is used to get the logs of a few servers which are sitting on VMs by connecting to the VM via SSH and using the tail command on the log file.
I wanted to know if there is a similar way or any way to get the logs of an app who's container is on openshift and not a VM, and if there is a way to get the logs of all the pods as one.
Thanks
Did you check the OpenShift documentation?
You can install a full EFK stack directly in OCP via an operator and /or forward the logs of everything running in OCP to an external EFK stack
https://docs.okd.io/latest/logging/cluster-logging.html
This works if the logs are written to stdout/stderr in the pods
Here are some other techniques around centralizing logs with kubernetes:
https://kubernetes.io/docs/concepts/cluster-administration/logging/
I have deployed a web application on Azure Kubernetes Service (AKS). I want to generate access logs (which include HTTP method, path, request body, if possible HTTP headers, and timestamp) of deployed web application and not of Kubernetes Cluster, Nodes and Pods. I have used Application Gateway, but it is creating logs of Kubernetes Cluster, Nodes or Pods. I have also used Log Analytics on Azure, but it is also not giving me required logs.
You need your application to generate those logs in some form. kubernetes doesnt provide networking logs. Easiest way is outputting to stdout, or, perhaps using Application insights. Another option is using something like Zipkin or Jaeger to capture those.
I have a website on digital ocean and on AWS I have ELK - Elastic Search, Kibana, Logstash - stack. I'm trying to understand how feed the logs files from web app to ELK stack.
Is it possible to feed them via the internet between different servers?
Do you I have to install nginx to be able to see kibana reports?
Do I have to install ssl certificates for my AWS stack?
Answer to you question is:
yes
no
no
I am working on a task to deploy ElasticSearch as an azure cloud service with 3 Master + Data Nodes and Kibana 4 as an Azure Website. The ES (2.3.1) service is active and I get the green health status on Chrome sense on retreiving health of ES cluster.
I have used the Kibana 4 binaries in https://github.com/iremmats/kibana4-azure-website for creating a json application that can be deployed as Azure website. The json application that I have created along with configuration for server.js, kibana.yml runs successfully on local VS using node.exe and I am able to discover my ES instance indices on it. However, the moment I deploy this nodejs application to Azure Website and attempt to browse Kibana 4 on Azure website I get Kibana: Bad Gateway exception. Looking into the network timeline of azure website hit, it throws HTTP 502 error on attempting to connect with ElasticSearch cloud service instance.
Below is network timeline screen shot.The IP address mentioned is the IP address of ES cloud service.
Kibana Bad Gateway error and network timeline
It seems that the issue was caused by Azure virtual network configuration for your ElasticSearch cloud service.
I think you may try to do two ways below for the configuration.
Adding a new endpoint for the cloud services to allow the inbound access for port 9200 at the endpoint setting on Azure portal.
Following the article How to manage NSGs using the Azure portal to configure the access rules for subnets in your virtual network between the website and cloud services.
However, I think it's not necessary for integrating the independency Kibana instance with the ElasticSearch instances. In fact, Azure has supplied the Elasticsearch and Kibana integration environment in the Azure Marketplace that you can directly create without any additional configuration. Please try to access the url https://ms.portal.azure.com/#blade/Microsoft_Azure_Marketplace/GalleryFeaturedMenuItemBlade/selectedMenuItemId/home/searchQuery/Elasticsearch%20and%20Kibana/resetMenuId/ to see it, or search the words "Elasticsearch and Kibana" in the search bar of Azure Marketplace to find it.
Any concern, please feel free to let me know.
I am trying to use elasticsearch for search functionality in my website. I have setup elasticsearch in my local development environment and it works fine.
I have setup the elasticsearch on Ubuntu Linux machine as given here
I have opened endpoints (ports) 9200 and 9300 on the Linux elasticsearch server.
I have created a virtual network and my website and this elasticsearch server are in the same virtual network.
I am able to connect to my Linux elasticsearch server from my local machine using a public IP.
I could not connect to Linux elasticsearch server from my Azure website either from internal IP like 10.0.0.4 and configured public IP.
I want to check whether connecting to elasticsearch port 9200 from Azure website is possible or not? Also please suggest a way to check existing configuration to troubleshoot the Azure website to elasticsearch connectivity issue.
It is definitely possible. We do it for audit logging and event logging from both azure websites and cloud services.
In the above article I mentioned, it says it is not possible, I tried but could not connect. There is something called hybrid connection but that can be setup only for windows servers. Here is the line that is mentioned in this article
http://code972.com/blog/2014/07/74-the-definitive-guide-for-elasticsearch-on-windows-azure
"Unfrotunately, Azure Websites do not currently support running on a Virtual Network, meaning you can only deploy websites as a Cloud Service if you want them to access the cluster this way."
I checked few more blog posts and found such comments. So, I have given up using Linux server and taken a Windows 2012R2 searver, configured elasticsearch and was able to access elasticsearch cluster now.
If anybody has a solid information on how to connect Azure website to Linux VM please give me links or steps.