Various of my docker containers export prometheus metrics, however our prometheus installation requires to extract all metrics only from only one endpoint. Unfortunately, this cannot be changed. Thus, I need to aggregate all metrics at one point from with the prometheus installation can scrape the metrics.
Moreover, it would be great if this program or script could provide additional logic on how to deal with the same metrics which are exported by different endpoints. For example, if I just concat the various metric sites together Prometheus has a problem with interpreting identical HELP Texts for the metrics.
Do you know of a way, script or a docker image which can be used to aggregate Prometheus metrics?
Best
Andy
Prometheus can do that. You just need to setup a Prometheus instance that can reach all endpoints and then you can use federation to scrape these metrics from the other Prometheus. Sorry if this sounds paradox, but this is actually the proper way to do it.
You can configure your intermediate Prometheus instance with little to no retention and use it as a proxy. This is certainly not the best way to use Prometheus, but if this is a hard requirement ...
Related
I currently am creating a faster test harness for our team and will be recording a baseline from our prod sdk run and our staging sdk run. I am running the tests via jest and want to eventually fire the parsed requests and their query params to a datastore of sorts and have a nice UI around it for tracking.
I thought that Prometheus and Grafana would be able to provide that, but after getting a little POC for myself working yesterday it seems that this combo is more used for tracking application performance rather than request log handling/manipulation/tracking.
Is this the right tool to be using for what I am trying to achieve and if so might someone shed some light on where I might find some more reading aligned with what I am trying to do?
Prometheus does only one thing and it well. It collects metrics and store them. It is used for monitoring your infrastructure or applications to monitor performance, availability, error rates etc. You can write rules using PromQL expression to create alert based on conditions and send them to alert manager which can send it to Pager duty, slack, email or any ticketing system. Even though Prometheus comes with a UI for visualising the data it's better to use Grafana since it's pretty good with it and easy to analyse data.
If you are looking tools for distributed tracing you can check Jaeger
I have a few IIS sites I would like to monitor using Prometheus. Specifically monitor and alert on outages. I cannot figure out how to grab a metric when a site experiences an outage. Ideally, I would like when a site goes down to be able to provide that information, scrape the metric to Prometheus and then using the Prometheus Alertsmanager send it to our Slack webhook. I know there are tools specifically for this such as Pingdom, Uptime Robot, StatusCake but if I could do this using Prometheus, a tool we are already using, that would be better.
I am currently running WMI Exporter to get metrics.
I believe you're interested in blackbox-exporter (see https://github.com/prometheus/blackbox_exporter) to monitor targets via HTTP requests.
Once you've installed the exporter and configured targets, you'll be interested in alerting on the probe_success metric.
It can be done using windows_exporter only. You should add W3SVC as the service to monitor. Hope this helps!
I am trying to understand the use of application insights for capturing the module logs and considering appinsights as a potential option.
I am keen on understanding how would the appinsights work considering there would be multiple devices each running the same modules where modules are configured to send log data to appinsights. The type of data I want to capture are container logs which are currently being sent to stderr/stdout streams.I am expecting this to work on windows devices , hence the logspout project may not be useful here (https://github.com/veyalla/logspout-loganalytics) but i want to do something similar.
I am trying to figure out a design where module logs from multiple edge devices can be captured using appinsights. It would be immensely useful for me to know if appinisghts is really suited for the problem I am trying to solve and how can it be used for multiple devices.
I'm not aware of a good/secure solution for Windows containers that does a continuous push of module logs to log analytics.
Since the built-in log pull via edgeAgent is experimental, we might change the API or make some modifications but we're unlikely to pull the feature entirely without an equivalent alternative.
i am using nodejs as my server with express. I am logging all my request and response on server. Is there any package available to read my logs and generate graphical report like how many requests we got and how many succeeded. What was the request received and responded. Is there a package which can track all these details for me?
It sounds like you're trying to get some performance metrics about your application which is great. There are many different ways you can go with this, here are a few suggestions for you to weigh up.
Non-real-time performance metrics
If you don't care about seeing the services real-time metrics you might want to create something to process them into a CSV and use something like excel or google sheets to generate graphs from them. If you need something immedietely and don't need to respond to things "in the moment" when a dip happens then this is a good quick and dirty solution.
Real-time performance metrics using SaaS software
If you want the metrics but don't want to host the systems yourself you might want to checkout services such as DataDog. They provide dashboards and graphs as a service. You can use something like statsd to get metrics into DataDog, or use their own integrations. They have a lot of integrations with cloud providers like AWS, GCP, and Azure for machine metrics (CPU etc). They also have packages for inteacting with your application itself such such as their ExpressJS package.
Real-time performance metrics using self-hosted solutions
I've often used a self-hosted approach as I find the pricing often scales a bit better. The setup is fairly simple.
Use a statsd package for all system components (nginx, nodejs, postgres, etc) to publish metrics to the statsd daemon.
The statsd daemon self-hosted somewhere (maybe a proxy cluster if you're working on large applications).
Self-hosted Graphite to consume metrics from the statsd daemon. Graphite is a software package designed for aggregating metrics and has an API for producing static graph images.
Self-hosted Grafana that pulls metrics from graphite. Grafana is a real-time dashboarding software. It allows you to create multiple dashboards that hook into various data sources such as Graphite or other time series data stores.
The self-hosting route can take a day to setup but it does mean you don't increase your costs per-host. It's also easy to put behind internal networks if that's a requirement for your organisation.
Personally, I would recommend either real-time performance metrics approaches. If your application is small and doesn't have many hosts then services like DataDog could be useful and cost effective but if you do need to scale up you'll find your costs sky rocketing. At that point you might decide to move over to a self-hosted infrastructure.
I'm attempting to query our DATADOG hub and display some metric graphs. However, it appears the default way to do this is using an embed script generated by DATADOG and utilizing that in your app. I'm actually wanting to draw the graphs on my side, using their API data so I'm better able to control the size, look and flexibility of the graphs.
Is this something that is possible? Rather new to DATADOG and everything seems to be done in an iFrame, which I do not want.
Additionally, I found a package which I believe may be of use for Node?: http://brettlangdon.github.io/node-dogapi/#embed-create
D.D. Graphs Docs: http://docs.datadoghq.com/api/#graphs
Any advice would be greatly appreciated, I have not seen anything similar on S.O.
I think what you actually want is the metrics-query API endpoint? http://docs.datadoghq.com/api/#metrics-query
There are also a few Node.JS libraries that may be able to handle this kind of metric querying for you: http://docs.datadoghq.com/libraries/#community-node