I have an application that is generating 3 kind of log files
Transaction log
Server log
Fatal log
and I want to analyse the performance of my server using appdynamics so what kind of data my logs should be generating to generate analytics for server health, performance, throughput, server utilization?
That's the beauty of APM is you don't need to deal with logging to get performance data. APM tools instrument the applications themselves regardless of what the code does (logging, generating metrics, etc). AppDynamics can collect log data, and provide similar capabilities to what a log analytics tool can do, but it's of less value than the transaction tracing and instrumentation you get. Your application has to be built in a supported language (Java, .NET, PHP, Python, C++, Node.js) and if it's web or mobile based you can also collect client side performance data and unify between both frontend and backend. If you have questions just reach out and I can answer them for you. Good luck!
You basically need the AppDynamics Controller and a AppDynamics Machine-Agent which will be installed on the machine to monitor. In the Machine-Agent configuration you set the URI of the controller and the agent starts to report machine metrics to the controller. Then you can configure alarms, see metrics, create dashboards, etc. I can deliver you more information if you want, but as Jonah Kowall said, take a look at the documentation as well AppDynamics Machine Agent Doc
Related
I currently am creating a faster test harness for our team and will be recording a baseline from our prod sdk run and our staging sdk run. I am running the tests via jest and want to eventually fire the parsed requests and their query params to a datastore of sorts and have a nice UI around it for tracking.
I thought that Prometheus and Grafana would be able to provide that, but after getting a little POC for myself working yesterday it seems that this combo is more used for tracking application performance rather than request log handling/manipulation/tracking.
Is this the right tool to be using for what I am trying to achieve and if so might someone shed some light on where I might find some more reading aligned with what I am trying to do?
Prometheus does only one thing and it well. It collects metrics and store them. It is used for monitoring your infrastructure or applications to monitor performance, availability, error rates etc. You can write rules using PromQL expression to create alert based on conditions and send them to alert manager which can send it to Pager duty, slack, email or any ticketing system. Even though Prometheus comes with a UI for visualising the data it's better to use Grafana since it's pretty good with it and easy to analyse data.
If you are looking tools for distributed tracing you can check Jaeger
i am using nodejs as my server with express. I am logging all my request and response on server. Is there any package available to read my logs and generate graphical report like how many requests we got and how many succeeded. What was the request received and responded. Is there a package which can track all these details for me?
It sounds like you're trying to get some performance metrics about your application which is great. There are many different ways you can go with this, here are a few suggestions for you to weigh up.
Non-real-time performance metrics
If you don't care about seeing the services real-time metrics you might want to create something to process them into a CSV and use something like excel or google sheets to generate graphs from them. If you need something immedietely and don't need to respond to things "in the moment" when a dip happens then this is a good quick and dirty solution.
Real-time performance metrics using SaaS software
If you want the metrics but don't want to host the systems yourself you might want to checkout services such as DataDog. They provide dashboards and graphs as a service. You can use something like statsd to get metrics into DataDog, or use their own integrations. They have a lot of integrations with cloud providers like AWS, GCP, and Azure for machine metrics (CPU etc). They also have packages for inteacting with your application itself such such as their ExpressJS package.
Real-time performance metrics using self-hosted solutions
I've often used a self-hosted approach as I find the pricing often scales a bit better. The setup is fairly simple.
Use a statsd package for all system components (nginx, nodejs, postgres, etc) to publish metrics to the statsd daemon.
The statsd daemon self-hosted somewhere (maybe a proxy cluster if you're working on large applications).
Self-hosted Graphite to consume metrics from the statsd daemon. Graphite is a software package designed for aggregating metrics and has an API for producing static graph images.
Self-hosted Grafana that pulls metrics from graphite. Grafana is a real-time dashboarding software. It allows you to create multiple dashboards that hook into various data sources such as Graphite or other time series data stores.
The self-hosting route can take a day to setup but it does mean you don't increase your costs per-host. It's also easy to put behind internal networks if that's a requirement for your organisation.
Personally, I would recommend either real-time performance metrics approaches. If your application is small and doesn't have many hosts then services like DataDog could be useful and cost effective but if you do need to scale up you'll find your costs sky rocketing. At that point you might decide to move over to a self-hosted infrastructure.
What are the differences in features between AppDynamics and Zipkin apart from the pricing since zipkin is opensource. Can any of them show request or response, in their console?
Zipkin only does tracing. APM tools like Appdynamics do other monitoring (browser, mobile, database, server, network). Code-level diagnostics with automated overhead controls and limiters. Don't forget log analytics and transaction analytics. It also collects metrics.
There is a lot more to APM than just tracing, which is what Zipkin does. You could do this with a stack of 20 open source tools, but you'd have to deal with disjointed UIs and data models not to mention the work associated with keeping them all working.
Context
I want to develop an automated script for broker (IIB9/10) resource monitoring, capturing information about broker running status, message flows deployed, jvm usage, number of threads running, etc.
The initial thought is to have a report generated using scripts and then displayed over a browser.
Question
Can this be entirely done using only Ant scripts (i am not sure as have not explored iterative processing in Ant in detail) or a combination of Ant and batch/shell scripts is the best bet?
I know Web user interface in IIB10 does most of it but i want to add some features.
I suggest you to take a look at message flow statistics and accounting:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/ac19100_.htm?lang=en
This is a feature of IIB by which it is capable of emitting resource statistics. The statistics are published to a topic in a well defined XML format. I would try solving your requirement by writing an application to read these messages and use the data in them to generate your graphs or other reports.
There is a support pack, IS03 which can give you an idea of such an application.
This will not cover everything you mentioned, for example monitoring what flows are deployed cannot be achieved like this, but it gives a comprehensive view of the load and performance of your applications:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bj10440_.htm?lang=en
And there is a resource statistics feature as well for monitoring resources used by your applications:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bj43310_.htm?lang=en
To get everything you will need a variety of tools I think. You can use Resource Stats and Accounting / Stats as suggested by Attila to get JVM and thread usage. The Broker publishes updates to a topic so you can create a simple subscriber to grab that info.
For deploy related info, stop / start state and so forth I would be looking at building simple Integration API or REST API applications to call from ant.
You can find documentation for these API's here:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/be43410_.htm?lang=en
and here:
http://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/SSMKHH_10.0.0/com.ibm.etools.mft.restapi.doc/index.html
I want to write one web application with node.js and MongoDB and I have got task to even test it. I would like to know if there are any tools like JMeter or anything else for load/stress testing of Node.js?
EDIT
My application is going to be information extraction kind of application and client expects extraction should not take more than 10 seconds for one document. Currently I have same application written in C# but its not scaling upto client's expectations. Then I came across this beautiful and fast Node.js. I think Node.js can help me alot.
Please enlighten !!!
Try nodeload: it's a collection of node.js modules for load testing HTTP services.
As a developer, you should be able to write load tests and get
informative reports without having to learn another framework. You
should be able to build by example and selectively use the parts of a
tool that fit your task. Being a library means that you can use as
much or as little of nodeload as makes sense, and you can create load
tests with the power of a full programming language. For example, if
you need to execute some function at a given rate, just use the
'nodeload/loop' module, and write the rest yourself
Just found out that this package is no longer under development so here are some active forks:
https://github.com/gamechanger/nodeload
https://github.com/Samuel29/NodeStressSuite
Why couldn't you test a node server with JMeter? For most load tests it doesn't matter what language your server is, you're just hitting it with a bunch of requests.
In any case, you could try loadtest which is implement in node.
Runs a load test on the selected HTTP or WebSockets URL. The API allows for easy integration in your own tests.
Edit:
This answer provides more options:
NodeJs stress testing tools/methods [closed]
Try artillery. Here are its features, the description of which is taken from the documentation:
Multiple protocols: Load test HTTP, WebSocket, Socket.io, Kinesis, HLS and more.
Scenarios: Support for complex scenarios to test multi-step interactions in your API or web app (great for ecommerce, transactional APIs, game servers etc).
Load testing & Functional testing: reuse the same scenario definitions to run performance tests or functional tests on your API or backend.
Performance metrics: get detailed performance metrics (latency, requests per second, concurrency, throughput).
Scriptable: write custom logic in JS, using any of the thousands of useful npm modules.
Integrations: statsd support out of the box for real-time reporting (integrate with Datadog, Librato, InfluxDB etc).
Extensible: write custom reporters, custom plugins, custom protocol engines etc.
and more! HTML reports, nice CLI, parameterization with CSV files.