Context
I want to develop an automated script for broker (IIB9/10) resource monitoring, capturing information about broker running status, message flows deployed, jvm usage, number of threads running, etc.
The initial thought is to have a report generated using scripts and then displayed over a browser.
Question
Can this be entirely done using only Ant scripts (i am not sure as have not explored iterative processing in Ant in detail) or a combination of Ant and batch/shell scripts is the best bet?
I know Web user interface in IIB10 does most of it but i want to add some features.
I suggest you to take a look at message flow statistics and accounting:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/ac19100_.htm?lang=en
This is a feature of IIB by which it is capable of emitting resource statistics. The statistics are published to a topic in a well defined XML format. I would try solving your requirement by writing an application to read these messages and use the data in them to generate your graphs or other reports.
There is a support pack, IS03 which can give you an idea of such an application.
This will not cover everything you mentioned, for example monitoring what flows are deployed cannot be achieved like this, but it gives a comprehensive view of the load and performance of your applications:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bj10440_.htm?lang=en
And there is a resource statistics feature as well for monitoring resources used by your applications:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bj43310_.htm?lang=en
To get everything you will need a variety of tools I think. You can use Resource Stats and Accounting / Stats as suggested by Attila to get JVM and thread usage. The Broker publishes updates to a topic so you can create a simple subscriber to grab that info.
For deploy related info, stop / start state and so forth I would be looking at building simple Integration API or REST API applications to call from ant.
You can find documentation for these API's here:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/be43410_.htm?lang=en
and here:
http://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/SSMKHH_10.0.0/com.ibm.etools.mft.restapi.doc/index.html
Related
I currently am creating a faster test harness for our team and will be recording a baseline from our prod sdk run and our staging sdk run. I am running the tests via jest and want to eventually fire the parsed requests and their query params to a datastore of sorts and have a nice UI around it for tracking.
I thought that Prometheus and Grafana would be able to provide that, but after getting a little POC for myself working yesterday it seems that this combo is more used for tracking application performance rather than request log handling/manipulation/tracking.
Is this the right tool to be using for what I am trying to achieve and if so might someone shed some light on where I might find some more reading aligned with what I am trying to do?
Prometheus does only one thing and it well. It collects metrics and store them. It is used for monitoring your infrastructure or applications to monitor performance, availability, error rates etc. You can write rules using PromQL expression to create alert based on conditions and send them to alert manager which can send it to Pager duty, slack, email or any ticketing system. Even though Prometheus comes with a UI for visualising the data it's better to use Grafana since it's pretty good with it and easy to analyse data.
If you are looking tools for distributed tracing you can check Jaeger
What is the best practice for logging mechanisam in ETL processing?
Actually we are developing ETL application .in this we want to use log analaytics to log data
Could anybody provide best practice for logging mechanism at industry standards.
i have googled below link :https://www.timmitchell.net/post/2016/03/14/etl-logging/
any help is appreciated.
Thanks in advance
I recently implemented one in one of the organisation. It is a custom built because of the technology choice. Following is what is included in the logging.
It acts as a wrapper around any ETL job aka there is a template developed and the template has in built logging
The template has a feature of master and child job and logs based on master or child
The logging captures the following:
Status of the job - success, failure, warning
Source details (e.g name of file or source table etc name)
data classification tagging
business owner of the incoming data source
row count of raw file vs the row count loaded
Send an alert to a distribution list if the job fails
Raises a ticket via service desk if job fails
It depends on your requirements, you may want to capture more or less.
Good luck
How do I test the behavior of a java restful web service in case of multiple concurrent requests? Is there any 3rd party tool that can be leveraged?
The service accepts POST method. It expects a couple of parameters in it's request body and produces the response in the form of JSON.
The functionality of the service is to perform database read operations using the request body parameters and populate the fetched data in the JSON.
I would recommend one of the following:
SoapUI - superior tool for web service testing. Has limited load testing capabilities. However it does not scale (no clustered mode is available) and has quite poor reporting (all you get is average, min and max response times)
Apache JMeter - multiprotocol load testing tool, supports web services load testing as well. Has better load capabilities and ways to define the load patterns and can represent load test results via HTML Reporting Dashboard. Check out Testing SOAP/REST Web Services Using JMeter article to learn how to conduct a web service load test using JMeter.
You can try Gatling to generate some load.
It has nice documentation and easy QuickStart .
For advanced usage it requires some knowledge of Scala, but it also features GUI tool for simple scenarios recording, so you can run some scripts by postman or whatever browser tool you use for debugging, record it and make that scenario automated.
After running scenarios it generates nice reports using Graphite, so you can see response times and general stats.
Later you can also use Gatling for load and performance tests of your web service, it's convenient and fast as soon as you start playing with it. It can easily generate up to 5k requests per second from my old Mac, or hold up to 1k connections.
One of the bests tools to test web services is SOAPUI.
You can use it for what you want.
Link to SOAPUI
You can check this link to see how to use SOAPUI and concurrent tests.
I have an application that is generating 3 kind of log files
Transaction log
Server log
Fatal log
and I want to analyse the performance of my server using appdynamics so what kind of data my logs should be generating to generate analytics for server health, performance, throughput, server utilization?
That's the beauty of APM is you don't need to deal with logging to get performance data. APM tools instrument the applications themselves regardless of what the code does (logging, generating metrics, etc). AppDynamics can collect log data, and provide similar capabilities to what a log analytics tool can do, but it's of less value than the transaction tracing and instrumentation you get. Your application has to be built in a supported language (Java, .NET, PHP, Python, C++, Node.js) and if it's web or mobile based you can also collect client side performance data and unify between both frontend and backend. If you have questions just reach out and I can answer them for you. Good luck!
You basically need the AppDynamics Controller and a AppDynamics Machine-Agent which will be installed on the machine to monitor. In the Machine-Agent configuration you set the URI of the controller and the agent starts to report machine metrics to the controller. Then you can configure alarms, see metrics, create dashboards, etc. I can deliver you more information if you want, but as Jonah Kowall said, take a look at the documentation as well AppDynamics Machine Agent Doc
I'm new to RabbitMQ and MQ's in general. I'm using the rabbit.js Node.js module to interface with RabbitMQ, so all my application layer is going to mainly be in Node.js. What I'm wondering is, how do I manage RabbitMQ? How can I see everything that's going on with RabbitMQ, from what's messages are left in the queue to general configuration and administration?
I'm looking for something visual, but more importantly, easy to use and simple.
RabbitMQ has a web interface (part of the rabbitmq_management plugin which ships with RabbitMQ, but needs to be enabled) that allows you to see the servers, exchanges, queues, etc.
It's pretty easy to use. One thing I would recommend is to set the time-interval on the graphs to 10 minutes. I find if you set them to longer, say, an hour plus, the information (due to the way it's bucketed, I think) gets a bit wonky.
Check out this link for more info: https://www.rabbitmq.com/management.html
There is also a JSON API that can be used to programmatically determine, for example, how many items are in a particular queue.
There's also a cmdline tool, called rabbitmqadmin (https://www.rabbitmq.com/management-cli.html) which can come in really handy for things like setting up test RabbitMQ test environments via a bash script and things of that nature.
check JXM.io sources (open source messaging backend for node.js / JXcore) that uses RabbitMQ for multi server integration and there is a nice article showing how to cluster RabbitMQ http://jxm.io/multi-server-messaging-backend-installation/