I would like to create a time series and inject it into InfluxDb for a demo. I thought about using the top command (top -pid 1393 -stats cpu), and use the CPU value. And then use NodeJS to extract the data and inject it into an InfluxDB. However, there are a couple of but...:
1- The top command has a display section: can it be removed?
2- In Node, I would call (repeatedly) "top -pid 1393 -stats cpu -l 1" with the "-l 1" option to only get a singe sample. I feel it is a misuse of the fact that top generates data at given intervals (basically, I recreate in Node what top does automatically)
Is there a better way to do this - in the ideal world, I would launch top in node and "pipe" the output stream to a variable in an async way (to execute the insertion into InfluxDB).
Thanks for any hints you may have.
Christian
In actual fact, there is a node module for this: see npm usage.
Note that you may have an issue when installing the usage module, generated by node-gyp rebuild (usage requires this module, which requires XCode in Mac platform - see https://github.com/nodejs/node-gyp). To solve this, look here: xcode-select active developer directory error)
Thanks - C
To monitor the process' resource consuming metrics (correct me if I took your intentions wrong), you don't need your NodeJS stuff at all.
All you need is running Telegraf agent, with this plugin configured, on the machine(s) you're targeting.
Point the output plugin to your Influx - and that's it.
As mention in other comments, there are specific tools for that task. Anyway, if you still want to do it programmatically, I would suggest to either:
run top in non-interactive mode, as explained here: https://unix.stackexchange.com/questions/255100/get-top-output-for-non-interactive-shell
read directly the information that you need from /proc/<pid of your process>/<file of interest>
Related
I want to understand how up/down arrow keys help to navigate through previously executed commands. which memory it access to get last executed command?
Currently i am working on unix similar Machine "Stratus VOS" where i can see only last 1 executed command and that too with F10 key.
I want to create a script/tool which can help to me to get at least last 10-15 executed commands to save my time during development work.
what i noticed so far on Stratus Machine:
1: fc command doesn't work.
2. no HISTFILE is maintained. no history command works.
what i thought to implement:
find which memory in system holds last executed command, access that memory(whenever Enter key is pressed ) and keep writing in file.
am i on right track? is there any better solution for this?
I would suggest that while possible your solution is possibly overcomplex and likely to be difficult and possibly unreliable. I would instead consider using the Stratus VOS GNU tools package which includes a full GNU Bash shell with all the features you expect including the command history support.
A little documentation on GNU tools for Stratus VOS appears to be available here though I would suggest contacting your vendor for more information or support.
I am looking for a quick tool to batch monitoring network traffic per socket and/or per process.
I.e., I would like to iterate over a given time and get the iterated traffic as text output/on stdout.
I checked several tools so far as iftop, nethogs, iptraf-ng, ifstat, tcptrack -- which offer either nice statistics of the info I look for or a batch mode, but I did not find a way to combine it.
Ideally, it would be something like iftop or nethogs (or iptraf) just with a batch option
ala
iftop -i eth# -iterateinsec 60 > nettraf.txt
Is there a way to do so (maybe with the tools I tried and missed its batch feature) or some other ready available tool?
Cheers and thanks,
Thomas
I would suggest you to use Sealion which is very simple use. There are default set of commands which will give you all statistics in a given timeline. Also you can add parameters to it.
I am new to casperjs and planning to use it to accurately simulate anywhere from a few dozen to low hundreds of concurrent sessions accessing a private server on a private network. Unlike typical HTTP load generators (Apache bench, httperf, ...), my purpose is to be able to control each session programmatically (increase delay between requests, have 'smarts' built into each script) and have each session have distinct source IP addresses.
My current thinking is to use OpenVZ containers (openvz.org) to create each 'virtual' client running casperjs (minimal functionality I need is following elements on the UI and taking screenshots). Would love to hear of anyone who has done something similar.
The crux of my question is: what would the 'slimmest' environment for running casperjs be? I'd like to strip down the OS as much as possible to be able to scale multiple clients. Specifically:
any recommended low-footprint UNIX/Linux distributions for CasperJS?
any specific recommendations on stripping down mainstream (CentOS, Debian, ...) distributions?
Thank you all in advance. I look forward to hearing your input on this specific question or similar experiences/tools for what I'm trying to achieve...
Fernando
CasperJS is headless, e.g. it doesn't need X running to function. Any bare bones Linux distribution will do you well.
any recommended low-footprint UNIX/Linux distributions for CasperJS?
Arch is very lightweight and has an easy to follow Beginners Guide. Arch's AUR has a package for CasperJS that's pretty straightforward to setup as well. Just make sure to grab the required base-devel package (pacman -S base-devel) before installing from the AUR as it's needed for the Arch Build System.
any specific recommendations on stripping down mainstream (CentOS, Debian, ...) distributions?
Not so much stripping down, but CrunchBang is based off of the latest Debian release. It may be worth taking a look at. It would be much less of a hassle to setup than Arch, and uses the same APT package manager as Debian / Ubuntu. It installs with the lightweight OpenBox window manager, but you can remove this and X all together if you'd like.
With that said, even a lightweight Linux environment won't help much with the amount of memory each CasperJS instance will use. You could probably pull off a few dozen depending on the amount of memory available, but a few hundred may not be feasible. It all depends on how much memory each website uses. Casperjs comes with some configuration options that may help reduce memory (e.g. don't load images, plugins, etc), but that may defeat the purpose of your tests.
The best advice I can give is to try it out for yourself. Write a simple script that will open the pages you are going to use and pass a callback to CasperJS's run() function to keep it alive (e.g. don't exit from Casper). It can be as simple as:
casper.start('http://example.com/site1', function () {});
casper.thenOpen('http://example.com/site2', function () {});
casper.run(function() {
// wait 60 seconds before exit . . . or remove to never exit
setTimeout(function() { casper.exit(); }, 60000);
}
Spin up multiple instances, and watch your total memory usage. You can use the cli tools top, or use this alias that totals the amount of memory usage for the current user.
alias memu="ps -u $(whoami) -o pid,rss,command | awk '{print \$0}{sum+=\$2} END {print \"Total\", sum/1024, \"MB\"}'"
From this you should be able to see roughly how much memory each instance takes, and how many you can run at once on one machine.
So I was looking into why a program was getting rid of my background, and the author of the program said to post .xsession-errors and many people did. Then my next question was: What is .xsession-errors? A google search reveals many results but nothing explaining what it is.
What I know so far:
It's some kind of error log. I can't figure out what it's related too (ubuntu itself? programs?)
I have one and it seems like all Ubuntu systems have it, though I cannot verify.
Linux graphical interfaces (such as GNOME) provide a way to run applications by clicking on icons instead of running them manually on the command-line. However, by doing so, output from the command-line is lost - especially the error output (STDERR).
To deal with this, some display managers (such as GDM) pipe the error output to ~/.xsession-errors, which can then be used for debugging purposes. Note that since all applications launched this way dump to the same log, it can get quite large and difficult to find specific messages.
Update: Per the documentation:
The ~/.xsession-errors X session log file has been deprecated and is
no longer used.
It has been replaced by the systemd journal (journalctl command).
It's the error log produced by your X windows system (which the Ubuntu GUI is built on top of).
Basically it's quite a low level error log for X11.
I'm trying to find the best way to use 'top' as semi-permanent instrumentation in the development of a box running embedded Linux. (The instrumentation will be removed from the final-test and production releases.)
My first pass is to simply add this to init.d:
top -b -d 15 >/tmp/toploop.out &
This runs top in "batch" mode every 15 seconds. Let's assume that /tmp has plenty of space…
Questions:
Is 15 seconds a good value to choose for general-purpose monitoring?
Other than disk space, how seriously is this perturbing the state of the system?
What other (perhaps better) tools could be used like this?
Look at collectd. It's a very light weight system monitoring framework coded for performance.
We use sysstat to monitor things like this.
You might find that vmstat and iostat with a delay and no repeat counter is a better option.
I suspect 15 seconds would be more than adequate unless you actually want to watch what's happening in real time, but that doesn't appear to be the case here.
As far as load, on an idling PIII 900Mhz w/ 768MB of RAM running Ubuntu (not sure which version, but not more than a year old) I have top updating every 0.5 seconds and it's about 2% CPU utilization. At 15s updates, I'm seeing 0.1% CPU utilization.
depending upon what exactly you want, you could use the output of uptime, free, and ps to get most, if not all, of top's information.
If you are looking for overall load, uptime is probably sufficient. However, if you want specific information about processes, you are adventurous, and have the /proc filessystem enabled, you may want to write your own tools. The primary benefit in this environment is that you can focus on exactly what you want and minimize the load introduced to the system.
The proc file system gives your application read access to the kernel memory that keeps track of many of the interesting variables. Reading from /proc is one of the lightest ways to get this information. Additionally, you may be able to get more information than provided by top. I've done this in the past to get amount of time spent in user and system by this process. Additionally, you can use this to get information about the number of file descriptors open by the process. You might also use this to get detailed information about how the network system is working.
Much of this information is pre-processed by other applications which can be used if you get the information you need. However, it is rather straight-forward to read the raw information. Do a man proc for more information.
Pity you haven't said what you are monitoring for.
You should decide whether 15 seconds is ok or not. Feel free to drop it way lower if you wish (and have a fast HDD)
No worries unless you are running a soft real-time system
Have a look at tools suggested in other answers. I'll add another sugestion: "iotop", for answering a "who is thrashing the HDD" questions.
At work for system monitoring during stress tests we use a tool called nmon.
What I love about nmon is it has the ability to export to XLS and generate beautiful graphs for you.
It generates statistics for:
Memory Usage
CPU Usage
Network Usage
Disk I/O
Good luck :)