Good afternoon everyone,
I have created a resource monitoring tool that works fairly well.
Pulls CPU average usage
Pulls average Memory usage
Even calculates % of NIC throughput (if you have a 1Gb NIC, it'll show percent that is being processed at a time)... and yes I know this is more of a rough estimate/theoretical max limit of a NIC.
I am having one issue with my script though. Portion of the code I am experiencing my issue with is below (I converted some to pseudo code for simplicity).
COUNT=1
read -rsp "When you are ready to begin, please press any key" -n1
echo "processing"
sleep 3
while [ ${COUNT} = "1" ; do
read -t 1 -n 1
if [$? = 0 ] ; then
exit 0
else
`Resource command` > ${cpulog} file for future graphs
`Resource command` > ${memlog} file for future graphs
`Resource command` > ${network} log file for future graphs
`etc`
fi
done
Basically, you hit any key to start the program, and whenever you press any key on the keyboard after the program has started (While loop), the program stops recording information and moves on.
Now this script works and does everything I need it to do. The issue I have come across is when you "press any key".
Note that there are two points in the script waiting for a key press.
If I were to press any key more than once at the first point, the second key input would get processed by my read -t 1 -n 1 command (at the second point), and thus fail to run my resource pulls. Since that happens immediately, the script fails.
Basically, I am trying to figure out if there is a way I can shutdown input after that first key stroke for a limited time while I retrieve a limited amount of data, or flush any input that was given prior to hitting my read -t 1 command. Thank you.
The script can call a small program, written in C, perl or similar, which calls the FIONREAD ioctl on stdin.
Then read the unexpected extra characters to be thrown away with a read call, see Perl Cookbook to determine unread bytes
You can actually enter the perl code on command line with perl -e. to keep it all within the bash script.
At the "processing line," add the following loop:
# Eat any remaining input
while read -t 1 -n 1
do
# Do nothing here
:
done
# Continue processing now that all input has been consumed...
It will add about 1 second delay to startup (more if the user is sitting there pressing keys), but otherwise does what you want.
Related
I'm working on a system on which ubuntu is running. I'm reading basic data like CPU frequency and temperature out of the thermal zones provided in /sys/class/thermal.
Unfortunately, I've got around 100 thermal_zones from which I need to read the data. I do it with:
for SENSOR_NODE in /sys/class/thermal/thermal_zone*; do printf "%s: %s\n" $(cat ${SENSOR_NODE}/type) $(cat ${SENSOR_NODE}/temp); done
To collect all data takes ~2.5-3 sec. which is way to long.
Since I want to collect the data every second my question is, if there is a way to "read" or "collect" the data faster?
Thank you in advance
There's only so much you can do while writing your code in shell, but let's start with the basics.
Command substitutions, $(...), are expensive: They require creating a FIFO, fork()ing a new subprocess, connecting the FIFO to that subprocess's stdout, reading from the FIFO and waiting for the commands running in that subshell to exit.
External commands, like cat, are expensive: They require linking and loading a separate executable; and when you run them without exec (in which case they inherit and consume the shell's process ID), they also require a new process to be fork()ed off.
All POSIX-compliant shells give you a read command:
for sensor_node in /sys/class/thermal/thermal_zone*; do
read -r sensor_type <"$sensor_node/type" || continue
read -r sensor_temp <"$sensor_node/temp" || continue
printf '%s: %s\n' "$sensor_type" "$sensor_temp"
done
...which lets you avoid the command substitution overhead and the overhead of cat. However, read reads content only one byte at a time; so while you're not paying that overhead, it's still relatively slow.
If you switch from /bin/sh to bash, you get a faster alternative:
for sensor_node in /sys/class/thermal/thermal_zone*; do
printf '%s: %s\n' "$(<"$sensor_node/type)" "$(<"$sensor_node/temp")"
done
...as $(<file) doesn't need to do the one-byte-at-a-time reads that read does. That's only faster for being bash, though; it doesn't mean it's actually fast. There's a reason modern production monitoring systems are typically written in Go or with a JavaScript runtime like Node.
I want to (hopefully easily) write from a bash script to any arbitrary program that is already running via that program's stdin.
Say I have some arbitrary program "Sum", that constantly takes user input from the terminal. Every integer it receives from stdin it adds to the current sum and outputs the new sum. Here's example terminal text of what I mean:
$: ./Sum
Sum: 0
Give me an integer: 2
Sum: 2
Give me an integer: 5
Sum: 7
How would automate this process in a bash script? If I had control of Sum's source code I could let it accept integer arguments. But if I don't have said control of the program, how can I automate the interaction with Sum? Here's a psuedo code of a bash snippet of what I want to do:
In example.sh:
#!/bin/bash
my_program_input_point=./Sum &
echo 2 > my_program_input_point
echo 5 > my_program_input_point
Thus on my terminal screen it would still look like this:
$: ./example.sh
Sum: 0
Give me an integer: 2
Sum: 2
Give me an integer: 5
Sum: 7
The difference is I wouldn't have typed any of it.
It feels like this task should be really easy, because basically anything you can do in a terminal, you can also easily do in a script. Except, apparently, directly interact with an arbitrary process once its started.
This topic answers some aspects of my question by using pipes. However the accepted answer only works if I have control of both programs on each side of the pipe, the "recipient" program is under no obligation to read from the pipe automatically.
This is assuming I can't modify the program/script in question. I also can't write a "pipe reader" script to wrap around the program (Sum), because that wrapper script would still be required to interact with the running process Sum.
The second answer to this question on serverfault.com (written by jfgagne) seems much closer, but I can't seem to get it working. Is there any easier way to do this that I just don't know about?
For information on how to capture and read an arbitrary program's output, see my next question for more information
The most straightforward way is to use a pipeline. The trick is that you can make the lefthand side of the pipeline a compound statement, or a function call. It doesn't have to be just a single command.
{
echo 2
echo 5
} | ./Sum
or
numbers() {
echo 2
echo 5
}
numbers | ./Sum
This lets you do whatever you want to generate the input. You don't have to have it all ahead of time. If you generate the input bit by bit, it'll be fed to ./Sum bit by bit.
You can use a named pipe. Whatever starts Sum is responsible for creating the named pipe some place convenient, then starting Sum with the named pipe as its standard input.
mkfifo /tmp/pipe
Sum < /tmp/pipe
Then your script could take the name of the pipe as an argument, then treat it as a file it can write to.
#!/bin/bash
p=$1
echo 2 > "$p"
echo 5 > "$p"
Then you could call your script with client /tmp/pipe.
You can write to the standard input file descriptor of the running process. Here is the same question: https://serverfault.com/questions/178457/can-i-send-some-text-to-the-stdin-of-an-active-process-running-in-a-screen-sessi
You need to write to /proc/*pid of the program*/fd/0 which is the file descriptor for the standard input of the process with that pid. Make sure you have access to do this.
I am planning to write a linux command which is going to process (with a custom logic) a very big file. Once this command is run, it will take hours to finish the task.
What is the best way to give eta output?
What about the idea of writing a 100 byte progress status file? In specific stages I can write bytes to the file. Say at 40% I can write 40 bytes to the file. Once the file size reaches 100 bytes size it means the process is finished. So to monitor the progress we have to check only the size of this file.
Is there any general way of handling such progress information? I don't want to include too much logic in my program for displaying the progress information. I am looking for a simple solution.
Any suggestions?
The normal way would be to set up a signal handler like dd does, and then you send the process a SIGUSR1 or somesuch using the kill command and it catches the signal and outputs a status message. Try
man dd
So, on my iMac, for example, dd uses SIGINFO, so
# Move a whole load of nothing to nowhere in the background
dd if=/dev/zero of=/dev/null &
[1] 11232
# No output, till I want some a few seconds later
kill -SIGINFO 11232
12875835+0 records in
12875834+0 records out
6592427520 bytes transferred in 9.380158 secs (702805581 bytes/sec)
# No more output, till I want some a few more seconds later
kill -SIGINFO 11232
19163866+0 records in
19163865+0 records out
9811898880 bytes transferred in 14.015095 secs (700095068 bytes/sec)
A signal handler is pretty easy to set up, even in a shell script. For example:
#!/bin/bash
trap "echo Progress report..." SIGINT
echo "My process id is $$"
while : ; do
sleep 10 # Do something - admittedly little :-)
done
I'm afraid you can't do this nicely in Linux but if the result of your program can be an output stream like in unzip then you just can use pv - Pipe Viewer command. You can redirect the output into pv input and it will show you the progress.
In Windows you can use Taskbar Extensions API with methods SetProgressState and SetProgressValue.
I have a function in a bash script that executes a long process called runBatch. Basically runBatch takes a file as an argument and loads the contents into a db. (runBatch is just a wrapper function for a database command that loads the content of the file)
My function has a loop that looks something like the below, where I am currently recording start time and elapsed time for the process to variables.
for batchFile in `ls $batchFilesDir`
do
echo "Batch file is $batchFile"
START_TIME=$(($(date +%s%N)/1000000))
runBatch $batchFile
ELAPSED_TIME=$(($(($(date +%s%N)/1000000))-START_TIME))
IN_SECONDS=$(awk "BEGIN {printf \"%.2f\",${ELAPSED_TIME}/1000}")
done
Then I am writing some information on each batch (such as time, etc.) to a table in a html page I am generating.
How would I go about recording the highest memory/cpu usage while the runBatch is running, along with the time, etc?
Any help appreciated.
Edit: I managed to get this done. I added a wrapper script around this script that runs this script in the background. I pass it's PID with $! to another script in the wrapper script that monitors the processes CPU and Memory usage with top every second. I compile everything into a html page at the end when the PID is no longer alive. Cheers for the pointers.
You should be able to get the PID of the process using $!,
runBatch $batchFile &
myPID=$!
and then you can run a top -b -p $myPID to print out a ticking summary of CPU.
Memory:
cat /proc/meminfo
Next grep whatever you want,
Cpu, it is more complicated - /proc/stat expained
Average load:
cat /proc/loadavg
For timing "runBatch" use
time runBatch
like
time sleep 10
Once you've got the pid of your process (e.g. like answered here) you can use (with watch(1) & cat(1) or grep(1)) the proc(5) file system, e.g.
watch cat /proc/$myPID/stat
(or use /proc/$myPID/status or /proc/$myPID/statm, or /proc/$myPID/maps for the address space, etc...)
BTW, to run batch jobs you should consider batch (and you might look into crontab(5) to run things periodically)
I have a script that produces a lot of output. The script pauses for a few seconds at point T.
Now I am using the less command to analyze the output of the script.
So I execute ./script | less. I leave it running for sufficient time so that the script would have finished executing.
Now I go through the output of the less command by pressing Pg Down key. Surprisingly while scrolling at the point T of the output I notice the pause of few seconds again.
The script does not expect any input and would have definitely completed by the time I start analyzing the output of less.
Can someone explain how the pause of few seconds is noticable in the output of less when the script would have finished executing?
Your script is communicating with less via a pipe. Pipe is an in-memory stream of bytes that connects two endpoints: your script and the less program, the former writing output to it, the latter reading from it.
As pipes are in-memory, it would be not pleasant if they grew arbitrarily large. So, by default, there's a limit of data that can be inside the pipe (written, but not yet read) at any given moment. By default it's 64k on Linux. If the pipe is full, and your script tries to write to it, the write blocks. So your script isn't actually working, it stopped at some point when doing a write() call.
How to overcome this? Adjusting defaults is a bad option; what is used instead is allocating a buffer in the reader, so that it reads into the buffer, freeing the pipe and thus letting the writing program work, but shows to you (or handles) only a part of the output. less has such a buffer, and, by default, expands it automatically, However, it doesn't fill it in the background, it only fills it as you read the input.
So what would solve your problem is reading the file until the end (like you would normally press G), and then going back to the beginning (like you would normally press g). The thing is that you may specify these commands via command line like this:
./script | less +Gg
You should note, however, that you will have to wait until the whole script's output loads into memory, so you won't be able to view it at once. less is insufficiently sophisticated for that. But if that's what you really need (browsing the beginning of the output while the ./script is still computing its end), you might want to use a temporary file:
./script >x & less x ; rm x
The pipe is full at the OS level, so script blocks until less consumes some of it.
Flow control. Your script is effectively being paused while less is paging.
If you want to make sure that your command completes before you use less interactively, invoke less as less +G and it will read to the end of the input, you can then return to the start by typing 1G into less.
For some background information there's also a nice article by Alexander Sandler called "How less processes its input"!
http://www.alexonlinux.com/how-less-processes-its-input
Can I externally enforce line buffering on the script?
Is there an off the shelf pseudo tty utility I could use?
You may try to use the script command to turn on line-buffering output mode.
script -q /dev/null ./script | less # FreeBSD, Mac OS X
script -c "./script" /dev/null | less # Linux
For more alternatives in this respect please see: Turn off buffering in pipe.