Using bash on Linux.
I'm running find piped into xargs which launches a second bash script to do some processing on each file. I would like to maintain a "running tally" of file sizes and how many errors are encountered by the second script. In other words, every time the second script runs, it calculates the file size and adds it to the total so far, and also same thing if it encounters an error in its processing of the file. And I need this information to be available to the parent script after the find | xargs finishes.
I can do this by having the second script save and update a text file — a crude way to maintain a "global variable" — but I'm wondering if there's a nicer, more efficient way.
Can you use a pipe or Process Substitution to get the information back from the second script?
find ... | xargs second_script |
while read information
do
something useful with it
done
Or:
while read information
do
something useful with it
done < <(find ... | xargs second_script)
Related
I have read most questions regarding this topic, but can't get an answer to my specific question:
I have a number of files in a directory, and I want to apply a command to each of these files and then create a new file with the outpot for every single file. I can only manage to write it into one file alltogether. As i expect to have ~ 500.000 files, i also would need the script to be as efficient as possible.
for f in *.bed; do sort -k1,1 -k2,2n; done
This command sorts each file accordingly and writes the ouput in the Shell - But i cannot manage to write to file in the for-loop without appending it with ">>" .
I'm thankful for any answer providing an approach or an already answered question on this topic!
You can use script like this:
for f in *.bed
do
sort -k1,1 -k2,2n $f >>new_filename
done
If you want to be sure new_filename is empty before run the loop you can clear the content in file with command (before for loop):
>new_filename
I need to be able to read the second last line of all the files within a specific directory.
These files are log files and, that specific line contains the status of tasks that ran, 'successful', 'fail', 'warning'.
I need to pull this to dump it after in reports.
At this stage i am looking only to pull the data, so the entire line, and will worry about the handling after.
As the line numbers are not set, they are irregular, I am looking at doing it with a 'while' loop, so it goes through the whole thing, but i am actually not getting the last 2 lines read, and also, i can read 1 file not all of them.
Any ideas on a nice little script to do this?
And anyone knows if this can be just done with just a linux command?
Use the tail command to get the last 2 lines, and then the head command to get the first of these:
for file in $DIR/*; do
tail -2 "$file" | head -1
done
Edit: This question was originally bash specific. I'd still rather have a bash solution, but if there's a good way to do this in another shell then that would be useful to know as well!
Okay, top level description of the problem. I would like to be able to add a hook to bash such that, when a user enters, for example $cat foo | sort -n | less, this is intercepted and translated into wrapper 'cat foo | sort -n | less'. I've seen ways to run commands before and after each command (using DEBUG traps or PROMPT_COMMAND or similar), but nothing about how to intercept each command and allow it to be handled by another process. Is there a way to do this?
For an explanation of why I'd like to do this, in case people have other suggestions of ways to approach it:
Tools like script let you log everything you do in a terminal to a log (as, to an extent, does bash history). However, they don't do it very well - script mixes input with output into one big string and gets confused with applications such as vi which take over the screen, history only gives you the raw commands being typed in, and neither of them work well if you have commands being entered into multiple terminals at the same time. What I would like to do is capture much richer information - as an example, the command, the time it executed, the time it completed, the exit status, the first few lines of stdin and stdout. I'd also prefer to send this to a listening daemon somewhere which could happily multiplex multiple terminals. The easy way to do this is to pass the command to another program which can exec a shell to handle the command as a subprocess whilst getting handles to stdin, stdout, exit status etc. One could write a shell to do this, but you'd lose much of the functionality already in bash, which would be annoying.
The motivation for this comes from trying to make sense of exploratory data analysis like procedures after the fact. With richer information like this, it would be possible to generate decent reporting on what happened, squashing multiple invocations of one command into one where the first few gave non-zero exits, asking where files came from by searching for everything that touched the file, etc etc.
Run this bash script:
#!/bin/bash
while read -e line
do
wrapper "$line"
done
In its simplest form, wrapper could consist of eval "$LINE". You mentioned wanting to have timings, so maybe instead have time eval "$line". You wanted to capture exit status, so this should be followed by the line save=$?. And, you wanted to capture the first few lines of stdout, so some redirecting is in order. And so on.
MORE: Jo So suggests that handling for multiple-line bash commands be included. In its simplest form, if eval returns with "syntax error: unexpected end of file", then you want to prompt for another line of input before proceeding. Better yet, to check for proper bash commands, run bash -n <<<"$line" before you do the eval. If bash -n reports the end-of-line error, then prompt for more input to add to `$line'. And so on.
Binfmt_misc comes to mind. The Linux kernel has a capability to allow arbitrary executable file formats to be recognized and passed to user application.
You could use this capability to register your wrapper but instead of handling arbitrary executable, it should handle all executable.
Is it possible to trigger a command with every new line in to a file?
For example: I have a log file say maillog. I want to get every new entry in to the log file as a mail.
If a new entry like " Mail Sent " added in to maillog file then my script should grep the new entry and send me a mail with the entry(data).
I know its crazy but i want to automate my Linux box with these kind of things.
Regards,
Not so crazy. Check periodically (once per hour, per day, what you like) the file for new parts by storing the original length of the file, compare the length, in case it has grown, handle the part which was appended:
length=0
while sleep 3600 # use wanted delay here
do
new_length=$(find "$file" -printf "%s")
if [ $length -lt $new_length ]
then
tail --bytes=$[new_length-length] "$file" | handle_part
fi
length=$new_length
done
Now you only have to write that handle_part function which could for instance mail its input somewhere.
Using this way (instead of the obvious tail -f) has the advantage that you can store the current length into a file and later on restarting your script read that length again. So you won't get the whole file after a restart of your script (e. g. due to a machine reboot).
If you want a faster response you could have a look at inotify which is a facility on Linux to monitor file actions; so that polling could be replaced.
Use tail -f, that watches a file and sents whatever is appended to it to stdout. If you have a script that performs the desired action, say mail_per_line, then you can set it up as
tail -f maillog | mail_per_line
In this case, mail_per_line runs once and gets all the lines. If you want to spawn a separate process each time a line comes in, use the shell built-in read:
tail -f maillog | while IFS='' read line; do
send_a_message "$line"
done
To counter the effect described by Alfe, that a restart of this program will cause all the previous logs to be processed again, consider using logrotate.
Mostly, I will not use | less for each and every command from the shell.
Pipe to less is used only when I actually run the command without is and find out that it does not fit on the page. That costs me two runs of the same shell command.
Is there a way so that every time a command result is more than a display page, it automatically gets piped to less?
Pipe it to less -F aka --quit-if-one-screen:
Causes less to automatically exit if the entire file can be dis-
played on the first screen.
The most significant problem with trying to do that is how to get it to turn off when running programs that need a tty.
What I would recommend is that, for programs and utilities you frequently use, create shell functions that wrap them and pipe to less -F. In some cases, you can name the function the same as the program and it will take precedence, but can be overridden.
Here is an example wrapper function which would need testing and perhaps some additional code to handle edge cases, etc.
#!/bin/bash
foo () {
if [[ -p /dev/stdout ]] # you don't want to pipe to less if you're piping to something else
then
command foo "$#" | less -F
else
command foo "$#"
fi
}
If you use the same name as I have in the example, it could break things that expect different behavior. To override the function to run the underlying program directly precede it with command:
command foo
will run foo without using the function of the same name.
You could always pipe to less -E (this will cause less to automatically quit at the end of the file). For commands with short output it would do what you want. I don't think you can automatically pipe to less when there is a lot of output.
In general, automatically piping to less requires the shell to be prescient about the output that will be produced by the commands it runs - and it is hard enough for humans to predict that without trying to make programs do so.
You could write a shell that does it for you - that captures the output (but what about stderr?) and paginates if necessary, but it would most certainly not be a standard shell.
I wrote this wrapper function and put it in my .profile. You can use this before a command and it will automatically pipe it to less if it is longer than 1 page.
lcmd ()
{
echo "$("$#")" | less -F;
};
So 'lcmd ls' would ls the current directory and pipe that output to less.