I am trying to create a sh that generates a report that will display differences against files from two folders/java projects. I am using dwdiff, and I need only the last line from each comparison (I don't care about differences, the percentage is what I need). I've created the following script:
DEST_FOLDER='./target/'
CLASSES_P1='classesP1.txt'
CLASSES_P2='classesP2.txt'
DEST_FILE='report.txt'
rm -r "$DEST_FOLDER"
mkdir -p "$DEST_FOLDER"
rm -f "$DEST_FOLDER/$CLASSES_P1"
rm -f "$DEST_FOLDER/$CLASSES_P2"
rm -f "$DEST_FOLDER/$DEST_FILE"
find ./p1 -name "*.java" >> "$DEST_FOLDER/$CLASSES_P1"
find ./p2 -name "*.java" >> "$DEST_FOLDER/$CLASSES_P2"
while read p; do
while read t; do
dwdiff -s $p $t | tail --lines=0 >> "$DEST_FOLDER/$DEST_FILE"
done < $DEST_FOLDER/$CLASSES_P2
done < $DEST_FOLDER/$CLASSES_P1
It works fine, but the results are not redirected to the given file. The file is created, but is empty, and the last line from each dwdiff result is displayed to console. Any ideas?
There are a few things going on
The output you want is going to stderr, not stdout. You can merge them with 2>&1 on the dwdiff command
The output you want actually appears to be printed first by dwdiff, but you see a different order due to the two different outputs. So you want head instead of tail
You want 1 line, not 0
So try dwdiff -s ... 2>&1 | head --lines=1
$ dwdiff -s /etc/motd /etc/issue 2>&1 | head --lines=1
old: 67 words 3 4% common 2 2% deleted 62 92% changed
Alternatively, if you want the new line instead of the old, and to simplify the ordering, try throwing away the diff output:
$ dwdiff -s /etc/motd /etc/issue 2>&1 1>/dev/null | tail --lines=1
new: 5 words 3 60% common 0 0% inserted 2 40% changed
Note that the order of redirection is important: First clone stdout into stderr, then redirect stdout to /dev/null.
Related
This question already has answers here:
Piping tail output though grep twice
(2 answers)
Closed 4 years ago.
There is a file located at $filepath, which grows gradually. I want to print every line that starts with an exclamation mark:
while read -r line; do
if [ -n "$(grep ^! <<< "$line")" ]; then
echo "$line"
fi
done < <(tail -F -n +1 "$filepath")
Then, I rearranged the code by moving the comparison expression into the process substitution to make the code more concise:
while read -r line; do
echo "$line"
done < <(tail -F -n +1 "$filepath" | grep '^!')
Sadly, it doesn't work as expected; nothing is printed to the terminal (stdout).
I prefer to write grep ^\! after tail. Why doesn't the second code snippet work? Why putting the command pipe into the process substitution make things different?
PS1. This is how I manually produce the gradually growing file by randomly executing one of the following commands:
echo ' something' >> "$filepath"
echo '!something' >> "$filepath"
PS2. Test under GNU bash, version 4.3.48(1)-release and tail (GNU coreutils) 8.25.
grep is not line-buffered when its stdout isn't connected to a tty. So it's trying to process a block (usually 4 KiB or 8 KiB or so) before generating some output.
You need to tell grep to buffer its output by line. If you're using GNU grep, this works:
done < <(tail -F -n +1 "$filepath" | grep '^!' --line-buffered)
^^^^^^^^^^^^^^^
I'm building a little bash script to run another bash script that's found in multiple directories. Here's the code:
cd /home/mainuser/CaseStudies/
grep -R -o --include="Auto.sh" [\w] | wc -l
When I execute just that part, it finds the same file 5 times in each folder. So instead of getting 49 results, I get 245. I've written a recursive bash script before and I used it as a template for this problem:
grep -R -o --include=*.class [\w] | wc -l
This code has always worked perfectly, without any duplication. I've tried running the first code with and without the " ", I've tried -r as well. I've read through the bash documentation and I can't seem to find a way to prevent, or even why I'm getting, this duplication. Any thoughts on how to get around this?
As a separate, but related question, if I could launch Auto.sh inside of each directory so that the output of Auto.sh was dumped into that directory; without having to place Auto.sh in each folder. That would probably be much more efficient that what I'm currently doing and it would also probably fix my current duplication problem.
This is the code for Auto.sh:
#!/bin/bash
index=1
cd /home/mainuser/CaseStudies/
grep -R -o --include=*.class [\w] | wc -l
grep -R -o --include=*.class [\w] |awk '{print $3}' > out.txt
while read LINE; do
echo 'Path '$LINE > 'Outputs/ClassOut'$index'.txt'
javap -c $LINE >> 'Outputs/ClassOut'$index'.txt'
index=$((index+1))
done <out.txt
Preferably I would like to make it dump only the javap outputs for the application its currently looking at. Since those .class files could be in any number of sub-directories, I'm not sure how to make them all dump in the top folder, without executing a modified Auto.sh in the top directory of each application.
Ok, so to fix the multiple find:
grep -R -o --include="Auto.sh" [\w] | wc -l
Should be:
grep -R -l --include=Auto.sh '\w' | wc -l
The reason this was happening, was that it was looking for instances of the letter w in Auto.sh. Which occurred 5 times in the file.
However, the overall fix that doesn't require having to place Auto.sh in every directory, is something like this:
MAIN_DIR=/home/mainuser/CaseStudies/
cd $MAIN_DIR
ls -d */ > DirectoryList.txt
while read LINE; do
cd $LINE
mkdir ProjectOutputs
bash /home/mainuser/Auto.sh
cd $MAIN_DIR
done <DirectoryList.txt
That calls this Auto.sh code:
index=1
grep -R -o --include=*.class '\w' | wc -l
grep -R -o --include=*.class '\w' | awk '{print $3}' > ProjectOutputs.txt
while read LINE; do
echo 'Path '$LINE > 'ProjectOutputs/ClassOut'$index'.txt'
javap -c $LINE >> 'ProjectOutputs/ClassOut'$index'.txt'
index=$((index+1))
done <ProjectOutputs.txt
Thanks again for everyone's help!
There is an input like:
folder1
folder2
folder3
...
foldern
I would like to iterate over taking multiple lines at once and processes each line, remove the first / (and more but for now this is enough) and echo the. Iterating over in bash with a single thread can be slow sometimes. The alternative way of doing this would be splitting up the input file to N pieces and run the same script with different input and output N times, at the end you can merge the results.
I was wondering if this is possible with xargs.
Update 1:
Input:
/a/b/c
/d/f/e
/h/i/j
Output:
mkdir a/b/c
mkdir d/f/e
mkdir h/i/j
Script:
for i in $(<test); do
echo mkdir $(echo $i | sed 's/\///') ;
done
Doing it with xargs does not work as I would expect:
xargs -a test -I line --max-procs=2 echo mkdir $(echo $line | sed 's/\///')
Obviously I need a way to execute the sed on the input for each line, but using $() does not work.
You probably want:
--max-procs=max-procs, -P max-procs
Run up to max-procs processes at a time; the default is 1. If
max-procs is 0, xargs will run as many processes as possible at
a time. Use the -n option with -P; otherwise chances are that
only one exec will be done.
http://unixhelp.ed.ac.uk/CGI/man-cgi?xargs
With GNU Parallel you can do:
cat file | perl -pe s:/:: | parallel mkdir -p
or:
cat file | parallel mkdir -p {= s:/:: =}
I have to figure out the physical disk number that belongs to each device in an OmniOS (Solaris 10) storage array. I can get the list of devices by
cfgadm -al | grep disk-path | cut -c 6-21 | tr 'a-z' 'A-Z'
where the output could look like
5000C5005CF65F14
5000C5004F30CC82
...
So my idea is to write a script where I dd each device and watch the leds, and then enter the number of the led that flashed. As there are leds on both sides of the storage array, I need to be able to run the script multiple times, and for each time I enter a disk location, I shouldn't have to enter it again.
My current idea is to loop over the list of device names I get from the above command and then do something like this
system("dd if=/dev/dsk/c1t${device}d0p0 of=/dev/null bs=1k count=100");
print "which led flashed: ";
my $disk = <STDIN>;
chomp $disk;
system("echo $disk $device >> disk.sorted");
which would produce lines like these
21 5000C5005CF65F14
09 5000C5004F30CC82
...
where I have seen led 21 flash in the first case and seen led 9 in the second case. There are 70 disks.
My problem
I can not come up with a good idea how I can write a script which can be run multiple times, and for each time it is run it will not destroy my previous values I have entered.
Any ideas how to do this?
I am prototyping it on Linux.
For each run of your skript, write the output into a different file, say out.1, out.2, and so on. Afterwards run
sort -k +2 out.*
you will have all results for one disk one after another. The sort will sort the contents of all files given according to the second column, which is the disk id.
Script 1
rm -f /tmp/a/*
rm -f /tmp/b/*
mkdir -p /tmp/a
mkdir -p /tmp/b
for f in $(cfgadm -al | grep disk-path | cut -c 6-21 | tr 'a-z' 'A-Z'); do touch /tmp/a/$f; done
Script 2
controller="c3t"
for f in /tmp/a/*; do
dd if=/dev/dsk/$controller${f##*/}d0p0 of=/dev/null bs=1k count=100
echo "Which led flashed? Press RET to skip to next"
read n
if ! [ -z $n ]; then echo $controller${f##*/} > /tmp/b/$n && rm -f $f; fi
done
cat /tmp/b/*
Script 3
for f in /tmp/b/*; do
echo $f
dd if=/dev/dsk/$(cat $f)d0p0 of=/dev/null bs=1k count=100
done
I work with some log system which creates a log file every hour, like follows:
SoftwareLog.2010-08-01-08
SoftwareLog.2010-08-01-09
SoftwareLog.2010-08-01-10
I'm trying to tail to follow the latest log file giving a pattern (e.g. SoftwareLog*) and I realize there's:
tail -F (tail --follow=name --retry)
but that only follow one specific name - and these have different names by date and hour. I tried something like:
tail --follow=name --retry SoftwareLog*(.om[1])
but the wildcard statement is resoved before it gets passed to tail and doesn't re-execute everytime tail retries.
Any suggestions?
I believe the simplest solution is as follows:
tail -f `ls -tr | tail -n 1`
Now, if your directory contains other log files like "SystemLog" and you only want the latest "SoftwareLog" file, then you would simply include a grep as follows:
tail -f `ls -tr | grep SoftwareLog | tail -n 1`
[Edit: after a quick googling for a tool]
You might want to try out multitail - http://www.vanheusden.com/multitail/
If you want to stick with Dennis Williamson's answer (and I've +1'ed him accordingly) here are the blanks filled in for you.
In your shell, run the following script (or it's zsh equivalent, I whipped this up in bash before I saw the zsh tag):
#!/bin/bash
TARGET_DIR="some/logfiles/"
SYMLINK_FILE="SoftwareLog.latest"
SYMLINK_PATH="$TARGET_DIR/$SYMLINK_FILE"
function getLastModifiedFile {
echo $(ls -t "$TARGET_DIR" | grep -v "$SYMLINK_FILE" | head -1)
}
function getCurrentlySymlinkedFile {
if [[ -h $SYMLINK_PATH ]]
then
echo $(ls -l $SYMLINK_PATH | awk '{print $NF}')
else
echo ""
fi
}
symlinkedFile=$(getCurrentlySymlinkedFile)
while true
do
sleep 10
lastModified=$(getLastModifiedFile)
if [[ $symlinkedFile != $lastModified ]]
then
ln -nsf $lastModified $SYMLINK_PATH
symlinkedFile=$lastModified
fi
done
Background that process using the normal method (again, I don't know zsh, so it might be different)...
./updateSymlink.sh 2>&1 > /dev/null
Then tail -F $SYMLINK_PATH so that the tail hands the changing of the symbolic link or a rotation of the file.
This is slightly convoluted, but I don't know of another way to do this with tail. If anyone else knows of a utility that handles this, then let them step forward because I'd love to see it myself too - applications like Jetty by default do logs this way and I always script up a symlinking script run on a cron to compensate for it.
[Edit: Removed an erroneous 'j' from the end of one of the lines. You also had a bad variable name "lastModifiedFile" didn't exist, the proper name that you set is "lastModified"]
I haven't tested this, but an approach that may work would be to run a background process that creates and updates a symlink to the latest log file and then you would tail -f (or tail -F) the symlink.
#!/bin/bash
PATTERN="$1"
# Try to make sure sub-shells exit when we do.
trap "kill -9 -- -$BASHPID" SIGINT SIGTERM EXIT
PID=0
OLD_FILES=""
while true; do
FILES="$(echo $PATTERN)"
if test "$FILES" != "$OLD_FILES"; then
if test "$PID" != "0"; then
kill $PID
PID=0
fi
if test "$FILES" != "$PATTERN" || test -f "$PATTERN"; then
tail --pid=$$ -n 0 -F $PATTERN &
PID=$!
fi
fi
OLD_FILES="$FILES"
sleep 1
done
Then run it as: tail.sh 'SoftwareLog*'
The script will lose some log lines if the logs are written to between checks. But at least it's a single script, with no symlinks required.
We have daily rotating log files as: /var/log/grails/customer-2020-01-03.log. To tail the latest one, the following command worked fine for me:
tail -f /var/log/grails/customer-`date +'%Y-%m-%d'`.log
(NOTE: no space after the + sign in the expression)
So, for you, the following should work (if you are in the same directory of the logs):
tail -f SoftwareLog.`date +'%Y-%m-%d-%H'`
I believe the easiest way is to use tail with ls and head, try something like this
tail -f `ls -t SoftwareLog* | head -1`