I am getting a stream of numbers in a pipe, and would like to perform some operations before passing them on to the next section, but I'm a little lost about how I would go about it without breaking the pipe.
for example
> echo "1 2 3 4 5" | some command | cat
1 4 9 16 25
>
Would you have any ideas on how to make something like this work? The actual operation I want to perform is simply adding one to every number.
echo 1 2 3 4 5|{
read line;
for i in $line;
do
echo -n "$((i * i)) ";
done;
echo
}
The {} creates a grouping. You could instead create a script for that.
I'd write:
echo "1 2 3 4 5" | {
for N in $(cat); do
echo $((N ** 2))
done | xargs
}
We can think of it as a "map" (functional programming). There are a lot of ways of writing a "map" function in bash (using stdin, function args, ...), for example:
map_stdin() {
local FUNCTION=$1
while read LINE; do
$FUNCTION $LINE
done
}
square() { echo "$(($1 * $1))"; }
$ echo "1 2 3 4 5" | xargs -n1 | map_stdin square | xargs
1 4 9 16 25
Or..
echo "1 2 3 4 5" | xargs -n 1 | while read number
do
echo $((number * number))
done
echo 1 2 3 4 5 | xargs -n 1 expr -1 +
echo 1 2 3 4 5 | xargs -n 1 bash -c 'echo $(($1*$1))' args
Using awk is another solution, which also works with floats
echo "1 2 3 4 5" | xargs -n1 | awk '{print $1^2}' | xargs
or use a loop
for x in 1 2 3 4 5; do echo $((x**2)); done | xargs
for x in $(echo "1 2 3 4 5"); do echo $x^2 | bc; done | xargs # alternative solution
for x in $(seq 5); do python -c "print($x**2)"; done | xargs # alternative solution but slower than the above
# or make it neat by defining a function to do basic math in bash, e.g.:
calc() { awk "BEGIN{print $*}"; }
for x in $(seq 5); do calc $x^2; done | xargs
Or you can pipe to expression to bc:
echo "1 2 3 4 5" | (
read line;
for i in $line;
do
echo $i^2 | bc;
done;
echo
)
If you prefer Python:
#!/bin/python
num = input()
while num:
print(int(num) + 1) # Whatever manipulation you want
try:
num = input()
except EOFError:
break
xargs, xargs, xargs
echo 1 2 3 4 5 | xargs -n1 echo | xargs -I NUMBER expr NUMBER \* NUMBER | xargs
Or, go parallel:
squareit () { expr $1 \* $1; }
export -f squareit
echo 1 2 3 4 5 | xargs -n1 | parallel --gnu squareit | xargs
Which would be way simpler if you passed your pipe as a standard set of args:
parallel --gnu "expr {} \* {}" ::: $(echo 1 2 3 4 5) | xargs
Or even:
parallel --gnu "expr {} \* {}" ::: 1 2 3 4 5 | xargs
Really worth taking a look at the examples in the doc: https://www.gnu.org/software/parallel/man.html
Yoi might like something like this:
echo "1 2 3 4 5" | perl -ne 'print $_ ** 2, " " for split / /, $_'
or even like this:
echo "1 2 3 4 5" | perl -ne 'print join " ", map {$_ ** 2} split / /, $_'
Related
Can someone explain why the below
$ echo $a
1 2 3
$ echo $a | xargs -n1 -I{} echo {}
1 2 3
isn't outputted as?
1
2
3
I would like to end up with
cp 1 1.old
cp 2 2.old
cp 3 3.old
and understand how -I works in xargs in the process.
Regarding your question-I and -n cannot work together but if you put it separately, it can work
a="1 2 3"; echo $a | xargs -n1 | xargs -I{} echo cp {} {}.old
You can use this printf | xargs:
a='1 2 3'
printf '%s\n' $a | xargs -I{} echo cp {} {}.old
Once satisfied with the output, you can remove echo before cp.
or else, without using printf, you can do this in xargs only:
xargs -I{} echo cp {} {}.old <<< "${a// /$'\n'}
cp 1 1.old
cp 2 2.old
cp 3 3.old
Here is a pure bash way of doing this:
for v in $a; do
echo cp "$v" "$v.old"
done
I have a text file, with many lines. I also have a selected number of lines I want to print out, in certain order. Let's say, for example, "5, 3, 10, 6". In this order.
Is there some easy and "canonical" way of doing this? (with "standard" Linux tools, and bash)
When I tried the answers from this question
Bash tool to get nth line from a file
it always prints the lines in order they are in the file.
A one liner using sed:
for i in 5 3 10 6 ; do sed -n "${i}p" < ff; done
A rather efficient method if your file is not too large is to read it all in memory, in an array, one line per field using mapfile (this is a Bash ≥4 builtin):
mapfile -t array < file.txt
Then you can echo all the lines you want in any order, e.g.,
printf '%s\n' "${array[4]}" "${array[2]}" "${array[9]}" "${array[5]}"
to print the lines 5, 3, 10, 6. Now you'll feel it's a bit awkward that the array fields start with a 0 so that you have to offset your numbers. This can be easily cured with the -O option of mapfile:
mapfile -t -O 1 array < file.txt
this will start assigning to array at index 1, so that you can print your lines 5, 3, 10 and 6 as:
printf '%s\n' "${array[5]}" "${array[3]}" "${array[10]}" "${array[6]}"
Finally, you want to make a wrapper function for this:
printlines() {
local i
for i; do printf '%s\n' "${array[i]}"; done
}
so that you can just state:
printlines 5 3 10 6
And it's all pure Bash, no external tools!
As #glennjackmann suggests in the comments you can make the helper function also take care of reading the file (passed as argument):
printlinesof() {
# $1 is filename
# $2,... are the lines to print
local i array
mapfile -t -O 1 array < "$1" || return 1
shift
for i; do printf '%s\n' "${array[i]}"; done
}
Then you can use it as:
printlinesof file.txt 5 3 10 6
And if you also want to handle stdin:
printlinesof() {
# $1 is filename or - for stdin
# $2,... are the lines to print
local i array file=$1
[[ $file = - ]] && file=/dev/stdin
mapfile -t -O 1 array < "$file" || return 1
shift
for i; do printf '%s\n' "${array[i]}"; done
}
so that
printf '%s\n' {a..z} | printlinesof - 5 3 10 6
will also work.
Here is one way using awk:
awk -v s='5,3,10,6' 'BEGIN{split(s, a, ","); for (i=1; i<=length(a); i++) b[a[i]]=i}
b[NR]{data[NR]=$0} END{for (i=1; i<=length(a); i++) print data[a[i]]}' file
Testing:
cat file
Line 1
Line 2
Line 3
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10
Line 11
Line 12
awk -v s='5,3,10,6' 'BEGIN{split(s, a, ","); for (i=1; i<=length(a); i++) b[a[i]]=i}
b[NR]{data[NR]=$0} END{for (i=1; i<=length(a); i++) print data[a[i]]}' file
Line 5
Line 3
Line 10
Line 6
First, generate a sed expression that would print the lines with a number at the beginning that you can later use to sort the output:
#!/bin/bash
lines=(5 3 10 6)
sed=''
i=0
for line in "${lines[#]}" ; do
sed+="${line}s/^/$((i++)) /p;"
done
for i in {a..z} ; do echo $i ; done \
| sed -n "$sed" \
| sort -n \
| cut -d' ' -f2-
I's probably use Perl, though:
for c in {a..z} ; do echo $c ; done \
| perl -e 'undef #lines{#ARGV};
while (<STDIN>) {
$lines{$.} = $_ if exists $lines{$.};
}
print #lines{#ARGV};
' 5 3 10 6
You can also use Perl instead of hacking with sed in the first solution:
for c in {a..z} ; do echo $c ; done \
| perl -e ' %lines = map { $ARGV[$_], ++$i } 0 .. $#ARGV;
while (<STDIN>) {
print "$lines{$.} $_" if exists $lines{$.};
}
' 5 3 10 6 | sort -n | cut -d' ' -f2-
l=(5 3 10 6)
printf "%s\n" {a..z} |
sed -n "$(printf "%d{=;p};" "${l[#]}")" |
paste - - | {
while IFS=$'\t' read -r nr text; do
line[nr]=$text
done
for n in "${l[#]}"; do
echo "${line[n]}"
done
}
You can use the nl trick: number the lines in the input and join the output with the list of actual line numbers. Additional sorts are needed to make the join possible as it needs sorted input (so the nl trick is used once more the number the expected lines):
#! /bin/bash
LINES=(5 3 10 6)
lines=$( IFS=$'\n' ; echo "${LINES[*]}" | nl )
for c in {a..z} ; do
echo $c
done | nl \
| grep -E '^\s*('"$( IFS='|' ; echo "${LINES[*]}")"')\s' \
| join -12 -21 <(echo "$lines" | sort -k2n) - \
| sort -k2n \
| cut -d' ' -f3-
I would like to simulate GNU's head -n -3, which prints all lines except the last 3, because head on FreeBSD doesn't have this feature. So I am thinking of something like
seq 1 10 | perl -ne ...
Here I have used 10 lines, but it can be any number larger than 3.
Can it be done in Perl or some other way on FreeBSD in BASH?
A super primitive solution would be
seq 1 10 | sed '$d' | sed '$d' | sed '$d'
seq 1 10 | perl -e '#x=("")x3;while(<>){print shift #x;push #x,$_}'
or
perl -e '#x=("")x3;while(<>){print shift #x;push #x,$_}' file
or
command | perl -pe 'BEGIN{#x=("")x3}push #x,$_;$_=shift #x'
perl -pe 'BEGIN{#x=("")x3}push #x,$_;$_=shift #x' file
seq 1 10 | perl -ne 'push #l, $_; print shift #l if #l > 3'
Pure bash and simple tools (wc and cut):
head -n $(($(wc -l file | cut -c-8)-3)) file
Disclaimer - I don't have access to FreeBSD right now, but this does work on OSX bash.
This works with a pipe as well as an input file:
seq 1 10 | perl -e'#x=<>;print#x[0..$#x-3]'
Nobody seems to have use sed and tac, so here's one:
$ seq 10 | tac | sed '1,3d' | tac
1
2
3
4
5
6
7
how about :
seq 1 10 | perl -ne 'print if ( !eof )' | perl -ne 'print if ( !eof )' | perl -ne 'print if ( !eof )'
This awk one-liner seems to do the job:
awk '{a[NR%4]=$0}NR>3{print a[(NR-3)%4]}' file
Or do it with bash alone if you have version 4.0 or newer:
seq 1 10 | (readarray -t LINES; printf '%s\n' "${LINES[#]:(-3)}")
Update: This one would remove the last three lines instead of showing only them.
seq 1 10 | (readarray -t L; C=${#L[#]}; printf '%s\n' "${L[#]:0:(C > 3 ? C - 3 : 0)}")
For convenience it could be placed on a function:
function exclude_last_three {
local L C
readarray -t L; C=${#L[#]}
printf '%s\n' "${L[#]:0:(C > 3 ? C - 3 : 0)}"
}
seq 1 10 | exclude_last_three
seq 11 20 | exclude_last_three
Here's a late answer, because I was running into something like this yesterday.
This solution is:
pure bash
one-liner
reads the input stream only once
reads the input stream line-by-line, not all at once
Tested on Ubuntu, Redhat and OSX.
$ seq 1 10 | { n=3; i=1; while IFS= read -r ln; do [ $i -gt $n ] && cat <<< "${buf[$((i%n))]}"; buf[$((i%n))]="$ln"; ((i++)); done; }
1
2
3
4
5
6
7
$
It works by reading lines into a circular buffer implemented as an n-element array.
n is the number of lines to cut off the end of the file.
For every line i we read, we can echo the line i-n from the circular buffer, then store the line i in the circular buffer. Nothing is echoed until the first n lines are read. (i mod n) is the index into the array which implements the circular buffer.
Because the requirement is for a one-liner, I tried to make it fairly brief, unfortunately at the expense of readability.
Another Awk solution that only uses minimal amount of buffers and prints lines quickly without needing to read all the lines first. It can also be used with pipes and large files.
awk 'BEGIN{X = 3; for(i = 0; i < X; ++i)getline a[i]}{i %= X; print a[i]; a[i++] = $0}'
If I have
days="1 2 3 4 5 6"
func() {
echo "lSecure1"
echo "lSecure"
echo "lSecure4"
echo "lSecure6"
echo "something else"
}
and do
func | egrep "lSecure[1-6]"
then I get
lSecure1
lSecure4
lSecure6
but what I would like is
lSecure2
lSecure3
lSecure5
which is all the days that doesn't have a lSecure string.
Question
My current idea is to use awk to split the $days and then loop over all combinations.
Is there a better way?
Note that grep -v inverts the sense of a plain grep and does not solve the problem as it does not generate the required strings.
I usually use the -f flag of grep for similar purposes. The <( ... ) code generates a file with all possibilities, grep only selects those not present in the func.
func | grep 'lSecure[1-6]' | grep -v -f- <( for i in $days ; do echo lSecure$i ; done )
Or, you may prefer it the other way round:
for i in $days ; do echo lSecure$i ; done | grep -vf <( func | grep 'lSecure[1-6]' )
F=$(func)
for f in $days; do
if ! echo $F | grep -q lSecure$f; then
echo lSecure$f
fi
done
An awk solution:
$ func | awk -v i="${days}" 'BEGIN{split(i,a," ")}{gsub(/lSecure/,"");
for(var in a)if(a[var] == $0){delete a[var];break}}
END{for(var in a) print "lSecure" a[var]}' | sort
We store it in an awk array a then while reading a line, get the last number, if it is present in array, then remove that from the array. So at the end, in the array, only those element which have not been found remains. Sort is just to present in a sorted manner :)
I am not sure exactly what you are trying to achieve, but you might consider using uniq -u which deletes repeated sequences. For example you can do this with it:
( echo "$days" | tr -s ' ' '\n'; func | grep -oP '(?<=lSecure)[1-6]' ) | sort | uniq -u
Output:
2
3
5
How do I sort a "python dictionary-style" array e.g. ( "A: 2" "B: 3" "C: 1" ) in bash by the value? I think, this code snippet will make it bit more clear about my question.
State="Total 4 0 1 1 2 0 0"
W=$(echo $State | awk '{print $3}')
C=$(echo $State | awk '{print $4}')
U=$(echo $State | awk '{print $5}')
M=$(echo $State | awk '{print $6}')
WCUM=( "Owner: $W;" "Claimed: $C;" "Unclaimed: $U;" "Matched: $M" )
echo ${WCUM[#]}
This will simply print the array: Owner: 0; Claimed: 1; Unclaimed: 1; Matched: 2
How do I sort the array (or the output), eliminating any pair with "0" value, so that the result like this:
Matched: 2; Claimed: 1; Unclaimed: 1
Thanks in advance for any help or suggestions. Cheers!!
Quick and dirty idea would be (this just sorts the output, not the array):
echo ${WCUM[#]} | sed -e 's/; /;\n/g' | awk -F: '!/ 0;?/ {print $0}' | sort -t: -k 2 -r | xargs
echo -e ${WCUM[#]} | tr ';' '\n' | sort -r -k2 | egrep -v ": 0$"
Sorting and filtering are independent steps, so if you only like to filter 0 values, it would be much more easy.
Append an
| tr '\n' ';'
to get it to a single line again in the end.
nonull=$(for n in ${!WCUM[#]}; do echo ${WCUM[n]} | egrep -v ": 0;"; done | tr -d "\n")
I don't see a good reason to end $W $C $U with a semicolon, but $M not, so instead of adapting my code to this distinction I would eliminate this special case. If not possible, I would append a semicolon temporary to $M and remove it in the end.
Another attempt, using some of the bash features, but still needs sort, that is crucial:
#! /bin/bash
State="Total 4 1 0 4 2 0 0"
string=$State
for i in 1 2 ; do # remove unnecessary fields
string=${string#* }
string=${string% *}
done
# Insert labels
string=Owner:${string/ /;Claimed:}
string=${string/ /;Unclaimed:}
string=${string/ /;Matched:}
# Remove zeros
string=(${string[#]//;/; })
string=(${string[#]/*:0;/})
string=${string[#]}
# Format
string=${string//;/$'\n'}
string=${string//:/: }
# Sort
string=$(sort -t: -nk2 <<< "$string")
string=${string//$'\n'/;}
echo "$string"