Assume two multi-line text files that are dynamically generated during execution of a bash shell script: file1 and file2
$ echo -e "foo-bar\nbar-baz\nbaz-qux" > file1
$ cat file1
foo-bar
bar-baz
baz-qux
$ echo -e "foo\nbar\nbaz" > file2
$ cat file2
foo
bar
baz
Further assume that I wish to use awk involving an operation on the text strings of both files. For example:
$ awk 'NR==FNR{var1=$1;next} {print $var1"-"$1}' FS='-' file1 FS=' ' file2
Is there any way that I can skip having to save the text strings as files in my script and, instead, pass along the text strings to awk as variables (or as here-strings or the like)?
Something along the lines of:
$ var1=$(echo -e "foo-bar\nbar-baz\nbaz-qux")
$ var2=$(echo -e "foo\nbar\nbaz")
$ awk 'NR==FNR{var1=$1;next} {print $var1"-"$1}' FS='-' "$var1" FS=' ' "$var2"
# awk: fatal: cannot open file `foo-bar
# bar-baz
# baz-qux' for reading (No such file or directory)
$ awk '{print FILENAME, FNR, $0}' <(echo 'foo') <(echo 'bar')
/dev/fd/63 1 foo
/dev/fd/62 1 bar
Related
I have a .txt file having files names as
z1.cap
z2.cap
z3.cap
z4.cap
Sample data present in these files are like shown below,
OTR 25896 PAT210 $TREMD DEST
OFR 21475 NAT102 #TREMD DEST
then I'm using below code to print desired values from files.
while read file_name
do
echo "progressing with file :${file_name}"
cat ${file_name} | grep "PAT210" | awk -F' ' '$5 == "(DEST" { print $file_name, $1}' | uniq >> OUTPUT_FILE
Now I want output which consists of 2 fields like,
z1.cap OTR
z2.cap OFR
and so on...
But i'm getting ouputs like,
- OTR
- OFR
Any help is aprreciated, Thanks.
To access the filename that awk is currently processing use the builtin variable FILENAME
To bind other shell variables from your shell to variables in awk use:
awk -v var1=$shvar1 -v var2=$shvar2 'your awk code using var1 and var2'
Assuming files.txt contains your list of files and with zero understanding of what exactly you are trying to achieve:
for file_name in $(cat files.txt)
do
echo "progressing with file :${file_name}"
awk -F' ' '($5 == "DEST") && ($3=="PAT210") { print FILENAME, $1}' $file_name | uniq >> OUTPUT_FILE
done
I removed the cat and incorporated the grep into your awk. The cat was unnecessary since awk can read the file itself.
You can remove the for loop entirely by saying
awk -F' ' '($5 == "DEST") && ($3=="PAT210") { print FILENAME, $1}' $(<files.txt) | uniq >> OUTPUT_FILE
The $(<files.txt) will send each filename to awk.
I have the text file with the column of the numbers, that I need to transform to the line with the numbers separated by ', '
For example:
$ cat file.txt
1034008
1034043
10340431
1034051
Then I use tr:
tr "\n" "', '" < file.txt > file2.txt
But, result is:
$ cat file2.txt
1034008'1034043'10340431'1034051
So, what I need to do to get the correct result?
tr can only to one-to-one mapping, not one-to-many
$ # convert all input lines to one line
$ # using , as separator, cannot give multiple character separator
$ paste -sd, ip.txt
1034008,1034043,10340431,1034051
$ # post process it
$ paste -sd, ip.txt | sed 's/,/, /g'
1034008, 1034043, 10340431, 1034051
$ # or use a tool that allows input record separator manipulation
$ perl -pe 's/\n/, / unless eof' ip.txt
1034008, 1034043, 10340431, 1034051
1.We can do this by sed.
The command N of sed can reads the next line into pattern space.So we use N to merge 2 lines into 1.But how to merge all lines into one?
We can set a lebel at the beginning and use t label to jump to the lebel to make a loop.
$ sed ':myLebel;N;s/\n/, /; t myLebel; ' file.txt > file2.txt
$ cat file2.txt
1034008, 1034043, 10340431, 1034051
2.In your question, we can use xargs to read all content into one line which is delimited by space,and then use sed to replace space to the strings you want.
$ cat file.txt | xargs |sed 's/ /, /g' > file2.txt
$ cat file2.txt
1034008, 1034043, 10340431, 1034051
Refer to:
How the 'N' command works in sed?
https://www.thegeekstuff.com/2009/12/unix-sed-tutorial-6-examples-for-sed-branching-operation/
pure bash, to avoid external commands (faster)
tk="$(< file.txt)"
echo "${tk//$'\n'/, }" > file2.txt
In a directory, there is several files such as:
file1
file2
file3
Is there a simple way to concatenate those files to get one line (connected by "OR") in bash as follows:
file1 OR file2 OR file3
Or do I need to write a script for it?
You can use this function to print all filenames (including ones with space, newline or special characters) with " OR " as separator (assuming your filename doesn't contain ASCII code 4):
orfiles() {
local IFS=$'\4'
local out="$*"
echo "${out//$'\4'/ OR }"
}
Then call it as:
orfiles *
How it works:
We set IFS (Internal Field Separator) to ASCII 4 locally inside the function
We store output of "$*" in local variable out. This will place \4 after each filename in variable $out.
Finally using BASH string substitution we globally replace \4 by " OR " while printing the output from $out.
In Unix systems IFS is only a single character delimiter therefore it cannot store multi character string " OR " and we have to do this in 2 steps as shown above.
You can simply do that with
printf '%s OR ' $(ls -1 *) | sed 's/OR $/''/'; echo -e '\n'
Where ls -1 * is the directory.
The moment that should be considered is that a filename could contain whitespace(s).
Use the following ls + awk solution:
ls -1 * | awk '{ r=(r)? r" OR "$0 : $0 }END{ print r }'
Workaround for filenames with newline(s):
echo -e $(ls -1b hello* | awk -v RS= '{gsub(/\n/," OR ",$0); gsub(/\\ /," ",$0); print $0}')
-b - ls option to print C-style escapes for nongraphic characters
ls -1|awk -v q='"' '{printf "%s%s", NR==1?"":" OR ", q $0 q}END{print ""}'
the ls & awk way to do it, with example that the filename containing spaces:
kent$ ls -1
file1
file2
'file with OR and space'
kent$ ls -1|awk -v q='"' '{printf "%s%s", NR==1?"":" OR ", q $0 q}END{print ""}'
"file1" OR "file2" OR "file with OR and space"
$ for f in *; do printf '%s%s' "$s" "$f"; s=" OR "; done; printf '\n'
file1 OR file2 OR file3
I have two files in Linux, in file a there are variables like these:
${VERSION} ${SOFTWARE_PRODUCER}
And the values of these variables are stored in file b:
VERSION=1.0.1
SOFTWARE_PRODUCER=Luc
Now how can I use command to replace the variables in file a with values in file b? Is something like sed able to do this task?
Thanks.
A simple bash loop would suffice:
$ cat a
This is file 'a' which has this variable ${VERSION}
and it has this also:
${SOFTWARE_PRODUCER}
$ cat b
VERSION=1.0.1
SOFTWARE_PRODUCER=Luc
$ cat script.bash
#!/bin/bash
while read line || [[ -n "$line" ]]
do
key=$(awk -F= '{print $1}' <<< "$line")
value=$(awk -F= '{print $2}' <<< "$line")
sed -i 's/${'"$key"'}/'"$value"'/g' a
done < b
$ ./script.bash
$ cat a
This is file 'a' which has this variable 1.0.1
and it has this also:
Luc
$
I wanna save this command to another text:
awk '{print $2}'
it extract's from text.
now i wanna save output too another text.
thanks
awk '{ print $2 }' text.txt > outputfile.txt
> => This will redirect STDOUT to a file. If file not exists, it will create it. If file exists it will clear out (in effect) the content and will write new data to it
>> => This means same as above but if file exists, this will append new data to it.
Eg:
$ cat /etc/passwd | awk -F: '{ print $1 }' | tail -10 > output.txt
$ cat output.txt
_warmd
_dovenull
_netstatistics
_avbdeviced
_krb_krbtgt
_krb_kadmin
_krb_changepw
_krb_kerberos
_krb_anonymous
_assetcache
Alternatively you can use the command tee for redirection. The command tee will redirect STDOUT to a specified file as well as the terminal screen
For more about shell redirection goto following link:
http://www.techtrunch.com/scripting/redirections-and-file-descriptors
There is a way to do this from within awk itself (docs)
➜ cat text.txt
line 1
line 2
line three
line 4 4 4
➜ awk '{print $2}' text.txt
1
2
three
4
➜ awk '{print $2 >"text.out"}' text.txt
➜ cat text.out
1
2
three
4
try this command.
cat ORGFILENAME.TXT | awk '{print $2}' > SAVENAME.TXT
thx.