how to pass the filename as variable to a awk command from a shell script - linux

in my shell script i have the following line
PO_list=$(awk -v col="$1" -F";" '!seen[$col]++ {print $col}' test.csv)
which generates a list with the values from column "col" which came from "$1" from file test.csv.
it might be possible to have several files in same location and for this would need to loop among them with a for sentence. For this I have to replace the filename test.csv with a variable, $i for example, which is the index from the list of files.
trying to fulfill my request, I was modifying my line with
PO_list=$(awk -v col="$1" -F";" '!seen[$col]++ {print $col}' $j)
unfortunately, i receive the error message:
awk: cannot open test.csv (No such file or directory)
Can anyone tell me why this error occur and how can I solve it, please?
Thank you,

As you commented in your previous question, you are calling it with
abc$ ./test.sh 2
So you just need to add another parameter when you call it:
abc$ ./test.sh 2 "test.csv"
and the script can be like this:
PO_list=$(awk -v col="$1" -F";" '!seen[$col]++ {print $col}' "$2")
# ^^^^
Whenever you want to use other parameters, remember they are positional. Hence, the first one is $1, second is $2 and so on.
In case the file happens to be in another directory, you can replace ./test.sh 2 "test.csv" by something like ./test.sh 2 "/full/path/of/test.csv" or whatever relative path you may need.

Related

split and write the files with AWK -Bash

INPUT_FILE.txt in c:\Pro\usr\folder1
ABCDEFGH123456
ABCDEFGH123456
ABCDEFGH123456
BBCDEFGH123456
BBCDEFGH123456
used the below AWK command in .SH script which runs from c:\Pro\usr\folder2 to split the file to multiple txt files with an extension of _kg based on first 8 characters.
awk '{ F=substr($0,1,8) "_kg" ".txt"; print $0 >> F; close(F) }' ' "c:\Pro\usr\folder1\input_file.txt"
this is working good , but the files are writing in the main location where the bash is pointing. How can I route the created files to another location like c:\Pro\usr\folder3.
Thanks
Following awk code may help you in same, written and tested with shown samples in GNU awk.
awk -v outPath='c:\\Pro\\usr\\folder3' -v FPAT='^.{8}' '{outFile=($1"_kg.txt");outFile=outPath"\\"outFile;print > (outFile);close(outFile)}' Input_file
Explanation: Creating an awk variable named outPath which has path mentioned by OP in samples. Then setting FPAT(field separator settings as a regex), where I am creating field of 8 characters starting from first character. In main program of awk, creating outFile variable which has output file names in it(1st field following by _kg.txt), then printing whole line to output file and closing the output file in backend to avoid "too many opened files" error.
Pass the destination folder as a variable to awk:
awk -v dest='c:\\Pro\\usr\\folder3\\' '{F=dest substr($0,1,8) "_kg" ".txt"; print $0 >> F; close(F) }' "c:\Pro\usr\folder1\input_file.txt"
I think the doubled backslashes are required.

Assign command output to variable in Bash?

I know this seems fairly trivial. But I have no idea where I am going wrong. I have a shell script where I download a package based on the input argument and then extract the package name. This is how I do it:
wget $1
echo $1 | awk -F/ '{print $NF}'
I run it like this bash scrip.sh http://apache.claz.org/phoenix/apache-phoenix-4.10.0-HBase-1.2/bin/apache-phoenix-4.10.0-HBase-1.2-bin.tar.gz
I download the package and then the second line splits the input variable along the / delimiter I get apache-phoenix-4.10.0-HBase-1.2-bin.tar.gz. Now I want to assign the result of the second line to a variable. I change my script to a dir=$($1 | awk -F/ '{print $NF}') and add an echo $dir to the script to see the result. However I keep running into this error : line 2: http://apache.claz.org/phoenix/apache-phoenix-4.10.0-HBase-1.2/bin/apache-phoenix-4.10.0-HBase-1.2-bin.tar.gz: No such file or directory
I tried wrapping the command into `` but the problem persists. I am not cd-ing into any directory so I have no idea why this error keeps showing up.

View logs from yesterday/today

I'm looking for a way to view multiple log files for the last two days into a single pass.
At first, I tried with GREP:
#!/bin/bash
yesterday=$(date --date="yesterday" +"%Y-%m-%d")
today=$(date +"%Y-%m-%d")
grep "$yesterday\|$today" *.log | less
This is nice but it doesn't output lines in between matches (lines that don't have the date in them, like error stack traces - which is what I'm really interested in)...
So I found this:
#!/bin/bash
yesterday=$(date --date="yesterday" +"%Y-%m-%d")
sed -ne '/$yesterday/,$p' *.log | less
For each file, it outputs everything from the first match to the end of the file. That's just perfect... except for one thing... When reading it, I don't know which file's content I'm looking at. I would like to see the file name at the start of each line, just like with a grep.
How can I prefix the file name to each line in my sed command?
Would there be a nicer / better way to do this?
Thanks ;-)
Not a sed solution but as you asked for a nicer / better way to do this... If you have GNU awk somewhere,
awk -v day="$yesterday" 'BEGINFILE {run=0} $0 ~ day {run=1} run == 1 {print FILENAME, $0}' *.log
should make it.
Explanation:
GNU awk processes all files in sequence. The GNU awk variable day is initialized to the shell expression "$yesterday" GNU awk executes the BEGINFILE rule before processing a new file. This rule clears the run variable. Whenever a line ($0) matches GNU awk variable day ("$yesterday") the run variable is set. And when the run variable is set, the name of the current file is printed (FILENAME), followed by the current line ($0).

renaming files using loop in unix

I have a situation here.
I have lot of files like below in linux
SIPTV_FIPTV_ID00$line_T20141003195717_C0000001000_FWD148_IPV_001.DATaac
SIPTV_FIPTV_ID00$line_T20141003195717_C0000001000_FWD148_IPV_001.DATaag
I want to remove the $line and make a counter from 0001 to 6000 for my 6000 such files in its place.
Also i want to remove the trailer 3 characters after this is done for each file.
After fix file should be like
SIPTV_FIPTV_ID0000001_T20141003195717_C0000001000_FWD148_IPV_001.DAT
SIPTV_FIPTV_ID0000002_T20141003195717_C0000001000_FWD148_IPV_001.DAT
Please help.
With some assumption, I think this should do it:
1. list of the files is in a file named input.txt, one file per line
2. the code is running in the directory the files are in
3. bash is available
awk '{i++;printf "mv \x27"$0"\x27 ";printf "\x27"substr($0,1,16);printf "%05d", i;print substr($0,22,47)"\x27"}' input.txt | bash
from the command prompt give the following command
% echo *.DAT??? | awk '{
old=$0;
sub("\\$line",sprintf("%4.4d",++n));
sub("...$","");
print "mv", old, $1}'
%
and check the output, if it looks OK
% echo *.DAT??? | awk '{
old=$0;
sub("\\$line",sprintf("%4.4d",++n));
sub("...$","");
print "mv", old, $1}' | sh
%
A commentary: echo *.DAT??? is meant to give as input to awk a list of all the filenames that you want to modify, you may want something more articulated if the example names you gave aren't representative of the whole spectrum... regarding the awk script itself, I used sprintf to generate a string with the correct number of zeroes for the replacement of $line, the idiom `"\\$..." with two backslashes to quote the dollar sign is required by gawk and does no harm in mawk, and as a last remark I have to say that in similar cases I prefer to make at least a dry run before passing the commands to the shell...

How to move files based on file (file name and location in file)

I tried but I failed, I have file like:
06faefb38081b44e35b4ee846bfb84b61694a5c4.zip D:/code/3635/
0a386c77a3ae35033006efec07bfc37f5236212c.zip D:/code/3622/
0b425b29c3e51f29b9af03db46df0aa8f726705b.zip D:/code/3624/
0ecb477c82ec540c8eb0642da144a400bbf3f7c0.zip D:/code/3624/
...
My goal is to move file in first column to location in second column. I tried with while+awk but this did not worked. I would appreciate any help!
awk '{ system("mv "$1" "$2) }' filename
With awk, you can use the system function to build a move command and excute it. Obviously ensure that you are running the command in the directory with the files.
Let's assume, you file has name "data.txt", thus your code might look like this:
while read line; do mv $line; done < data.txt
What you need is just add a mv (space) to the beginning of each line. So you have 100+ ways to do that. If you love awk:
awk '$1="mv "$1' file
will create the mv command, to execute them, you can:
awk '$1="mv "$1' file |sh
I prefer this than the system() function call.

Resources