I have the following code which successfully reads the line of the file that I want:
tail -n 9 myfile | awk 'NR==1'
although I do not want that this writes anything in my script. So I tried to assign a parameter into it, but it doesn't work in this way:
this=tail -n 9 myfile | awk 'NR==1'
Eventually, I want to read the second argument, which is a number, by ${1}. Could you tell me how can I do that?
It sounds like you just want to capture the output in a variable. Thus you can do this:
awkOutput=$(tail -n 9 myfile | awk 'NR==1')
and later you cant print it out
echo $awkOutput
Related
This question already has answers here:
Conditional step in a pipeline
(2 answers)
Can I make a shell function in as a pipeline conditionally "disappear", without using cat?
(1 answer)
Closed 4 months ago.
EMPTY_VAR=''
MMDDYYYY='6.18.1997'
PIPE_VAR=' | xargs echo "1+" | bc'
echo "$MMDDYYYY" | cut -d "." -f 2${EMPTY_VAR}
>> 18
Command above would give me correct output, which is 18, but if I try to use PIPE_VAR instead it would give me bunch of errors:
echo "$MMDDYYYY" | cut -d "." -f 2${PIPE_VAR}
cut: '|': No such file or directory
cut: xargs: No such file or directory
cut: echo: No such file or directory
cut: '"1+"': No such file or directory
cut: '|': No such file or directory
cut: bc: No such file or directory
OR:
echo "$MMDDYYYY" | cut -d "." -f 2"$PIPE_VAR"
cut: invalid field value ‘| xargs echo "1+" | bc’
Try 'cut --help' for more information.
What I'm really trying to find out is that even possible to combine commands like this?
You can't put control operators like | in a variable, at least not without resorting to something like eval. Syntax parsing comes before parameter expansion when evaluating the command line, so Bash is only ever going to see that | as a literal character and not pipeline syntax. See BashParsing for more details.
Conditionally adding a pipeline is hard to do well, but having a part of the pipeline conditionally execute one command or another is more straightforward. It might look something like this:
#!/bin/bash
MMDDYYYY='6.18.1997'
echo "$MMDDYYYY" | cut -d "." -f 2 |
if some_conditional_command ; then
xargs echo "1+" | bc
else
cat
fi
It looks like you're trying to calculate the next day. That's hard to do with plain arithmetic, particularly with month/year ends.
Let date do the work. This is GNU date. It can't parse 6.18.1997 but it can parse 6/18/1997
for MMDDYYYY in '2.28.1996' '2.28.1997'; do
date_with_slashes=${MMDDYYYY//./\/}
next_day=$(date -d "$date_with_slashes + 1 day" '+%-m.%-d.%Y')
echo "$next_day"
done
2.29.1996
3.1.1997
I was using:
bash $ head -n 2 *.xml | grep (..stuff..)
to stream first 2 lines of all xml files to grep command. However, I realized that this was not reliable for the structure of these files.
What I need instead is to stream start of each xml file until a particular substring (which all these files have) is encountered.
head does not provide that level of granularity. The substring is simply the start of a tag (e.g. something like "< tag start"). I would be grateful for any ideas. Thanks!
If you know the max number of lines you have before the matching string you can do something like this:
# cat testfile
123
9
1
1
2
3
4000
TAG
456
# grep -m 1 -B 10 TAG testfile | grep -v TAG
123
9
1
1
2
3
4000
#
Sounds like you want either of these (using GNU awk for nextfile) depending on if you want the tag line printed or not:
awk '/< tag start/{nextfile} 1' *.xml
awk '1; /< tag start/{nextfile}' *.xml
or less efficiently with any awk:
awk 'FNR==1{f=1} /< tag start/{f=0} f' *.xml
awk 'FNR==1{f=1} f; /< tag start/{f=0}' *.xml
or bringing back some efficiency in this case:
for file in *.xml; do
awk '/< tag start/{exit} 1' "$file"
done
I appreciate all the responses. I found that really I only needed the content of a single tag, rather than from the beginning of the xml files. This simplified the parsing. So for instance:
<mt:myTag LOTSOFSTUFF >"
, I really only needed LOTSOFSTUFF. So I simply did:
grep -oP "<mt:myTag(.*)>" *.xml | grep_more
and that worked exactly. Thanks again. I really appreciated and sorry I did not realize my use case was simpler than I made it out to be.
While trying to read the version number of vim, I get a lot additional lines which I need to ignore. I tried to read the manual of head and tried the following command:
vim --version | head -n 1
I want to know if this is the correct approach?
Yes, that is one way to get the first line of output from a command.
If the command outputs anything to standard error that you would like to capture in the same manner, you need to redirect the standard error of the command to the standard output stream:
utility 2>&1 | head -n 1
There are many other ways to capture the first line too, including sed 1q (quit after first line), sed -n 1p (only print first line, but read everything), awk 'FNR == 1' (only print first line, but again, read everything) etc.
I would use:
awk 'FNR <= 1' file_*.txt
As #Kusalananda points out there are many ways to capture the first line in command line but using the head -n 1 may not be the best option when using wildcards since it will print additional info. Changing 'FNR == i' to 'FNR <= i' allows to obtain the first i lines.
For example, if you have n files named file_1.txt, ... file_n.txt:
awk 'FNR <= 1' file_*.txt
hello
...
bye
But with head wildcards print the name of the file:
head -1 file_*.txt
==> file_1.csv <==
hello
...
==> file_n.csv <==
bye
I have a situation here.
I have lot of files like below in linux
SIPTV_FIPTV_ID00$line_T20141003195717_C0000001000_FWD148_IPV_001.DATaac
SIPTV_FIPTV_ID00$line_T20141003195717_C0000001000_FWD148_IPV_001.DATaag
I want to remove the $line and make a counter from 0001 to 6000 for my 6000 such files in its place.
Also i want to remove the trailer 3 characters after this is done for each file.
After fix file should be like
SIPTV_FIPTV_ID0000001_T20141003195717_C0000001000_FWD148_IPV_001.DAT
SIPTV_FIPTV_ID0000002_T20141003195717_C0000001000_FWD148_IPV_001.DAT
Please help.
With some assumption, I think this should do it:
1. list of the files is in a file named input.txt, one file per line
2. the code is running in the directory the files are in
3. bash is available
awk '{i++;printf "mv \x27"$0"\x27 ";printf "\x27"substr($0,1,16);printf "%05d", i;print substr($0,22,47)"\x27"}' input.txt | bash
from the command prompt give the following command
% echo *.DAT??? | awk '{
old=$0;
sub("\\$line",sprintf("%4.4d",++n));
sub("...$","");
print "mv", old, $1}'
%
and check the output, if it looks OK
% echo *.DAT??? | awk '{
old=$0;
sub("\\$line",sprintf("%4.4d",++n));
sub("...$","");
print "mv", old, $1}' | sh
%
A commentary: echo *.DAT??? is meant to give as input to awk a list of all the filenames that you want to modify, you may want something more articulated if the example names you gave aren't representative of the whole spectrum... regarding the awk script itself, I used sprintf to generate a string with the correct number of zeroes for the replacement of $line, the idiom `"\\$..." with two backslashes to quote the dollar sign is required by gawk and does no harm in mawk, and as a last remark I have to say that in similar cases I prefer to make at least a dry run before passing the commands to the shell...
The line I seek is stored in the file data.txt and is the only line of text that occurs only once.
How do I go about finding that particular line using linux?
This is a little bit old, but I think you are looking for this...
cat data.txt | sort | uniq -u
This will show the unique values that only occur once in the file. I assume you are familiar with "over the wire" if you are asking?? If so, this is what you are looking for.
To provide some context (I need more rep to comment) this is a question that features in an online "wargame" called Bandit that involves using the command line to discover passwords on an online Linux server to advance up the levels.
For those who would like to see data.txt in full I've Pastebin'd it here however it looks like this:
NN4e37KW2tkIb3dC9ZHyOPdq1FqZwq9h
jpEYciZvDIs6MLPhYoOGWQHNIoQZzE5q
3rpovhi1CyT7RUTunW30goGek5Q5Fu66
JOaWd4uAPii4Jc19AP2McmBNRzBYDAkO
JOaWd4uAPii4Jc19AP2McmBNRzBYDAkO
9WV67QT4uZZK7JHwmOH0jnhurJMwoGZU
a2GjmWtTe3tTM0ARl7TQwraPGXgfkH4f
7yJ8imXc7NNiovDuAl1ZC6xb0O0mMBx1
UsvVyFSfZZWbi6wgC7dAFyFuR6jQQUhR
FcOJhZkHlnwqcD8QbvjRyn886rCrnWZ7
E3ugYDa6Wh2y8C8xQev7vOS8O3OgG1Hw
E3ugYDa6Wh2y8C8xQev7vOS8O3OgG1Hw
ME7nnzbId4W3dajsl6Xtviyl5uhmMenv
J5lN3Qe4s7ktiwvcCj9ZHWrAJcUWEhUq
aouHvjzagN8QT2BCMB6e9rlN4ffqZ0Qq
ZRF5dlSuwuVV9TLhHKvPvRDrQ2L5ODfD
9ZjR3NTHue4YR6n4DgG5e0qMQcJjTaiM
QT8Bw9ofH4x3MeRvYAVbYvV1e1zq3Xim
i6A6TL6nqvjCAPvOdXZWjlYgyvqxmB7k
tx7tQ6kgeJnC446CHbiJY7fyRwrwuhrs
One way to do it is to use:
sort data.txt | uniq -u
The sort command is like cat in that it displays the contents of the file however it sorts the file lexicographically by lines (it reorders them alphabetically so that matching ones are together).
The | is a pipe that redirects the output from one command into another.
The uniq command reports or omits repeated lines and by passing it the -u argument we tell it to report only unique lines.
Used together like this, the command will sort data.txt lexicographically by each line, find the unique line and print it back in the terminal for you.
sort -u data.txt | while read line; do if [ $(grep -c $line data.txt) == 1 ] ;then echo $line; fi; done
was mine solution, until I saw here easy one:
sort data.txt | uniq -u
Add more information to you post.
How data.txt look like?
Like this:
11111111
11111111
pass1111
11111111
Or like this
afawfdgd
password
somethin
gelse...
And, do you know the password is in file or you search for not repeat string.
If you know password, use something like this
cat data.txt | grep 'password'
If you don`t know the password and this password is only unique line in file you must create a script.
For example in Python
file = open("data.txt","r")
f = file.read()
for line in f:
if 'pass' in line:
print pass
Of course replace pass with something else.
For example some slice from line.
And one with only one tool in use, awk:
awk '{a[$1]++}END{for(i in a){if(a[i] == 1){print i} }}' data.txt
sort data.txt | uniq -c | grep 1\ ?*
and it will print the only text that occurs only one time
do not forget to put space after the backslash
sort data.txt | uniq -c | grep 1
you will find only one that accures one time