As I am very new to UNIX and struggling with two commands to get the desired output.
What will be the command to print the name of the largest file in size in
/usr/include directory.
I tried but not giving the filename whose size is largest :
find /usr/include -type f -exec ls -s {} \; | sort -n | tail -n 5
command to print user ids on a Linux environment which can’t be
used as login ids. [hint: consider /etc/passwd file]
I tried: cat /etc/passwd
Please let me know if these are the correct commands.
ls -lS /usr/include | head -2 | grep -v total
Flag S is to sort files by size. Head then grabs the first 2 lines of the long output and grep removes the summary line.
awk -F: '{ if($3<100) print $1;}' /etc/passwd
UID is the third field in /etc/passwd. Generally UIDs 0-99 are reserved for predefined system accounts. So this command prints all usernames with UID less than 100.
Related
I have a directory with thousands of files (100K for now). When I use wc -l ./*, I'll get:
c1 ./test1.txt
c2 ./test2.txt
...
cn ./testn.txt
c1+c2+...+cn total
Because there are a lot of files in the directory, I just want to see the total count and not the details. Is there any way to do so?
I tried several ways and I got following error:
Argument list too long
If what you want is the total number of lines and nothing else, then I would suggest the following command:
cat * | wc -l
This catenates the contents of all of the files in the current working directory and pipes the resulting blob of text through wc -l.
I find this to be quite elegant. Note that the command produces no extraneous output.
UPDATE:
I didn't realize your directory contained so many files. In light of this information, you should try this command:
for file in *; do cat "$file"; done | wc -l
Most people don't know that you can pipe the output of a for loop directly into another command.
Beware that this could be very slow. If you have 100,000 or so files, my guess would be around 10 minutes. This is a wild guess because it depends on several parameters that I'm not able to check.
If you need something faster, you should write your own utility in C. You could make it surprisingly fast if you use pthreads.
Hope that helps.
LAST NOTE:
If you're interested in building a custom utility, I could help you code one up. It would be a good exercise, and others might find it useful.
Credit: this builds on #lifecrisis's answer, and extends it to handle large numbers of files:
find . -maxdepth 1 -type f -exec cat {} + | wc -l
find will find all of the files in the current directory, break them into groups as large as can be passed as arguments, and run cat on the groups.
awk 'END {print NR" total"}' ./*
Would be an interesting comparison to find out how many lines don't end with a new line.
Combining the awk and Gordon’s find solutions and avoiding the "." files.
find ./* -maxdepth 0 -type f -exec awk 'END {print NR}' {} +
No idea if this is better or worse but it does give a more accurate count (for me) and does not count lines in "." files. Using ./* is just a guess that appears to work.
Still need depth and ./* requires "0" depth.
I did get the same result with the "cat" and "awk" solutions (using the same find) since the "cat *" takes care of the new line issue. I don't have a directory with enough files to measure time. Interesting, I'm liking the "cat" solution.
This will give you the total count for all the files (including hidden files) in your current directory :
$ find . -maxdepth 1 -type f | xargs wc -l | grep total
1052 total
To count for files excluding hidden files use :
find . -maxdepth 1 -type f -not -path "*/\.*" | xargs wc -l | grep total
(Apologies for adding this as an answer—but I do not have enough reputation for commenting.)
A comment on #lifecrisis's answer. Perhaps cat is slowing things down a bit. We could replace cat by wc -l and then use awkto add the numbers. (This could be faster since much less data needs to go throught the pipe.)
That is
for file in *; do wc -l "$file"; done | awk '{sum += $1} END {print sum}'
instead of
for file in *; do cat "$file"; done | wc -l
(Disclaimer: I am not incorporating many of the improvements in other answers, but I thought the point was valid enough to write down.)
Here are my results for comparison (I ran the newer version first so that any cache effects would go against the newer candidate).
$ time for f in `seq 1 1500`; do head -c 5M </dev/urandom >myfile-$f |sed -e 's/\(................\)/\1\n/g'; done
real 0m50.360s
user 0m4.040s
sys 0m49.489s
$ time for file in myfile-*; do wc -l "$file"; done | awk '{sum += $1} END {print sum}'
30714902
real 0m3.455s
user 0m2.093s
sys 0m1.515s
$ time for file in myfile-*; do cat "$file"; done | wc -l
30714902
real 0m4.481s
user 0m2.544s
sys 0m4.312s
iF you want to know only total number Lines in directory excluding total line
ls -ltr | sed -n '/total/!p' | awk '{print NR}'
Previous comment will give total count of lines which includes only count of lines in all files
Below command will provide the total count of lines from all files in path
for i in `ls- ltr | awk ‘$1~”^-rw”{print $9}’`; do wc -l $I | awk ‘{print $1}’; done >>/var/tmp/filelinescount.txt
Cat /var/tmp/filelinescount.txt| sed -r “s/\s+//g”|tr “\n” “+”| sed “s:+$::g”| sed ’s/^/“/g’| sed ’s/$/“/g’ | awk ‘{print “echo” “ “ $0”+bc”}’| sh
I created a script called monitornsuaccounts.sh that should append its output file to useraccountstatus.log. useraccountstatus.log is in the directory /var/local/nsu/logs/.
The output of this script should state every username and the following information about each username: username, last login, user home directory and associated groups. Preferably there should be columns with each information.
The command I use for the usernames is sudo cat /etc/passwd | grep ‘/home’. Last is to find the last login of each user. Groups is to the find the group of each user. When I run the command, the output file only shows the data I need for my current user rather than all users. Any recommendations that anyone has would be greatly appreciated.
#!/bin/bash
usernames=sudo cat /etc/passwd | grep ‘/home’
echo “$usernames” > /home/daniel/names.txt
mlast=$(cat names.txt | xargs -n1 last)
mgroup=$(cat names.txt | xargs -n1 groups)
cat names.txt > /var/local/nsu/logs/useraccountstatus.log
echo “$mlast” >>/var/local/nsu/logs/useraccountstatus.log
echo “$mgroup” >>/var/local/nsu/logs/useraccountstatus.log
There are a lot of issues in your script.
Your definition of users. Are you sure that this is what you want? For example: root does not have a directory under /home.
Watch your quotes. cat /etc/passwd | grep ‘/home’ returns nothing, while cat /etc/passwd | grep 'home' returns a list of stanzas in /etc/passwd
You'll probably want just a list of usernames, not a list of stanzas. Something along the line of
cat /etc/passwd | grep 'home' | sed 's/:.*//'
Why sudo in sudo cat /etc/passwd?
Look at your assignment in the
usernames=sudo cat /etc/passwd | grep ‘/home’
This does not make sense. You might try to do a
usernames=`sudo cat /etc/passwd | grep '/home'| sed 's/:.*//'`
And that is just the first line of the script.
Anyway, if your script does not work as intended, you will need to do some debugging. First question, especially if you are inexperienced, is "do the commands that I write give the result that I expect?" So in your case, you should have tried cat /etc/passwd | grep ‘/home’ and you would have seen that it does not give you the expected results. Even with the correct quotes, you'll get a list of stanzas, which is also not what you expected. Have you looked at /home/daniel/names.txt and was the content of the file what you wanted? I guess not: it was empty.
Just a quick hint, to get you started in the right direction (although there are still some issues and pepole might object to the backtics)
#!/bin/bash
usernames=`sudo cat /etc/passwd | grep '/home'| sed 's/:.*//'`
mlast=`echo $usernames | xargs -n1 last`
mgroup=`echo $usernames| xargs -n1 groups`
echo $usernames > /var/local/nsu/logs/useraccountstatus.log
echo "$mlast" >>/var/local/nsu/logs/useraccountstatus.log
echo "$mgroup" >>/var/local/nsu/logs/useraccountstatus.log
You will want to polish this and make the output more useful.
I have a directory with over 100,000 files. I want to know if the string "str1" exists as part of the content of any of these files.
The command:
grep -l 'str1' * takes too long as it reads all of the files.
How can I ask grep to stop reading any further files if it finds a match? Any one-liner?
Note: I have tried grep -l 'str1' * | head but the command takes just as much time as the previous one.
Naming 100,000 filenames in your command args is going to cause a problem. It probably exceeds the size of a shell command-line.
But you don't have to name all the files if you use the recursive option with just the name of the directory the files are in (which is . if you want to search files in the current directory):
grep -l -r 'str1' . | head -1
Use grep -m 1 so that grep stops after finding the first match in a file. It is extremely efficient for large text files.
grep -m 1 str1 * /dev/null | head -1
If there is a single file, then /dev/null above ensures that grep does print out the file name in the output.
If you want to stop after finding the first match in any file:
for file in *; do
if grep -q -m 1 str1 "$file"; then
echo "$file"
break
fi
done
The for loop also saves you from the too many arguments issue when you have a directory with a large number of files.
I'm currently making a shell program and I want to display the total amount of bytes in a specific file using the pipe command. I know that the pipe command takes whatever is on the left side and gives it to the right as input. (Assuming you are in the directory the file is in)
I know that the command (wc -c) displays the number of bytes in a file but I'm not sure how to pipe it. What I've tried was:
ls fileName.sh | wc -c
wc takes the filename as argument, not as input. Try this:
wc -c fileName.sh
The wc program takes multiple arguments. You can do this to apply it to all entries in the current working directory:
wc -c $(ls)
Another approach is to use xargs to convert input to arguments:
ls | xargs wc -c
You may need to use a more complex line if you have spaces in your filenames. ls can output a single file per line, and xargs can be told to split only on \n:
ls -1 | xargs -d '\n' wc -c
If you prefer to use find instead of ls (a more powerful tool), the -print0 option for find plays along with the -0 option to xargs.
I have a few files in a directory with names similar to
_system1.log
_system2.log
_system3.log
other.log
but they are not created in that order.
Is there a simple, non-hardcoded, way to cat the files starting with the underscore in date order?
Quick 'n' dirty:
cat `ls -t _system*.log`
Safer:
ls -1t _system*.log | xargs -d'\n' cat
Use ls:
ls -1t | xargs cat
ls -1 | xargs cat
You can concatenate and also store them in a single file according to their time of creation and also you can specify the files which you want to concatenate. Here, I find it very useful. The following command will concatenate the files which are arranged according to their time of creaction and have common string 'xyz' in their file name and store all of them in outputfile.
cat $(ls -t |grep xyz)>outputfile