Since swiching over to a better management system I am wanting to remove all the redundant logs at the top of each of our source files. In Notepad++ I was able to achieve the result by using "replace in files" and replacing matches to \A(//.*\n)+ with blank. On Linux however I am having no such luck and am needing to resort to 'xargs' and 'sed'.
The sed expression I'm using is:
sed '1,/^[^\/]/{/^[^\/]/b; d}'
Ugly to be sure but it does seem to work.
The problem I'm having is when I try to run that through 'xargs' in order to feed it all the source files in our system I am unable to redirect the output to 'stripped' files, which I then intend to copy over the originals.
I want something in the line of:
find . -name "*.com" -type f -print0 | xargs -0 -I file sed '1,/^[^\/]/{/^[^\/]/b; d}' "file" > "file.stripped"
However I'm having grief passing the ">" through to the receiving environment (shell) as I'm already using too many quote marks. I have tried all manner of escaping and shell "wrappers" but I just can't get it to play ball.
Anyone care to point me in the right direction?
Thanks,
Slarti.
I made a similar scenario with a simpler sed expression just as an example, see if it works for you:
I created 3 files with the string "abcd" inside each:
# ls -l
total 12
-rw-r--r-- 1 root root 5 Oct 6 09:05 test.aaaaa.com
-rw-r--r-- 1 root root 5 Oct 6 09:05 test2.aaaaa.com
-rw-r--r-- 1 root root 5 Oct 6 09:05 test3.aaaaa.com
# cat test*
abcd
abcd
abcd
Running the find command as you showed using the -exec option instead of xargs, and replacing the sed expression for a silly one that simply replaces every "a" for "b" and the option -i, that writes directly do the input file:
# find . -name "*.com" -type f -print0 -exec sed -i 's/a/b/g' {} \;
./test2.aaaaa.com./test3.aaaaa.com./test.aaaaa.com
# cat test*
bbcd
bbcd
bbcd
In your case it should look like this:
# find . -name "*.com" -type f -print0 -exec sed -i '1,/^[^\/]/{/^[^\/]/b; d}' {} \;
Related
This question already has answers here:
Argument list too long error for rm, cp, mv commands
(31 answers)
Closed 5 years ago.
Below is the current days file. Previous days file converted to .gz by system. I wanted to find the total count of last days specific .gz files. I tried the below command which gives me the error. Please suggest
bash-3.2$ ls -lrth|tail
299K Mar 23 2017 N08170323091903766
333K Mar 23 2017 N08170323091903771
328K Mar 23 2017 N09170323091903776
367K Mar 23 2017 N09170323091903782
347K Mar 23 2017 N04170323092003784
368K Mar 23 2017 N08170323092003783
***bash-3.2$ ls -lrth N08170322*|wc -l***
bash: /usr/bin/ls: Arg list too long
0
***bash-3.2$ zcat N08170322*.gz|wc -l***
bash: /usr/bin/zcat: Arg list too long
0
This is happening because you have too many files in the directory.
You can easily get around the first issue:
ls | grep -c N08170322
or, to be even more precise:
ls | grep -c '^N08170322'
would give you the list of files. However, a better way to do this is:
find . -name "N08170322*" -exec ls {} + | wc -l
which will address the ls parsing issue mentioned in #hek2mgl's comment.
If you really want to count the lines of all the zipped files in one shot, you can do this:
find . -name "N08170322*" -exec zcat {} + | wc -l
See also:
Argument list too long error for rm, cp, mv commands
Use this
find . -name "N08170322*" -exec ls {} \; |wc -l
As explained in the below answer, you are getting argument list too long as there are multiple files in the directory. To overcome it, you can club it with find and exec
Edit: Created use case to check if command works with/without ls
These are the 3 empty files I created.
$ find -name "file*" -exec ls {} \;
./file1
./file2
./file3
Running wc -l without ls, prints number of lines in each file.
$ find -name "file*" -exec wc -l {} \;
0 ./file1
0 ./file2
0 ./file3
Running it with ls gives me number count of number of files, which is what OP wants.
$ find -name "file*" -exec ls {} \; | wc -l
3
I must parse ls -Al output and get file or directory name
ls -Al output :
drwxr-xr-x 12 s162103 studs 12 march 28 2012 personal domain
drwxr-xr-x 2 s162103 studs 3 march 28 22:32 public_html
drwxr-xr-x 7 s162103 studs 8 march 28 13:59 WebApplication1
I should use only ls -Al | <something>
for example:
ls -Al | awk '{print $8}'
but this doesn't work because $8 is not name if there's spaces in directory name,it is a part of name. maybe there's some utilities that cut last name or delete anything before? I need to find any solution. Please, help!
EDITED: I know what parse ls -Al is bad idea but I should exactly parse it with construction above! No way to use some thing like this
for f in *; do
somecommand "$f"
done
Don't parse ls -Al, if all you need is the file name.
You can put all file names in an array:
files=( * )
or you can iterate over the files directly:
for f in *; do
echo "$f"
done
If there is something specific from ls that you need, update your question to specify what you need.
How about thisls -Al |awk '{$1=$2=$3=$4=$5=$6=$7=$8="";print $0}'
I know it's a cheap trick but since you don't want to use anything other than ls -Al I cant think anything better...
Based on #squiguy request on comments, I post my comment as an answer:
What about just this?
ls -1A
instead of l (L, the letter), a 1 (one, the number). It will only list the names of the files.
It's also worth noting that find can do what you're looking for:
Everything in this directory, equivalent to ls:
find . -maxdepth 1
Recursively, similar to ls -R:
find .
Only directories in a given directory :
find /path/to/some/dir -maxdepth 1 -type d
md5sum every regular file :
find . -type f -exec md5sum {} \;
Hope awk works for you:
ls -Al | awk 'NR>1{for(i=9;i<NF;i++)printf $i" ";print $i}'
In case you're interested in sed:
ls -Al | sed '1d;s/^\([^ ]* *\)\{8\}//'
In this question, and a hundred other places, there are mostly identical Linux solutions for deleting all .svn directories. It works beautifully until the paths happen to include blanks. So, is there a technique to recursively remove .svn files in directories that contain blanks? Perhaps a way to tell find to wrap its answers in quotes?
you can tell find to use null as an output delimiter instead of newline with the -print0 action.
then you can tell xargs to use null as an input delimiter with the -0 argument.
example:
find . -name '*.svn' -print0 | xargs -0 -I{} rm \'{}\'
the -I{} argument to xargs tells it to replace {} with the current line from standard input. and personally, i like to include the backslash escaped quotes around the filenames as well, just to be doubly sure.
find . -name '*.svn' | while read x; do rm -r "$x" ; done
Yes, wrapping in quotes seems to do the trick.
mkdir "x y"
mkdir x\ y/.svn
find . -name '.svn' | awk '{print "rm -rf \""$0"\""}' | bash
And finally:
ls x\ y
total 8
drwxrwxr-x 6 dylan dylan 4096 Dec 13 06:09 ..
drwxrwxr-x 2 dylan dylan 4096 Dec 13 06:11 .
find . -type d -name '.svn' -delete
Current versions of GNU find have gained the "-delete" action.
If what you need is a clean copy of your repository, have you envisaged the use of the SVN export command?
You will then get a copy of all the directories present in your repository, including the ones with spaces in their name, but without any .svn folder.
At the moment our backup script explicitly runs svnadmin hotcopy on each of our repositories every night. Our repos are all stored under a parent directory (/usr/local/svn/repos)
Our backup script has a line for each of the repos under that directory along the lines of:
svnadmin hotcopy /usr/local/svn/repos/myrepo1 /usr/local/backup/myrepo1
Instead of having to manually add a new line for each every new repo we bring online, I was hoping to using the find command to run svnadmin hotcopy for every directory it finds under /usr/local/svn/repos.
So far I've got:
find /usr/local/svn/repos/ -maxdepth 1 -mindepth 1 -type d -exec echo /usr/local/backup{} \;
,where I'm substituting "svnadmin hotcopy" with "echo" for simplicity's sake.
The output of which is:
/usr/local/backup/usr/local/svn/repos/ure
/usr/local/backup/usr/local/svn/repos/cheetah
/usr/local/backup/usr/local/svn/repos/casemgt
/usr/local/backup/usr/local/svn/repos/royalliver
/usr/local/backup/usr/local/svn/repos/ure_andras
/usr/local/backup/usr/local/svn/repos/profserv
/usr/local/backup/usr/local/svn/repos/frontoffice
/usr/local/backup/usr/local/svn/repos/ure.orig
/usr/local/backup/usr/local/svn/repos/projectcommon
/usr/local/backup/usr/local/svn/repos/playground
/usr/local/backup/usr/local/svn/repos/casegen
The problem being the full path is included in {}. I need only the last element of the directory name passed to -exec
The output I want being:
/usr/local/backup/ure
/usr/local/backup/cheetah
/usr/local/backup/casemgt
/usr/local/backup/royalliver
/usr/local/backup/ure_andras
/usr/local/backup/profserv
/usr/local/backup/frontoffice
/usr/local/backup/ure.orig
/usr/local/backup/projectcommon
/usr/local/backup/playground
/usr/local/backup/casegen
I'm pretty much stuck at this point. Can anyone help me out here?
Thanks in advance,
Dave
You were on the right track. Try this:
find /usr/local/svn/repos/ -maxdepth 1 -mindepth 1 -type d -printf "%f\0" | xargs -0 -I{} echo svnadmin hotcopy /usr/local/svn/repos/\{\} /usr/local/backup/\{\}
The %f is like basename and the null plus the -0 on xargs ensures that names with spaces, etc., get passed through successfully.
Just remove the echo and make any adjustments you might need and it should do the trick.
put a cut command at the end
find /usr/local/svn/repos/ -maxdepth 1 -mindepth 1 -type d -exec echo /usr/local/backup{} \| cut -f1,2,3,9 -d"/"
How about adding a sed filter cuting out the middle part?
sed 's/usr.local.svn.repos.//g'
Added like this
find /usr/local/svn/repos/ -maxdepth 1 -mindepth 1 -type d -exec echo /usr/local/backup{} ";" | sed 's/usr.local.svn.repos.//g'
ls -al /usr/local/svn/repos/ |grep '^d' |sed s/^...............................................................//" |xargs -L 1 -I zzyggy echo /usr/local/svn/repos/zzyggy
It's a bit long but it does the trick. You don't have to do everything with find when there are lots of other shell commands, although if I had to write this kind of script, I would do it in Python and leave the shell for interactive work.
ls -al lists all the files in the named directory with attributes
grep '^d' selects the lines beginning with d which are directories
sed strips off all the characters to the left of the actual directory name. You may need to add or delete some dots
xargs takes the list of directory names and issues it one at a time. I specified zzyggy as the name to substitute in the executed command, but you can choose what you like. Of course, you would replace echo with your svnadmin command.
If it was in a shell script you should really do this
SVNDIRNAME="/usr/local/svn/repos"
ls -al $SVNDIRNAME |grep '^d' |sed s/^...............................................................//" |xargs -L 1 -I zzyggy echo $SVNDIRNAME/zzyggy
but I decided to show the wrong and right way just to explain this point. I'm going to tag this with some shell tag, but I still think that a Python script is a superior way to solve this kind of problem in the 21st century.
With bash on linux, how would I write a command to recursively traverse shares mounted, and run commands on each file, to get the file type and size, permissions etc, and then output all of this to a file?
A CIFS share mount would look like a regular directory tree in the linux shell.
The command to search as you need is therefore generic.
From the base directory,
find . -type f -exec ls -lsrt {} \; > file.txt
Ok, this does not give you the file-type detail;
that can be done with a -exec file filename on each file.
mount -v | grep smbfs | awk '{print $3}' | xargs ls -lsR
which you can redirect to a file.
mount -v | awk '/smbfs/{
cmd="ls -lsR "$3
while((cmd | getline d)>0){
print d "->file "$3
}
close(cmd)
}'
find $(mount -t smbfs | awk '{print $3}') -mount -type f -ls -execdir file {} \;
...
33597911 4 -rw-rw-r-- 2 peter peter 5 Dec 6 00:09 ./test.d\ ir/base
./base: ASCII text
3662 4 -rw-rw-r-- 2 peter peter 4 Dec 6 02:26 ./test.txt...
./test.txt...: ASCII text
3661 0 -rw-rw-r-- 2 peter peter 0 Dec 6 02:45 ./foo.txt
./foo.txt: empty
...
If you used -exec file {} +, it would run file once with multiple arguments, but then the output wouldn't be nicely interleaved with find's -ls output. (GNU find's -execdir {} + currently behaves the same as -execdir {} \;, due to a bug workaround. Use -exec file {} \; if you want the full path in the file output as well as in the -ls output above it.
find -ls output is not quite the same as ls -l, since it includes inode an number of blocks as the first two fields.