I have the following alias in my .aliases:
alias gi grep -i
and I want to look for foo case-insensitively in all the files that have the string bar in their name:
find -name \*bar\* | xargs gi foo
This is what I get:
xargs: gi: No such file or directory
Is there any way to use aliases in xargs, or do I have to use the full version:
find -name \*bar\* | xargs grep -i foo
Note: This is a simple example. Besides gi I have some pretty complicated aliases that I can't expand manually so easily.
Edit: I used tcsh, so please specify if an answer is shell-specific.
Aliases are shell-specific - in this case, most likely bash-specific. To execute an alias, you need to execute bash, but aliases are only loaded for interactive shells (more precisely, .bashrc will only be read for an interactive shell).
bash -i runs an interactive shell (and sources .bashrc).
bash -c cmd runs cmd.
Put them together:
bash -ic cmd runs cmd in an interactive shell, where cmd can be a bash function/alias defined in your .bashrc.
find -name \*bar\* | xargs bash -ic gi foo
should do what you want.
Edit: I see you've tagged the question as "tcsh", so the bash-specific solution is not applicable. With tcsh, you dont need the -i, as it appears to read .tcshrc unless you give -f.
Try this:
find -name \*bar\* | xargs tcsh -c gi foo
It worked for my basic testing.
This solution worked perfect for me in bash:
https://unix.stackexchange.com/a/244516/365245
Problem
[~]: alias grep='grep -i'
[~]: find -maxdepth 1 -name ".bashrc" | xargs grep name # grep alias not expanded
[~]: ### no matches found ###
Solution
[~]: alias xargs='xargs ' # create an xargs alias with trailing space
[~]: find -maxdepth 1 -name ".bashrc" | xargs grep name # grep alias gets expanded
# Name : .bashrc
Why it works
[~]: man alias
alias: alias [-p] [name[=value] ... ]
(snip)
A trailing space in VALUE causes the next word to be checked for
alias substitution when the alias is expanded.
Turn "gi" into a script instead
eg, in /home/$USER/bin/gi:
#!/bin/sh
exec /bin/grep -i "$#"
don't forget to mark the file executable.
The suggestion here is to avoid xargs and use a "while read" loop instead of xargs:
find -name \*bar\* | while read file; do gi foo "$file"; done
See the accepted answer in the link above for refinements to deal with spaces or newlines in filenames.
This is special-character safe:
find . -print0 | xargs -0 bash -ic 'gi foo "$#"' --
The -print0 and -0 use \0 or NUL-terminated strings so you don't get weird things happening when filenames have spaces in them.
bash sets the first argument after the command string as $0, so we pass it a dummy argument (--) so that the first file listed by find doesn't get consumed by $0.
For tcsh (which does not have functions), you could use:
gi foo `find -name "*bar*"`
For bash/ksh/sh, you can create a function in the shell.
function foobar
{
gi $1 `find . -type f -name "*"$2"*"`
}
foobar foo bar
Remember that using backquotes in the shell is more advantageous than using xargs from multiple perspectives. Place the function in your .bashrc.
Using Bash you may also specify the number of args being passed to your alias (or function) like so:
alias myFuncOrAlias='echo' # alias defined in your ~/.bashrc, ~/.profile, ...
echo arg1 arg2 | xargs -n 1 bash -cil 'myFuncOrAlias "$1"' arg0
(should work for tcsh in a similar way)
# alias definition in ~/.tcshrc
echo arg1 arg2 | xargs -n 1 tcsh -cim 'myFuncOrAlias "$1"' arg0 # untested
The simplest solution in you case would be to expand your alias inline. But that is valid for csh/tcsh only.
find -name \*bar\* | xargs `alias gi` foo
for bash it will be more tricky, not so handy but still might be useful:
find -name \*bar\* | xargs `alias gi | cut -d "'" -f2` foo
After trying many solutions with xargs that didn't work for me, went for an alternative with a loop, see examples below:
for file in $(git ls-files *.txt); do win2unix $file; done
for file in $(find . -name *.txt); do win2unix $file; done
Put your expression that generates a list of files inside $() as in the examples above. I've used win2unix which is a function in my .bashrc that takes a file path and converts it to Linux endings. Would expect aliases to also work.
Note that I did not have spaces in my paths or filenames.
Related
I understand that I have to use an array of arguments, but have no experience doing so. I am using Emacs for my shell scripting.
This is what I have so far:
#!/bin/bash
find $# -type f -exec file {} + | grep UTF-8
answer because I can't comment yet:
"$#" and "${name[#]}" should always used with surrounding double quotes. Otherwise words with spaces are broken. See "man bash" for details.
I don't understand why you want to use $# (every parameter), but I would solve your problem as follows:
#!/bin/bash
ARR=($(find . -type f -exec file {} + | grep script | sed -r 's/([^:]*).*/\1/'))
for i in ${ARR[#]}; do
if [ -x $i ]; then
echo "$i is an executable script"
fi
done
Find every file (including binaries), filter shell scripts with grep and take only take the file name with sed:
find . -type f -exec file {} + | grep script | sed -r 's/([^:]*).*/\1/')
You can loop over the array items by using the "#" index. There are several others more which might be useful in the future:
for i in ${ARR[#]}; do
#code
done
Finally, check if the script is executable with the -x option from [
if [ -x $i ]; then
#do something
fi
p.s. isn't vim better than emacs? ;-)
Just loop over the arguments with a for
#!/bin/bash
for f in $#; do find -name "$f" -type f -exec file {} \; | grep UTF-8
To recognize aliases in "xargs", I have set an alias
alias xargs="xargs bash -ic"
If I now execute the below snippet, no arguments are passed to the command to xargs.
find . -name pom.xml | xargs grep projectid
Infact, no arguments are passed to the command even in this case.
bash -ic grep projectid pom.xml
The documentation for bash says
-c If the -c option is present, then commands are read from the first non-option
argument command_string. If there are arguments after the command_string, they
are assigned to the positional parameters, starting with $0.
So what am I doing wrong?
bash --version
GNU bash, version 4.3.39(2)-release (x86_64-unknown-cygwin)
UPDATE:
Thanks to #knittl for his inputs. A work around solution for now to avoid all the extra punctuations in #knittl's answer.
1. Download xargs_bash_alias.sh
2. Set an alias
alias xargs="<path>/xargs_bash_alias.sh"
Now your xargs commands would recognize your other bash aliases.
There are two things you need to be aware of. First, proper quoting:
find . -name pom.xml -print0 | xargs -0 bash -c "grep projectid"
Second, you need to pass your positional arguments somehow:
find . -name pom.xml -print0 | xargs -0 bash -c 'grep projectid "$#"' -
Use - as first argument to bash, so positional arguments start at $1, just like in a normal shellscript.
"$#" expands to quoted positional arguments starting from 1.
Since xargs passes multiple arguments at once, you either need to use "$#" (quoted!) inside your bash script or run xargs with the -n1 option.
I am trying to manipulate the filename from the find command:
find . -name "*.xib" -exec echo '{}' ';'
For example, this might print:
./Views/Help/VCHelp.xib
I would like to make it:
./Views/Help/VCHelp.strings
What I tried:
find . -name "*.xib" -exec echo ${'{}'%.*} ';'
But, the '{}' is not being recognized as a string or something...
I also tried the following:
find . -name "*.xib" -exec filename='{}' ";" -exec echo ${filename%.*} ";"
But it is trying to execute a command called "filename" instead of assigning the variable:
find: filename: No such file or directory
You can't use Parameter Expansion with literal string. Try to store it in a variable first:
find . -name '*.xib' -exec bash -c "f='{}' ; echo \${f%.xib}.strings" \;
-exec sees first argument after it as the command, therefore you can't simply give it filename='{}' because find doesn't use sh to execute what you give it. If you want to run some shell stuff, you need to use sh or bash to wrap up.
Or use sed:
find . -name '*.xib' | sed 's/.xlib$/.strings/'
For such a simple search, you can use a pure bash solution:
#!/bin/bash
shopt -s globstar
shopt -s nullglob
found=( **.xib )
for f in "${found[#]}"; do
echo "${f%xib}strings"
done
Turning the globstar shell option on enables the ** to "match all files and zero or more directories and subdirectories" (as quoted from the bash reference manual). The nullglob option helps if there's no match: in this case, the glob will be expanded to nothing instead of the ugly **.xib. This solution is safe regarding filenames containing funny characters.
find . -name "*.xib" | sed -e 's/\.xib$/.strings/'
I'm trying to use GNU parallel to post a lot of files to a web server. In my directory, I have some files:
file1.xml
file2.xml
and I have a shell script that looks like this:
#! /usr/bin/env bash
CMD="curl -X POST -d#$1 http://server/path"
eval $CMD
There's some other stuff in the script, but this was the simplest example. I tried to execute the following command:
ls | parallel -j2 script.sh {}
Which is what the GNU parallel pages show as the "normal" way to operate on files in a directory. This seems to pass the name of the file into my script, but curl complains that it can't load the data file passed in. However, if I do:
find . -name '*.xml' | parallel -j2 script.sh {}
it works fine. Is there a difference between how ls and find are passing arguments to my script? Or do I need to do something additional in that script?
GNU parallel is a variant of xargs. They both have very similar interfaces, and if you're looking for help on parallel, you may have more luck looking up information about xargs.
That being said, the way they both operate is fairly simple. With their default behavior, both programs read input from STDIN, then break the input up into tokens based on whitespace. Each of these tokens is then passed to a provided program as an argument. The default for xargs is to pass as many tokens as possible to the program, and then start a new process when the limit is hit. I'm not sure how the default for parallel works.
Here is an example:
> echo "foo bar \
baz" | xargs echo
foo bar baz
There are some problems with the default behavior, so it is common to see several variations.
The first issue is that because whitespace is used to tokenize, any files with white space in them will cause parallel and xargs to break. One solution is to tokenize around the NULL character instead. find even provides an option to make this easy to do:
> echo "Success!" > bad\ filename
> find . "bad\ filename" -print0 | xargs -0 cat
Success!
The -print0 option tells find to seperate files with the NULL character instead of whitespace.
The -0 option tells xargs to use the NULL character to tokenize each argument.
Note that parallel is a little better than xargs in that its default behavior is the tokenize around only newlines, so there is less of a need to change the default behavior.
Another common issue is that you may want to control how the arguments are passed to xargs or parallel. If you need to have a specific placement of the arguments passed to the program, you can use {} to specify where the argument is to be placed.
> mkdir new_dir
> find -name *.xml | xargs mv {} new_dir
This will move all files in the current directory and subdirectories into the new_dir directory. It actually breaks down into the following:
> find -name *.xml | xargs echo mv {} new_dir
> mv foo.xml new_dir
> mv bar.xml new_dir
> mv baz.xml new_dir
So taking into consideration how xargs and parallel work, you should hopefully be able to see the issue with your command. find . -name '*.xml' will generate a list of xml files to be passed to the script.sh program.
> find . -name '*.xml' | parallel -j2 echo script.sh {}
> script.sh foo.xml
> script.sh bar.xml
> script.sh baz.xml
However, ls | parallel -j2 script.sh {} will generate a list of ALL files in the current directory to be passed to the script.sh program.
> ls | parallel -j2 echo script.sh {}
> script.sh some_directory
> script.sh some_file
> script.sh foo.xml
> ...
A more correct variant on the ls version would be as follows:
> ls *.xml | parallel -j2 script.sh {}
However, and important difference between this and the find version is that find will search through all subdirectories for files, while ls will only search the current directory. The equivalent find version of the above ls command would be as follows:
> find -maxdepth 1 -name '*.xml'
This will only search the current directory.
Since it works with find you probably want to see what command GNU Parallel is running (using -v or --dryrun) and then try to run the failing commands manually.
ls *.xml | parallel --dryrun -j2 script.sh
find -maxdepth 1 -name '*.xml' | parallel --dryrun -j2 script.sh
I have not used parallel but there is a different between ls & find . -name '*.xml'. ls will list all the files and directories where as find . -name '*.xml' will list only the files (and directories) which end with a .xml.
As suggested by Paul Rubel, just print the value of $1 in your script to check this. Additionally you may want to consider filtering the input to files only in find with the -type f option.
Hope this helps!
Neat.
I had never used parallel before. It appears, though that there are two of them.
One is the Gnu Parrallel, and the one that was installed on my system has Tollef Fog Heen
listed as the author in the man pages.
As Paul mentioned, you should use
set -x
Also, the paradigm that you mentioned above doesn't seem to work on my parallel, rather, I have
to do the following:
$ cat ../script.sh
+ cat ../script.sh
#!/bin/bash
echo $#
$ parallel -ij2 ../script.sh {} -- $(find -name '*.xml')
++ find -name '*.xml'
+ parallel -ij2 ../script.sh '{}' -- ./b.xml ./c.xml ./a.xml ./d.xml ./e.xml
./c.xml
./b.xml
./d.xml
./a.xml
./e.xml
$ parallel -ij2 ../script.sh {} -- $(ls *.xml)
++ ls --color=auto a.xml b.xml c.xml d.xml e.xml
+ parallel -ij2 ../script.sh '{}' -- a.xml b.xml c.xml d.xml e.xml
b.xml
a.xml
d.xml
c.xml
e.xml
find does provide a different input, It prepends the relative path to the name.
Maybe that is what is messing up your script?
I was trying to execute a script n times with a different file as argument each time using this:
ls /user/local/*.log | xargs script.pl
(script.pl accepts file name as argument)
But the script is executed only once. How to resolve this ? Am I not doing it correctly ?
ls /user/local/*.log | xargs -rn1 script.pl
I guess your script only expects one parameter, you need to tell xargs about that.
Passing -r helps if the input list would be empty
Note that something like the following is, in general, better:
find /user/local/ -maxdepth 1 -type f -name '*.log' -print0 |
xargs -0rn1 script.pl
It will handle quoting and directories safer.
To see what xargs actually executes use the -t flag.
I hope this helps:
for i in $(ls /usr/local/*.log)
do
script.pl $i
done
I would avoid using ls as this is often an alias on many systems. E.g. if your ls produces colored output, then this will not work:
for i in ls; do ls $i;done
Rather, you can just let the shell expand the wildcard for you:
for i in /usr/local/*.log; do script.pl $i; done