If I run this command it works fine in the terminal:
for dirname in $(ls -d dir/checkpoint/features.txt/20*);do;echo "hello";done
But when run through /bin/sh -c it gives an error
/bin/sh -c "for dirname in $(ls -d dir/checkpoint/features.txt/20*);do;echo "hello";done"
ERROR:
/bin/sh: -c: line 1: syntax error near unexpected token `dir/checkpoint/features.txt/201108000'
/bin/sh: -c: line 1: `dir/checkpoint/features.txt/201108000'
My default shell is /bin/bash. I cant seem to understand what is causing this. My default implementation for running all shell commands in my program is by appending /bin/sh -c to them. It is the first time i am seeing this issue. Any suggestions?
Don't try to parse the output of ls, especially with a for construct. There are many, many ways that this can go wrong.
This is a good place to use find instead. Try this:
/bin/sh -c "find dir/checkpoint/features.txt -mindepth 1 -maxdepth 1 -type d -iname '20*' -exec echo \"hello\" \;"
Besides eliminating the error-prone use of ls, you avoid the sub-shell and all of the issues that it brings with it.
Follow-up in response to your comment:
I'm assuming that you're using awk -F/ '{print $NF}' to grab the name of the folder in which the file lives (that is, the last directory name before the filename). The commands basename and dirname can be used to do this for you. This should make your script a bit easier. Place the following into a script file:
#!/bin/sh
folder=$(basename $(dirname $1))
mkdir -p #{nfs_checkpoint}/${folder}
cat #{result_location}/${folder}/20* > #{nfs_checkpoint}/${folder}/features.txt
And execute it like this:
/bin/sh -c "find dir/checkpoint/features.txt -mindepth 1 -maxdepth 1 -type d -iname '20*' -exec yourscript.sh {} \;"
Related
I'm trying to create a script which will find missing topics from multiple log files. These logfiles are filled top down, so the newest logs are at the bottom of the file. I would like to grep only the last line from this file which includes UNKNOWN_TOPIC_OR_PARTITION. This should be done in multiple files with completely different names. Is grep the best solution or is there another solution that suits my needs. I already tried adding tail, but that doesn't seem to work.
missingTopics=$(grep -Ri -m1 --exclude=*.{1,2,3,4,5} UNKNOWN_TOPIC_OR_PARTITION /app/tibco/log/tra/domain/)
You could try a combination of find, tac and grep:
find /app/tibco/log/tra/domain -type f ! -name '*.[1-5]' -exec sh -c \
'tac "$1" | grep -im1 UNKNOWN_TOPIC_OR_PARTITION' "sh" '{}' \;
tac prints files in reverse, the -exec sh -c SCRIPT "sh" '{}' \; action of find executes the shell SCRIPT each time a file matching the previous tests is found. The SCRIPT is executed with "sh" as parameter $0 and the path of the found file as parameter $1.
If performance is an issue you can probably improve it with:
find . -type f ! -name '*.[1-5]' -exec sh -c 'for f in "$#"; do \
tac "$f" | grep -im1 UNKNOWN_TOPIC_OR_PARTITION; done' "sh" '{}' +
which will spawn less shells. If security is also an issue you can also replace -exec by -execdir (even if with this SCRIPT I do not immediately see any exploit).
I'm writing a shell script to run on a docker image based on Alpine. It's shell is /bin/sh.
What I'm trying to do is execute a function for the results of a find command. The following works in my local bash and sh shells.
myscript.sh:
#!/bin/sh
function get_tags {
# do stuff
}
export -f get_tags
# get all YAML files in ./assets/config that have 'FIND' somewhere in the filename
# pass each to the get_tags function
find ./assets/config -type f \( -iname "Find*.yaml" -or -iname "Find*.yml" \) -exec sh -c 'get_tags "$0"' {} \;
When I run it on the alpine image, however, I get the following error:
./myscript.sh: export: line 31: illegal option -f
Is there another way I can do this?
My question is NOT "what is the difference between sh and bash". My question is: how do I accomplish the task of running a function on the output of the find command.
You need to use bash, like this:
#!/bin/bash
fun() { echo "fun ${1}" ; }
export -f fun
find . -name 'foo' -exec bash -c 'fun "${1}"' -- {} \;
The key here is to run bash -c 'fun "${1}"' -- {} \;. You can't call the function directly (and pass arguments to it). You need to wrap it into a minimal script where this minimal script receives the argument passed by find and passes it through to the function.
Note: I'm passing two arguments to bash -c: the string -- and the actual filename {}. I'm doing this by convention, because argument counting starts at $0 when a script is executed by bash -c, in opposite to $1 when running a script the normal way (in a file, not via bash -c)
bash -c 'fun "${0}"' {} \; would work, but people might think $0 is the script name like they know it from normal scripts.
Exporting functions is a Bash feature. Alpine Linux does not come with Bash.
You can instead use a while read loop to process the results, as this is POSIX and will work on all shells:
get_tags() {
echo "Getting tags for $1"
}
find ./assets/config -type f \( -iname "Find*.yaml" -o -iname "Find*.yml" \) |
while IFS="" read -r file
do
get_tags "$file"
done
I have the following bash script:
#!/bin/bash
find . -maxdepth 1 -mmin +1 -type f -name "240*.ts"
| xargs -L 1 bash -c 'mv "${1}" "$(get_crtime${1} | awk '{print $5}').ts"' \;
The idea is to find files that are older than one minute matching a certain pattern (in my case, files that start with '240') and rename them from their original name (240-1458910816045.ts) to a desired format (15:00:16.ts).
Inside the script I am using get_crtime command which is a custom function included in /etc/bash.bashrc and has the following implementation:
get_crtime() {
for target in "${#}"; do
inode=$(stat -c '%i' "${target}")
fs=$(df "${#}" | awk '{a=$1}END{print a}')
crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null |
grep -oP 'crtime.*--\s*\K.*')
printf "%s\t%s\n" "${target}" "${crtime}"
done
}
When I call the function from the shell, like this:
get_crtime 240-1458910816045.ts | awk '{print $5}'
I get the desired output:
15:00:16
Which is a portion from the file creation date.
My problem is when I include the function call inside my initial script I get the following error:
}).ts": -c: line 0: unexpected EOF while looking for matching `)'
}).ts": -c: line 1: syntax error: unexpected end of file
I think this is caused by incorrect invoking of awk, so I thought to remove it and leave just:
find . -maxdepth 1 -mmin +1 -type f -name "240*.ts"
| xargs -L 1 bash -c 'mv "${1}" "$(get_crtime ${1}).ts"' \;
I get the following error, which is more suggestive:
;: get_crtime: command not found
How can I call the custom function inside the bashrc inside the initial command without getting the last error?
Thank you!
The OS is Ubuntu
The shell is bash
You can't use single quotes inside a single-quote delimited script. Look:
$ bash -c 'printf "%s\n" "$(date | awk '{print $0}')"'
-bash})": -c: line 0: unexpected EOF while looking for matching `)'
-bash})": -c: line 1: syntax error: unexpected end of file
$ bash -c 'printf "%s\n" "$(date | awk "{print \$0}")"'
Fri, Mar 25, 2016 8:59:31 AM
I'm not recommending you use double quotes around your awk script though - create a script to do the mv, etc. for you or figure out some other way to implement it that'll solve your function access problem too.
In this example, used modification time of file, which can be get by stat -c '%y'. The xargs -I param creates possibility to place file name two times, first for stat, second for mv. Then using Parameter Expansion bash features to extract only time from human readable stat output:
find . -maxdepth 1 -mmin +1 -type f -name "240*.ts" | \
xargs -I_ bash -c 'MTIME=$(stat -c '%y' "_") && MTIME=${MTIME#* } && mv "_" ${MTIME%.*}.ts'
You need to export the function:
export -f get_crtime
That will make it available to child bash processes (but not to other shells).
Also, as #EdMorton points out, you cannot use single quotes inside a single quoted-string, which was the problem with the invocation of awk. So you'll need to come up with a different way of quoting the interior argument to awk, or fix get_crtime to just return the string you want.
By the way, you might consider using finds -exec action instead of xargs. That would allow you to use a loop over a number of files, which would be a bit more efficient.
eg.
find . -maxdepth 1 -mmin +1 -type f -name "240*.ts" \
-exec bash -c 'for f in "$#"; do
mv "$f" "$(get_crtime "$f" | awk {print\$5}).ts"
done' _ {} +
I've been working with find -exec and find | xargs for the past few hours exploring and experimenting, and now I've found a variation of the command that I haven't seen anywhere else.
For example this find command to get all files in the child subdirectories and copy them to the current directory
find . -type f -regex './[^\.].*/[^.*].*' -exec sh -c 'cp "$#" .' thiscanbeanything {} +
will all execute on one line like so:
cp ./testy/bar ./testy/baz ./testy/foo .
Instead of the usual:
find . -type f -regex './[^\.].*/[^.*].*' -exec sh -c 'cp {} .' \;
which executes on multiple lines
cp ./testy/bar .
cp ./testy/baz .
cp ./testy/foo .
Moreover in the first command the output will be only:
cp ./testy/baz ./testy/foo .
Unless the sh -c 'cmd' is followed by something else, which in my example was thiscanbeanything.
Could someone elucidate what's going on, or if this is even viable?
To understand what is going on, have a look at this:
$ sh -c 'echo 0=$0 1=$1 2=$2' thiscanbeanything one two
0=thiscanbeanything 1=one 2=two
This executes sh with the option -c 'echo 0=$0 1=$1 2=$2' and three arguments thiscanbeanything one two.
Normally, $0 is the name of the script being executed. When running sh -c, there is no script but the name is taken from the first argument that your provide which, in this case, is thiscanbeanything.
Documentation
This behavior is documented in man bash under the -c option:
-c string If the -c option is present, then commands are read from string. If there are arguments after the string, they are
assigned to the positional parameters, starting with $0.
Good day,
I've found an easy way to find files that have certain content, but I would like to create a bash script to do it quickier,
The script is:
#!/bin/bash
DIRECTORY=$(cd `dirname .` && pwd)
ARGUMENTS="'$#'"
echo find: $ARGUMENTS on $DIRECTORY
find $DIRECTORY -iname '*' | xargs grep $ARGUMENTS -sl
So if I write:
$ script.sh text
It should find in that directory files that contains 'text'
But when I execute this script it always fails, but the echo command shows exactly what I need, what's wrong with this script?
Thank you!
Luis
References: http://www.liamdelahunty.com/tips/linux_find_string_files.php
There are problems with quoting that will break in this script if either the current directory or the search pattern contains a space. The following is more simply, and fixes both issues:
find . -maxdepth 1 -type f -exec grep "$#" {} +
With the proper quoting of $#, you can even pass options to grep, such as -i.
./script -i "some text"
Try this version, with the following changes:
1.Use $1 instead of $# unless you intend to run multiple find/grep to search for multiple patterns.
2.Use find $DIR -type f to find all files instead of find $DIR -iname '*'
3.Avoid piping by using the -exec command line option of find.
4.Do not single quote the command line arguments to your script, this was the main problem with the version you had. Your grep string had escaped single quotes \'search_string\'
#!/bin/bash
DIRECTORY=$(cd `dirname .` && pwd)
ARGUMENTS="$1"
echo find: $ARGUMENTS on $DIRECTORY
find $DIRECTORY . -type f -exec grep -sl "$ARGUMENTS" {} \;
There is no point extracting all the command line arguments and passing it to grep. If you want to search for a string with spaces, pass the string within single quotes from the command line as follows:
/home/user/bin/test-find.sh 'i need to search this'
Why not just run the following?:
grep -R text .