From a central Linux box (RHEL 6.3) I'm attempting to push a set of zip files to a series of other Linux hosts and then unzip those files on each target host. My central host is hpdb1.
#Push zip files to other hosts
for i in {2..8}; do
scp -r /software/stage/*.zip root#hpdb$i:/software/stage
done
#Unzip files to /software/stage
for i in {2..8}; do
ssh hpdb$i "for f in /software/stage/*.zip; do unzip /software/stage/"$f" done";
done
The first for loop to push the files works fine. However, when running the nested for loop I get the following error:
[root#hpdb1 ~]# for i in {2..8}; do ssh hpdb$i "for f in /software/stage/*.zip; do unzip /software/stage/"$f"; done"; done
unzip: cannot find or open /software/stage/, /software/stage/.zip or /software/stage/.ZIP.
unzip: cannot find or open /software/stage/, /software/stage/.zip or /software/stage/.ZIP.
Looks like the $f variable is not getting interpreted. Any ideas?
Updated for answer
This code works.
for i in {2..7}; do
ssh hpdb$i 'for f in /software/stage/*.zip; do unzip "$f" -d /software/stage; done';
done
The problem may be the nested double-quotes. You probably want the outer quotes to be single-quotes so that the embedded $s are not expanded before getting sent to the remote server.
My first though is to use the other quote character like, so:
for i in {2..8}; do ssh hpdb$i 'for f in /software/stage/*.zip; do unzip /software/stage/â$fâ; done'; done
Although you could use different quotes as suggested by another answer, this will alter variable expansion behaviour and may be undesirable in some cases.
You can simply escape the enclosed quotes instead, by pre-pending them with a backslash:
for i in {2..8}; do
ssh hpdb$i "for f in /software/stage/*.zip; do unzip /software/stage/\"$f\"; done";
done
Related
This question already has answers here:
How can I use a variable that contains a space?
(2 answers)
Tilde expansion in quotes
(3 answers)
Closed 3 years ago.
I'm writing a bash script to copy files from my local machine to a remote server using rsync. There is a space in the folder name "local folder" on my local machine; I don't know if this makes any difference.
If I write it as plain text it works:
rsync -arv --filter=":- .gitignore" /local\ folder/path/ user#123.4.5.6:~/
I want to put the source and destination paths as variables, but I can't get it to work. The first thing I tried is this:
SOURCE="/local\ folder/path/"
DESTINATION="user#123.4.5.6:~/"
rsync -arv --filter=":- .gitignore" $SOURCE $DESTINATION
I see this error:
rsync: change_dir "/local folder/path//~" failed: No such file or directory (2)
It seems to be a) running source and destination together, and b) not seeing the server address.
I've tried a number of variations, including braces:
rsync -arv --filter=":- .gitignore" ${SOURCE} ${DESTINATION}
Using quotes:
rsync -arv --filter=":- .gitignore" "${SOURCE}" "${DESTINATION}"
and putting the options into an array:
OPTIONS=( --protect-args -arv --filter=":- .gitignore")
rsync "${OPTIONS[#]}" ${SOURCE} ${DESTINATION}
I have also tried this after checking it in https://www.shellcheck.net/
#!/bin/bash
SOURCE="/folder name/path"
DESTINATION=user#123.4.5.6:~/
rsync -arv --filter=":- .gitignore" "$SOURCE" $DESTINATION
and also:
#!/bin/bash
SOURCE="/folder\ name/path"
DESTINATION=user#123.4.5.6:~/
rsync -arv --filter=":- .gitignore" "$SOURCE" $DESTINATION
Each time I get the same error. What simple thing am I missing here? I've looked at various examples including:
https://www.redpill-linpro.com/sysadvent/2015/12/03/rsync-tricks.html
https://serverfault.com/questions/354112/rsync-and-bash-command-substitution
The space isn't the issue, or at least not the only issue. I've tried using double quotes around my variable names as suggested here
Thanks!
You need to quote your Variables: Try
rsync -arv --filter=":- .gitignore" "$SOURCE" "$DESTINATION"
I have a folder with lots of files which name has the following structure:
01.artist_name - song_name.mp3
I want to go through all of them and rename them using the regexp:
/^d+\./
so i get only :
artist_name - song_name.mp3
How can i do this in bash?
You can do this in BASH:
for f in [0-9]*.mp3; do
mv "$f" "${f#*.}"
done
Use the Perl rename utility utility. It might be installed on your version of Linux or easy to find.
rename 's/^\d+\.//' -n *.mp3
With the -n flag, it will be a dry run, printing what would be renamed, without actually renaming. If the output looks good, drop the -n flag.
Use 'sed' bash command to do so:
for f in *.mp3;
do
new_name="$(echo $f | sed 's/[^.]*.//')"
mv $f $new_name
done
...in this case, regular expression [^.].* matches everything before first period of a string.
Some of my files are separated into different directories such as /apps, /games, /docs etc....
Within each directory, is a subdirectory called _CHECKSUM. Inside this directory, is a file called openssl.sh.
For example:
openssl sha1 /path/to/apps/*.iso | sed 's/\/.*.\///' > /path/to/apps/_CHECKSUM/sum.sha1
This outputs to a file called sum.sha1 within the _CHECKSUM directory, of which the contents could look like this:
SHA1(anApp.iso)= b398c8b175411e6174942d7b4acbc5c90473a852
SHA1(anotherApp.iso)= cc150483feed3d4b607749f31eddccefd0ba5478
SHA1(yetAnotherApp.iso)= d9682a2eca25b70dddf7a906374c27ee35614c7d
However, some directories contain multiple filetypes, so the script would have to look like this:
openssl sha1 /path/to/games/*.{7z,iso} | sed 's/\/.*.\///' > /path/to/games/_CHECKSUM/sum.sha1
producing something like this:
SHA1(myFaveGame.7z)= b398c8b175411e6174942d7b4acbc5c90473a852
SHA1(anotherGoodGame.iso)= cc150483feed3d4b607749f31eddccefd0ba5478
I don't want to always run these scripts manually, so I created the following script, /path/to/scripts/openssl_recursive.sh:
#!/bin/bash
# finds every openssl.sh recursively and executes it.
IFS=$'\n'
for file in $(find /path/to -name "openssl.sh" | sort -n)
do
echo "executing $file ..."
sh $file
echo "done.";
done
This seems to work fine for all directories where just one file type exists. However, for the openssl.sh scripts that contain multiple extensions, an empty sum.sha1 file is created.
Why is it that if I run the openssl.sh directly, it will create the correct result in sum.sha1 for directories with multiple filetypes, yet if I run the openssl_recursive.sh, this results in an empty sum.sha1?
as stated here, modern Debian and Ubuntu systems symlink sh to dash by default, which is a lighter version and lacks some advanced features.
So this may not be the same shell, and probably doesn't like "rich" wildcard constructs like *.{7z,iso}. You must have fallen into that category.
On the other hand, bash accepts those wildcards happily.
So a working solution is forcing the use of /bin/bash env variable:
#!/bin/bash
# finds every openssl.sh recursively and executes it.
IFS=$'\n'
for file in $(find /path/to -name "openssl.sh" | sort -n)
do
echo "executing $file ..."
/bin/bash $file
echo "done.";
done
I am having a hell of a time attempting to get a bash script to work as expected (as it does in a normal bash session) on a Docker run.
The goal is to replace all of the symlinked files within the official java container with their actual file within the JAVA_HOME directory, so everything is contained within the java directory and not outside of it,
e.g.
$JAVA_HOME/jre/lib/security/java.policy <--- is symlinked to ---> /etc/java-7-openjdk/security/java.policy
The end result should be the file located at: $JAVA_HOME/jre/lib/security/java.policy
The setup:
docker run java:7u91 /bin/bash -cxe "find /usr/lib/jvm/**/jre -type l | while read f; do echo $f; cp --remove-destination $(readlink $f) $f; done;"
I had attempted several different methods of effectively this, with xargs and exec all to no avail.
Any suggestions at this point would be appreciated.
It looks like this is what is happening: $(readlink $f) is not returning anything on the files that are not symbolic links (only works on symbolic links). Therefore, that expression is essentially nothing/empty.
So, only the $f is returning a value. Therefore, if the expression was evaluated, it would print cp --remove-destination VALUE_OF_$F;, and the $f would look like it was the first parameter of the cp command, with no second parameter present. That is why the 'destination' is missing.
Also, you need to consider the fact that putting your command inside of double quotes like that is presenting a problem. The variables will be parsed on the host rather than in the docker container. Replace the double quotes with single quotes to prevent that from happening.
Source directory is determined like so:
SHOW=${PWD##*/}
SRC=wells#server.com:"/mnt/bigfish/video/TV/${SHOW}/"
So it comes out something like:
wells#server.com:/mnt/bigfish/video/TV/The Name Of the Show With Spaces/
Then trying to run rsync like so:
rsync -avz -e ssh "${SRC}" .
But it tells me that ""/mnt/bigfish/video/TV/The" is not a directory, ""/mnt/bigfish/video/TV/Name" is not a directory, etc, for however many space-delimited words are in the name of the source directory.
How can I rectify this egregiously annoying issue?
UPDATE I'm running this on OS 10.6, and I ended up string-replacing spaces with escaped spaces like so:
SRC=wells#kittenfactory.com:"/mnt/bigfish/video/TV/${SHOW// /\ }/"
From the rsync manual:
-s, --protect-args
This option sends all filenames and most options to the
remote
rsync without allowing the remote shell to interpret them.
This
means that spaces are not split in names, and any
non-wildcard
special characters are not translated (such as ~, $, ;, &,
etc.). Wildcards are expanded on the remote host by
rsync
(instead of the shell doing it).
As your question is dedicated to OS X, according to the Apple rsync manual you can accomplish this using either simple quotes or the wildcard ?:
rsync -av host:'file\ name\ with\ spaces' /dest
rsync -av host:file?name?with?spaces /dest
Just had to do this and using the simple quotes works perfectly:
rsync -r --partial --progress --exclude=".cvs" --exclude=".svn" --exclude=".git" --rsh=ssh root#datakeep.local:'/volume1/devel/__To\ SORT/___XXXXX\ Saves\ 2011-04' ./Saves2011
This works:
rsync -avz -e ssh "wells#server.com:\"/mnt/bigfish/video/TV/${SHOW}/\""
So set:
SRC=wells#server.com:\"/mnt/bigfish/video/TV/${SHOW}/\"
At least, here on Debian it works like a charm, no OS 10 available to test with here.
You can do this on OSX if you're dealing with arguments in a script:
ESCAPED_SRC="$(echo "$SRC" | tr ' ' '\\ ')"
ESCAPED_DEST="$(echo "$DEST" | tr ' ' '\\ ')"
rsync -ravP "$ESCAPED_SRC" "$ESCAPED_DEST"