Suppose I have this shell script call cpdir:
(cd $1 ; tar -cf - . ) | (cd $2 ; tar -xvf - )
When I ran it, the main shell should create two processes (subshells) to execute both groups of commands concurrently. However, how can the shell make sure that both processes change to appropriate directories, then first process package the content of the directory and send to the second process for unpacking?
Why is there no race condition? Is it a rule that every command of every process will execute in order, although processes can be parallel?
i.e. first process will run "cd $1", and then second process will run "cd $2" (or it should be execute the same time as the first process? Not sure), then first process will package everything and finally send to second process.
Although, one little thing I don't know about tar:
tar -cf - .
I know the dot (.) is the content of current directory. However, what's the '-' in the command?
You don't need to use cd because tar has a -C option which tells it to change to a directory. So you can simply use a command such as:
tar -C $1 -cvf - . | tar -C $2 -xvf -
- means stdin/stdout. The first hyphen tells tar to write to stdout. The second one tells tar to read from stdin.
Since - is the default, you don't even need to specify it. You can shorten your command to:
tar -C $1 -c . | tar -C $2 -x
As those groups run in independent processes, it doesn't matter which cd command runs first: each process has its own working directory.
So changing working directory does not affect the respectively other process.
You are piping the commands in your case. The result you expect is not very clear to me.
By the way, my GNU tar has no "-" value for this "-f" option.
So you commands might be not portable.
Related
#!/usr/bin/env bash
# usage: wttr [location], e.g. wttr Berlin, wttr New\ York
# Standard location if no parameters were passed
location=''
language=''
time=`date`
# Expand terminal display
if [ -z "$language" ]; then
language=${LANG%_*}
fi
curl \
-H -x "Accept-Language: ${language}" \
-x wttr.in/"${1:-${location}}" |
head -n 7 |
tee /home/of/weather.txt |
tee -a /home/of/weather.log |
tee /home/of/BACKUP/weather_"$time".txt
#cp weather.txt /home/of/BACKUP
#mv -f /home/of/BACKUP/weather.txt /home/of/BACKUP/weather_"$time".txt
I'm very new to Linux Bash and Shell scripting and can't figure out the following.
I have a problem with the shell script above.
It works fine so far (curling ASCII data from website and writing it to weather.txt and .log).
It is also in set in crontab to run every 5 minutes.
Now I need to make a backup of weather.txt under /home/of/, in /home/of/BACKUP with the filename weather_<timestamp>.txt.
I tried to delete (rm weather*.txt) the old timestamped files in /home/of/BACKUP and then copy and rename the file everytime the cronjob is running.
I tried piping cp and mv and so on but somehow I end up with producing many duplicates as due to the timestamp the filenames are different or nothing at all when I try to delete the content of the folder first.
All I need is ONE backup file of weather.txt as weather_<timestamp>.txt which gets updated every 5 minutes with the actual timestamp bit I can't figure it out.
If I understand your question at all, then simply
rm -f /home/of/BACKUP/weather_*.txt
cp /home/of/weather.txt /home/of/BACKUP/weather_"$time".txt
cp lets you rename the file you are copying to; it doesn't make sense to separately cp and then mv.
For convenience, you might want to cd /home/of so you don't have to spell out the full paths, or put them in a variable.
dir=/home/of
rm -f "$dir"/BACKUP/weather_*.txt
cp "$dir"/weather.txt "$dir"/BACKUP/weather_"$time".txt
If you are running out of the cron of the user named of then your current working directory will be /home/of (though if you need to be able to run the script manually from anywhere, that cannot be guaranteed).
Obviously, make sure the wildcard doesn't match any files you actually want to keep.
As an aside, you can simplify the tee commands slightly. If this should only update the files and not print anything to the terminal, you could even go with
curl \
-H -x "Accept-Language: ${language}" \
-x wttr.in/"${1:-${location}}" |
head -n 7 |
tee /home/of/weather.txt \
>>/home/of/weather.log
I took out the tee to the backup file since you are deleting it immediately after anyway. You could alternatively empty the backup directory first, but then you will have no backups if the curl fails.
If you want to keep printing to the terminal, too, probably run the script with redirection to /dev/null in the cron job to avoid having your email inbox fill up with unread copies of the output.
I am trying to execute the scp command in such a way that it can copy .csv files from source to sink, except a few specific CSV file.
For example in the source folder I am having four files:
file1.csv, file2.csv, file3.csv, file4.csv
Out of those four files, I want to copy all files, except file4.csv, to the sink location.
When I was using the below scp command:
scp /tmp/source/*.csv /tmp/sink/
It would copy all the four CSV files to the sink location.
How can I achieve the same by using the scp command or through writing a shell script?
You can use rsync with the --exclude switch, e.g.
rsync /tmp/source/*.csv /tmp/sink/ --exclude file4.csv
Bash has an extended globbing feature which allows for this. On many installations, you have to separately enable this feature with
shopt -e extglob
With that in place, you can
scp tmp/source/(!fnord*).csv /tmp/sink/
to copy all *.csv files except fnord.csv.
This is a shell feature; the shell will expand the glob to a list of matching files - scp will have no idea how that argument list was generated.
As mentioned in your comment, rsync is not an option for you. The solution presented by tripleee works only if the source is on the client side. Here I present a solution using ssh and tar. tar does have the --exclude flag, which allows us to exclude patterns:
from server to client:
$ ssh user#server 'tar -cf - --exclude "file4.csv" /path/to/dir/*csv' \
| tar -xf - --transform='s#.*/##' -C /path/to/destination
This essentially creates a tar-ball which is send over /dev/stdout which we pipe into a tar extract. To mimick scp we need to remove the full path using --transform (See U&L). Optionally you can add the destination directory.
from client to server:
We do essentially the same, but reverse the roles:
$ tar -cf - --exclude "file4.csv" /path/to/dir/*csv \
| ssh user#server 'tar -xf - --transform="s#.*/##" -C /path/to/destination'
You could use a bash array to collect your larger set, then remove the items you don't want. For example:
files=( /tmp/src/*.csv )
for i in "${!files[#]}"; do
[[ ${files[$i]} = *file4.csv ]] && unset files[$i]
done
scp "${files[#]}" host:/tmp/sink/
Note that our for loop steps through array indices rather than values, so that we'll have the right input for the unset command if we need it.
I am currently working on a Text-to-speech project and I need to write bash script which will, when it is called, execute two commands. If the first command returns the proper answer (if returns an answer at all), the second command will be called and executed.
My question is, how can I write a script, that executes shell commands in a specific certain file system location?
For example, I need to be in the directory /opt/text/example and execute this command:
sudo ./bin/sample_read -I ../languages/ -I ../languages -v dave -T 2 \
-i /opt/text/example.txt -F 22 -O embedded-pro -o out_file.pcm
and then to wait for the answer, then (if it is good) execute the second command.
The second command is
aplay -f S16_LE -r 22050 -c 1 out_file.pcm
This should help:
pushd /path/to/directory
my_var=$(command1)
if [ "$my_var" == "expected_result" ]; then
command2
fi
popd
You basically run command1 and store its output in my_var. Then you compare the content of $my_var with whatever you're expecting.
Also pushd <path>/popd allow you to move to a directory and back.
How could I pipe the result from a which command to cd?
This is what I am trying to do:
which oracle | cd
cd < which oracle
But none of them works.
Is there a way to achieve this (rather than copy/paste of course)?
Edit : on second thought, this command would fail, because the destination file is NOT a folder/directory.
So I am thinking and working out a better way to get rid of the trailing "/oracle" part now (sed or awk, or even Perl) :)
Edit :
Okay that's what I've got in the end:
cd `which oracle | sed 's/\/oracle//g'`
You use pipe in cases where the command expects parameters from the standard input. ( More on this ).
With cd command that is not the case. The directory is the command argument. In such case, you can use command substitution. Use backticks or $(...) to evaluate the command, store it into variable..
path=`which oracle`
echo $path # just for debug
cd $path
although it can be done in a much simpler way:
cd `which oracle`
or if your path has special characters
cd "`which oracle`"
or
cd $(which oracle)
which is equivalent to backtick notation, but is recommended (backticks can be confused with apostrophes)
.. but it looks like you want:
cd $(dirname $(which oracle))
(which shows you that you can use nesting easily)
$(...) (as well as backticks) work also in double-quoted strings, which helps when the result may eventually contain spaces..
cd "$(dirname "$(which oracle)")"
(Note that both outputs require a set of double quotes.)
With dirname to get the directory:
cd $(which oracle | xargs dirname)
EDIT: beware of paths containing spaces, see #anishpatel comment below
cd `which oracle`
Note those are backticks (generally the key to the left of 1 on a US keyboard)
OK, here a solution that uses correct quoting:
cd "$(dirname "$(which oracle)")"
Avoid backticks, they are less readable, and always quote process substitutions.
You don't need a pipe, you can do what you want using Bash parameter expansion!
Further tip: use "type -P" instead of the external "which" command if you are using Bash.
# test
touch /ls
chmod +x /ls
cmd='ls'
PATH=/:$PATH
if cmdpath="$(type -P "$cmd")" && cmdpath="${cmdpath%/*}" ; then
cd "${cmdpath:-/}" || { echo "Could not cd to: ${cmdpath:-/}"; exit 1; }
else
echo "No such program in PATH search directories: ${cmd}"
exit 1
fi
besides good answer above, one thing needs to mention is that cd is a shell builtin, which run in the same process other than new process like ls which is a command.
https://unix.stackexchange.com/questions/50022/why-cant-i-redirect-a-path-name-output-from-one-command-to-cd
http://en.wikipedia.org/wiki/Shell_builtin
In response to your edited question, you can strip off the name of the command using dirname:
cd $(dirname `which oracle`)
Cron installation is vixie-cron
/etc/cron.daily/rmspam.cron
#!/bin/bash
/usr/bin/rm /home/user/Maildir/.SPAM/cur/*;
I Have this simple bash script that I want to add to a cron job (also includes spam learning commands before) but this part always fails with "File or directory not found" From what I figure is the metachar isn't being interperted correctly when run as a cron job. If I execute the script from the commandline it works fine.
I'd like a why for this not working and of course a working solution :)
Thanks
edit #1
came back to this question when I got popular question badge for it. I first did this,
#!/bin/bash
find /home/user/Maildir/.SPAM/cur/ -t file | xargs rm
and just recently was reading through the xargs man page and changed it to this
#!/bin/bash
find /home/user/Maildir/.SPAM/cur/ -t file | xargs --no-run-if-empty rm
short xargs option is -r
If there are no files in the directory, then the wildcard will not be expanded and will be passed to the command directly. There is no file called "*", and then the command fails with "File or directory not found." Try this instead:
if [ -f /home/user/Maildir/.SPAM/cur/* ]; then
rm /home/user/Maildir/.SPAM/cur/*
fi
Or just use the "-f" flag to rm. The other problem with this command is what happens when there is too much spam for the maximum length of the command line. Something like this is probably better overall:
find /home/user/Maildir/.SPAM/cur -type f -exec rm '{}' +
If you have an old find that only execs rm one file at a time:
find /home/user/Maildir/.SPAM/cur -type f | xargs rm
That handles too many files as well as no files. Thanks to Charles Duffy for pointing out the + option to -exec in find.
Are you specifying the full path to the script in the cronjob?
00 3 * * * /home/me/myscript.sh
rather than
00 3 * * * myscript.sh
On another note, it's /bin/rm on all of the linux boxes I have access to. Have you double-checked that it really is /usr/bin/rm on your machine?
try adding
MAILTO=your#email.address
to the top of your cron file and you should get any input/errors mailed to you.
Also consider adding the command as a cronjob
0 30 * * * /usr/bin/rm /home/user/Maildir/.SPAM/cur/*
Try using a force option and forget about adding a path to rm command. I think it should not be needed...
rm -f
This will ensure that even if there are no files in the directory, rm command will not fail. If this is a part of a shell script, the * should work. It looks to me that you might have an empty dir...
I understand that the rest of the script is being executed, right?
Is rm really located in /usr/bin/ on your system? I have always thought that rm should reside in /bin/.