I feel silly asking this...
I am not an expert on shell scripting, but I am finally in enough of a sysadmin role that I want to do this correctly.
I have a production server that hosts a webapp. Here is my routine.
1 - ssh to server
2 - cd django_src/django_apps/team_proj
3 - svn update
4 - sudo /etc/init.d/apache2 restart
5 - logout
I want to create a shell script for steps 2,3,4.
I can do this, but it will be a very plain and simple bash script simply containing the actual commands I type at the command line.
My question: What is the best way to script this kind of repetitive procedure in bash (Linux, Ubuntu) for a remote server?
Thanks!
The best way is simply as you suggest. Some things you should do for your script would be:
put set -e at the top of the script (after the shebang). This will cause your script to stop if any of the commands fail. So if it cannot cd to the directory, it will not run svn update or restart apache. You can do this programmatically by putting || exit 0 after each command, but if that's all you're doing, you may as well use set -e
Use full paths in your script. Do not assume the directory that the script is run from. In this specific case, the cd command has a relative path. Use a full (absolute) path, or use an environment variable like $HOME.
You may want to set up sudo so that it can run the command without asking for a password. This makes your script non-interactive which means it can be run in the background and from cron jobs and such.
As time goes by, you may add features and take command line arguments to parameterise the script. But don't bother doing this up front. Just evolve your scripts as you need.
There is nothing wrong with a simple bash script simply containing the actual commands you type at the command line. Don't make it more complicated than necessary.
I'd setup a cron job doing that automatically.
Since you're using python, check out fabric - you can use it to automate these kind of tasks. First install fabric:
$ sudo easy_install fabric
then write your fabric script:
from __future__ import with_statement
from fabric.api import *
def svnupdate():
with cd('django_src/django_apps/team_proj'):
run('svn update')
sudo('/etc/init.d/apache2 restart')
Save as fabfile.py, then run using the fab command:
$ fab -H hostname svnupdate
Tell me that's not cool! :-)
you can do this with the shell (bash,ksh,zsh + ssh + tools), or programming languages such as Python,Perl(Ruby or PHP or Java) etc, basically a language that supports SSH protocol and operating system functions. The "best" one is the one that you are more comfortable and have knowledge in. If you are doing sysadmin, the shell is the closest thing you can use. Then after you have done your script, you can use the crontab (cron) , or the at command to schedule your task. check their man page for more information
You can easily do the above using bash/Bourne etc.
However I would take the time and effort to learn Perl (or some similarly powerful scripting language). Why ?
the language constructs are much more powerful
there are no end of libraries to interface to the systems/features you want to script
because of the library support, you won't have to spawn off different commands to achieve what you want (possibly valuable on a loaded system)
you can decompose frequently-used scripts into your own libraries for later use
I choose Perl particularly because it's been designed (perhaps designed is too strong a word for Perl) for these sort of tasks. However you may want to check out Ruby/Python or other suggestions from SO contributers.
For the basic steps look at camh's answer. If you plan to run the script via cron, then implement some simple logging, e.g. by appending start time of each command with exit code to a textfile which you can later analyze for failures of the script.
Expect -- scripting interactive applications
Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.... Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily.
http://expect.nist.gov
bonus: Your tax dollars at work!
I would probably do something like this...
project_update.sh
#!/bin/bash
#
# $1 - user#host
# $2 - project directory
[[ -z $1 || -z $2 ]] && { echo "usage: $(basename $0) user#host project_dir"; exit 1; }
declare host=$1 proj_dir=$2
ssh $host "cd $proj_dir;svn update;sudo /etc/init.d/apache2 restart" && echo "Success"
Just to add another tip - you should not give users access to some application in an unknown state. svn up might break during the update, users might see a page that's half-new half-old, etc. If you're deploying the whole application at once, I'd suggest doing svn export instead to a new directory and then either mv current old ; mv new current, or even keeping current as a link to the directory you're using now. Still not perfect and not blocking every possible race condition, but it definitely takes less time than svn up on the live copy.
Related
Quick question regarding this error in Bash. I have some bash scripts that I run at work on some servers. We have been trying to implement Rundeck as a way to automate. When I call these scripts through Rundeck I get the error bash: no job control in this shell. Now I am new to linux and shell scripting but what I have figured is this is due to interactivity of the shell. The scripts I am calling will call other scripts (perl and shell) as part of their operation. But since I am not logged on in a terminal, they can't do this and fail. That is the best understanding I have.
I have tried adding a -l to my #!/bin/bash in my script I send through Rundeck. I have also tried to utilize the ad-hoc command option through Rundeck and run the jobs individually. Still not luck. I thought perhaps it was a pathing issue so I tired setting the path PATH=$PATH but no change.
Basically I need to allow these scripts to open their subprocesses. So the question is, how I modify how I call these scripts so they have full control? Is that possible? Sorry if this lacks info, I just don't know how properly put it out there. let me know what other info is needed. Thanks
Edit: Some code snips on where the message comes in:
system "pntadm -R $Net >/dev/null 2>&1";
system "pntadm -C $Net $BUILD{$Net}{netmask} $MYip";
I have a Linux computer and want to do the following (two independent actions/processes if possible):
Run a series of CLI commands every x minutes towards a remote computer to see if a file exists there. If it does, I want to start downloading this file to one of my directories. The files could be big and there might be many remote computers, so ideally I should treat each connection as an own process.
Check to see if a new file has arrived in my file system. If it has, I want to go through this file, analyze the content with some algoritms and then store the result in a database that I have installed. Then delete the file that was analyzed.
Any recommendations on how to do this the "best" and most reliable way? Scripting? Java/C/etc? Multi threading or just a single process that is looping through the content? The result should be something that should run for months without stopping.
Any suggestions and/or sample code very welcome!
Thanks!
Z
For #1, you can use crantab(http://unixhelp.ed.ac.uk/CGI/man-cgi?crontab+5) to run your script.
I think you can use Shell, Python, Ruby to complete your tasks
for downloading you can use a mix of something like that script (not tested and incomplete):
function download_and_analyze(link){
Z=mktemp
cd $Z
wget -c -t0 -q -C $link
# analyze algorithm here
cd /tmp
rm -rf $Z
}
for A in $REMOTEFILE; do
download_and_analyze($A) &
done
this is just a scratch to you implement it as a shell script. Realiability is guaranteed by wget.
you also can use rsync if it's on another computer acessible by ssh.
cheers
Some weeks ago, a senior team member removed an important oracle database file(.dbf) unexpectedly. Fortunately, We could restore the system by using back-up files which was saved some days ago.
After seeing that situation, I decided to implement a solution to make atleast a double confirmation when typing rm command on the prompt. (checks more than rm -i)
Even though we aliasing rm -i as default, super speedy keyboardists usually make mistakes like that member, including me.
At first, I replaced(by using alias) basic rm command to a specific bash script file which prints and confirms many times if the targets are related on the oracle database paths or files.
simply speaking, the script operates as filter before to operate rm. If it is not related with oracle, then rm will operate as normal.
While implementing, I thought most of features are well operated as I expected only user prompt environment except one concern.
If rm command are called within other scripts(provided oracle, other vendor modifying oracle path, installer, etc) or programs(by using system call).
How can i distinguish that situation?
If above provided scripts met modified rm, That execution doesn't go ahead anymore.
Do you have more sophisticated methods?
I believe most of reader can understand my lazy explanation.
If you couldn't get clear scenery from above, let me know. I will elaborate more.
We read at man bash:
Aliases are not expanded when the shell is not interactive, unless the
expand_aliases shell option is set using shopt.
Then if you use alias to make rm invoke your shell script, other scripts won't use it by default. If it's what you want, then you're already safe.
The problem is if you want your version of rm to be invoked by scripts and do something smart when it happens. Alias is not enough for the former; even putting your rm somewhere under $PATH is not enough for programs explicitly calling /bin/rm. And for programs that aren't shell scripts, unlink system call is much more likely to be used than something like system("rm ...").
I think that for the whole "safe rm" thing to be useful, it should avoid prompts even when invoked interactively. Every user will develop the habit of saying "yes" to it, and there is no known way around that. What might work is something that moves files to recycle bin instead of deletion, making damage easy to undo (as I seem to recall, there were ready to use solutions for this).
The answer is into the alias manpage:
Note aliases are not expanded by default in non-interactive
shell, and it can be enabled by setting the expand_aliases shell
option using shopt.
Check it by yourself with man alias ;)
Anyway, i would do it in the same way you've chosen
To distinguish the situation: You can create an env variable say, APPL, which will be set to say export APPL="DATABASE . In your customized rm script, perform the double checkings only if the APPL is DATABASE (which indicates a database related script), not otherwise which means the rm call is from other scripts.
If you're using bash, you can export your shell function, which will make it available in scripts, too.
#!/usr/bin/env bash
# Define a replacement for `rm` and export it.
rm() { echo "PSYCH."; }; export -f rm
Shell functions take precedence over builtins and external utilities, so by using just rm even scripts will invoke the function - unless they explicitly bypass the function by invoking /bin/rm ... or command rm ....
Place the above (with your actual implementation of rm()) either in each user's ~/.bashrc file, or in the system-wide bash profile - sadly, its location is not standardized (e.g.: Ubuntu: /etc/bash.bashrc; Fedora /etc/bashrc)
I am trying to create a script that will run a program on each file in a list. I have been trying to do this using a .csh file (I have no clue if this is the best way), and I started with something as simple as hello world
echo "hello world"
The problem is that I cannot execute this script, or verify that it works correctly. (I was trying to do ./testscript.csh which is obviously wrong). I haven't been able to find anything that really explains how to run C Scripts, and I'm guessing there's a better way to do this too. What do I need to change to get this to work?
You need to mark it as executable; Unix doesn't execute things arbitrarily based on extension.
chmod +x testscript.csh
Also, I strongly recommend using sh or bash instead of csh, or you will soon learn about the idiosyncrasies of csh's looping and control flow constructs (some things only work inside them if done a particular way, in particular with the single-line versions things are very limited).
You can use ./testscript.csh. You will however need to make it executable first:
chmod u+x testscript.csh
Which means set testscript to have execute permissions for the user (who ever the file is owned by - which in this case should be yourself!)
Also to tell the OS that this is a csh script you will need put
#! /path/to/csh
on the first line (where /path/to/csh is the full path to csh on your system. You can find that out by issuing the command which csh).
That should give you the behvaiour you want.
EDIT As discussed in some of the comments, you may want to choose an alternative shell to C Shell (csh). It is not the friendliest one for scripting.
You have several options.
You can run the script from within your current shell. If you're running csh or tcsh, the syntax is source testscript.csh. If you're running sh, bash, ksh, etc., the syntax is . ./testscript.sh. Note that I've changed the file name suffix; source or . runs the commands in the named file in your current shell. If you have any shell-specific syntax, this won't work unless your interactive shell matches the one used by the script. If the script is very simple (just a sequence of simple commands), that might not matter.
You can make the script an executable program. (I'm going to repeat some of what others have already written.) Add a "shebang" as the first line. For a csh script, use #!/bin/csh -f. The -f avoids running commands in your own personal startup scripts (.cshrc et al), which saves time and makes it more likely that others will be able to use it. Or, for a sh script (recommended), used #!/bin/sh (no -f, it has a completely different meaning). In either case, run chmod +x the_script, then ./the_script.
There's a trick I often use when I want to perform some moderately complex action. Say I want to delete some, but not all, files in the current directory, but the criterion can't be expressed conveniently in a single command. I might run ls > tmp.sh, then edit tmp.h with my favorite editor (mine happens to be vim). Then I go through the list of files and delete all the ones that I want to leave alone. Once I've done that, I can replace each file name with a command to remove it; in vim, :%s/.*/rm -f &/. I add a #!/bin/sh at the top save it, chmod +x foo.sh, then ./foo.sh. (If some of the file names might have special characters, I can use :%s/.*/rm -f '&'/.)
Problem: Customer X is a Windows user who wants to be able to trigger pre-packaged bash commands by using mnemonic keywords or "tag hints" when she is logged in to her RedHat box via shell.
Example: Customer X logs into host using ssh and wants to do some routine file operations. She wants to be able to type
copy file
and get back a listing of pre-fab fill-in-the-blank bash commands to choose from
cp <#source#> <#dest#> ### simple copy
cp -R <#startdir#> <#destdir#> ### recursive copy
she then wants to be able to select one of these items, fill in the blank(s) and just hit enter to run the command.
Customer X is willing to specify ahead of time what commands she is likely to want to use (in windows-speak) and then hire the developer to translate those into bash commands, and then put them together in a script that allows her to talk windows-speak to bash and get back the list of commands.
NOTE: Customer X doesn't like apropos because it assumes familiarity with terms used in bash, as opposed to windows-speak. For example:
apropos shortcut
doesn't give her anything about creating symlinks (even though that is exactly what she wants) because she doesn't know what windows shortcuts are called in linux. Obviously, windows concepts don't carry over 100% so she will have to learn eventually, but she's a busy person and is asking for this as a way to "ease" her into linux understanding.
Question: What is the best way to get started on something like this? Is there a perl, python, ruby script out there that does something like this already? Is there something in bash that can simulate this kind of feature request?
What you probably want is to override bash's command-not-found handler. Here's the section in /etc/bash.bashrc in a standard Ubuntu install that installs the handler:
...
# if the command-not-found package is installed, use it
if [ -x /usr/lib/command-not-found ]; then
function command_not_found_handle {
# check because c-n-f could've been removed in the meantime
if [ -x /usr/lib/command-not-found ]; then
/usr/bin/python /usr/lib/command-not-found -- $1
return $?
else
return 127
fi
}
fi
...
In effect, if a command is not found, a user specified program is executed with that command as a parameter. In the case of Ubuntu, it's a Python program that checks to see if the command the user typed is a valid application that can be installed, and if it is, informs the user that he/she can install it.
What you probably want to do is compare it to you hashref of commands and usage strings and display the appropriate one if there's a match.