I'm writing a bash script to grab an archive (specifically, the gcc-4.9.1 source) from ftp.gnu.org, which it does using a here document. I'd also like to direct ftp's output to a log file, in order to make the script's output cleaner while keeping any information that might be needed. I'd also like to use bash's double-bar to run an error-catching function that prints some output and exits if ftp returns unsuccessfully.
#get the archive
echo "getting the archive from ftp://${HOST}/${FTPPATH}${ARCHIVE}"
/usr/bin/ftp -inv $HOST > ${VERNAME}_logs/ftp.log || errorcatcher "failed to connect to ftp server, check ${VERNAME}_logs/ftp.log" <<FTPSCRIPT
user anonymous
cd $FTPPATH
binary
get $ARCHIVE
FTPSCRIPT
The problem is, this hangs, and ftp.log looks like this:
Trying 208.118.235.20...
Connected to ftp.gnu.org (208.118.235.20).
220 GNU FTP server ready.
ftp>
ftp>
ftp>
The commands are clearly not getting passed to the ftp client, and I imagine either the output redirection or the double pipes are causing this, as without them both the script successfully gets the archive.
Is there any syntax that allows me to pass interactive commands to ftp while still allowing output redirection and conditional execution following the return?
You are feeding the wrong command with the here document.
/usr/bin/ftp [ ...args... ] || errorcatcher [...] <<FTPSCRIPT
should be
/usr/bin/ftp [ ...args... ] <<FTPSCRIPT || errorcatcher [...]
The contents of the here document do not begin until the line following the <<. You could even write
/usr/bin/ftp -inv $HOST > ${VERNAME}_logs/ftp.log <<FTPSCRIPT ||
user anonymous
cd $FTPPATH
binary
get $ARCHIVE
FTPSCRIPT
errorcatcher "failed to connect to ftp server, check ${VERNAME}_logs/ftp.log"
if you find that more readable (I'm not sure that I do, but it's an option), or also
/usr/bin/ftp -inv $HOST > ${VERNAME}_logs/ftp.log <<FTPSCRIPT \
|| errorcatcher "failed to connect to ftp server, check ${VERNAME}_logs/ftp.log"
user anonymous
cd $FTPPATH
binary
get $ARCHIVE
FTPSCRIPT
Related
What is a good way to be able to directly send to STDIN and receive from STDOUT of a process? I'm specifically interested in SSH, as I want to do the following:
[ssh into a remote server]
[run remote commands]
[run local commands]
[run remote commands]
etc...
For example, let's say I have a local script "localScript" that will output the next command I want to run remotely, depending on the output of "remoteScript". I could do something like:
output=$(ssh myServer "./remoteScript")
nextCommand=$(./localScript $output)
ssh myServer "$nextCommand"
But it would be nice to do this without closing/reopening the SSH connection at every step.
You can redirect SSH input and output to FIFO-s and then use these for two-way communication.
For example local.sh:
#!/bin/sh
SSH_SERVER="myServer"
# Redirect SSH input and output to temporary named pipes (FIFOs)
SSH_IN=$(mktemp -u)
SSH_OUT=$(mktemp -u)
mkfifo "$SSH_IN" "$SSH_OUT"
ssh "$SSH_SERVER" "./remote.sh" < "$SSH_IN" > "$SSH_OUT" &
# Open the FIFO-s and clean up the files
exec 3>"$SSH_IN"
exec 4<"$SSH_OUT"
rm -f "$SSH_IN" "$SSH_OUT"
# Read and write
counter=0
echo "PING${counter}" >&3
cat <&4 | while read line; do
echo "Remote responded: $line"
sleep 1
counter=$((counter+1))
echo "PING${counter}" >&3
done
And simple remote.sh:
#!/bin/sh
while read line; do
echo "$line PONG"
done
The method you are using works, but I don't think you can reuse the same connection everytime. You can, however, do this using screen, tmux or nohup, but that would greatly increase the complexity of your script because you will now have to emulate keypresses/shortcuts. I'm not even sure if you can if you do directly in bash. If you want to emulate keypresses, you will have to run the script in a new x-terminal and use xdotool to emulate the keypresses.
Another method is to delegate the whole script to the SSH server by just running the script on the remote server itself:
ssh root#MachineB 'bash -s' < local_script.sh
I am doing an ftp and I want to check the status. I don't want to use '$?' as mostly it returns 0 (Success) for ftp even though internally ftp didn't go through.
I know I can check the log file and do a grep from there for "Transfer complete" (221 status). That works fine but I don't want to do it as I have many different reports doing ftp. So creating multiple log files for all of them is what I want to avoid.
Can I get the logged information in a local script variable and process it inside the script itself?
Something similar to these (I've tried both but neither worked):
Grab FTP output in BASH SCRIPT
FTP status check whether successful or not
Below is something similar to what I am trying to do:
ftp -inv ${HOST} > log_file.log <<!
user ${USER} ${PASS}
bin
cd "${TARGET}"
put ${FEEDFILE}
bye
!
Any suggestions on how can I get the entire ftp output in a script variable and then check it within the script?
To capture stdout to a variable you can use bash's command substitution, so either OUTPUT=`cmd` or OUTPUT=$(cmd).
Here's an example how to capture the output from ftp in your case:
CMDS="user ${USER} ${PASS}
bin
cd \"${TARGET}\"
put \"${FEEDFILE}\"
bye"
OUTPUT=$(echo "${CMDS}" | ftp -inv ${HOST})
I have a small problem regarding "sftp".
I have a script, which simply transfers a file to a remote sftp server. But when this script runs it fails at sftp and my script fails.
So, i have to manually transfer the file,using command which is same as the command that i have used in the script, and it works fine.
So my problem is that the sftp command runs smoothly when i run it manually, but creates problem when the same command is run through the script.
this is the code that I'm using
sftp -v -b sftp_input.txt UserId#aa.bb.cc.dd
if (($? > 0 ));
then
echo "sftp error. Exiting.."
exit
fi
where sftp_input.txt contains the cmd to put the file to remote server.
Please advice.....
The script can't work because it's malformed. You forgot to separate the if statement and also forgot the closing fi. Here's the correct form for your script:
sftp -v -b sftp_input.txt UserId#aa.bb.cc.ddd
if (($? > 0 )); then
echo "sftp error. Exiting.."
exit
fi
If you want it all in one line, then:
sftp -v -b sftp_input.txt UserId#aa.bb.cc.ddd; if (($? > 0 )); then echo "sftp error. Exiting.."; exit; fi
But as you can see it's a bad idea. Better to write readable and well indented code.
I have a bash script that creates backups incrementally(daily) and full(on Mondays). Every 7 days the script combines the week of backups(full and Incremental) and sends them off to an FTP server, the problem i am having is i want to delete the files from my backup directory after the FTP upload is finished, but i cant do that until i know the file was successfully uploaded. I need to figure out how to capture the '226 transfer complete' so i can use that in an 'IF' statement to delete the backup files. Any help is greatly appreciated. also he is my FTP portion of the script
if [ -a "$WKBKDIR/weekending-$YEAR-$MONTH-$DAY.tar.gz" ]; then
HOST=192.168.40.30 #This is the FTP servers host or IP address.
USER=username #This is the FTP user that has access to the server.
PASS=password #This is the password for the FTP user.
ftp -inv $HOST << EOF
user $USER $PASS
cd /baks
lcd $WKBKDIR
put weekending-$YEAR-$MONTH-$DAY.tar.gz
bye
EOF
fi
I could use whatever mean i needed i suppose, FTP was something already setup for another backup function for something else, thanks
2nd EDIT Ahmed the rsync works great in test from command line, its a lot faster than FTP, the server is on the local network so SSH not that big of a deal but nice to have for added security, i will finish implementing in my script tomorrow, thanks again
FTP OPTION
The simple solution would be to do something like this:
ftp -inv $HOST >ftp-results-$YEAR-$MONTH-$DAY.out 2>&1 <<-EOF
user $USER $PASS
cd /baks
bin
lcd $WKBKDIR
put weekending-$YEAR-$MONTH-$DAY.tar.gz
bye
EOF
Also there is an issue with your here-document syntax; there is no space between << and the delimiter word (in your case EOF) and I added a - because you are putting white-spaces before the ACTUAL delimeter (it's tabbed in for the if / fi block) so the [-] is required
Now when you do this; you can parse the output file to look for the successful put of the file. For example:
if grep -q '226 transfer complete' ftp-results-$YEAR-$MONTH-$DAY.out; then
echo "It seems that FTP transfer completed fine, we can schedule a delete"
echo "rm -f $PWD/weekending-$YEAR-$MONTH-$DAY.tar.gz" >> scheduled_cleanup.sh
fi
and just run scheduled_cleanup.sh using cron at a given time; this way you will have some margin before the files are cleaned up
If your remote FTP server has good SITE or PROXY options you may be able to get the remote FTP to run a checksum on the uploaded file after successful upload and return the result.
SCP / RSYNC OPTION
Using FTP is clunky and dangerous, you should really try and see if you can have scp or ssh access to the remote system.
If you can then generate an ssh key if you don't have one using ssh-keygen:
ssh-keygen -N "" -t rsa -f ftp-rsa
put the ftp-rsa.pub file into the $HOST/home/$USER/.ssh/authorized_keys and you have a much nicer method for uploading files:
if scp -B -C weekending-$YEAR-$MONTH-$DAY.tar.gz $USER#$HOST:/baks/ ; then
echo Upload successful 1>&2
else
echo Upload failed 1>&2
fi
Or better yet using rsync:
if rsync --progress -a weekending-$YEAR-$MONTH-$DAY.tar.gz $HOST:/baks/ ; then
echo Upload successful 1>&2
else
echo Upload failed 1>&2
fi
et voilà you are done since rsync works over ssh you are happy and secure
Try the next
#!/bin/bash
runifok() { echo "will run this when the transfer is OK"; }
if [ -a "$WKBKDIR/weekending-$YEAR-$MONTH-$DAY.tar.gz" ]; then
HOST=192.168.40.30 #This is the FTP servers host or IP address.
USER=username #This is the FTP user that has access to the server.
PASS=password #This is the password for the FTP user.
ftp -inv <<EOF | grep -q '226 transfer complete' && runifok
user $USER $PASS
cd /baks
lcd $WKBKDIR
put weekending-$YEAR-$MONTH-$DAY.tar.gz
bye
EOF
fi
test it and when will run ok - replace the echo in the runifok function for your commands what want execute after the upload is succesful.
I'm using simple script to automate ftp. The script looks like this:
ftp -nv $FTP_HOST<<END_FTP
user $FTP_USER $FTP_PASS
binary
mkdir $REMOTE_DIR
cd $REMOTE_DIR
lcd $LOCAL
put $FILE
bye
END_FTP
But I would like to pipe STDERR to the syslog and STDOUT to a logfile. Normally I would do something like that: ftp -nv $FTP_HOST 1>>ftp.log | logger<<END_FTP but in this case that won't work because of <<END_FTP. How should I do it properly to make the script work? Note that I want to redirect only output from the FTP command inside my script and not the whole script.
This works without using a temp file for the error output. The 2>&1 sends the error output to where standard output is going — which is the pipe. The >> changes where standard output is going — which is now the file — without changing where standard error is going. So, the errors go to logger and the output to ftp.log.
ftp -nv $FTPHOST <<END_FTP 2>&1 >> ftp.log | logger
user $FTP_USER $FTP_PASS
binary
mkdir $REMOTE_DIR
cd $REMOTE_DIR
lcd $LOCAL
put $FILE
bye
END_FTP
How about:
exec > mylogfile; exec 2> >(logger -t myftpscript)
in front of you ftp script
Another way of doing this I/O redirection is with the { ... } operations, thus:
{
ftp -nv $FTPHOST <<END_FTP >> ftp.log
user $FTP_USER $FTP_PASS
binary
mkdir $REMOTE_DIR
cd $REMOTE_DIR
lcd $LOCAL
put $FILE
bye
END_FTP
# Optionally other commands here...stderr will go to logger too
} 2>&1 | logger
This is often the best mechanism when more than one command, but not all commands, need the same I/O redirection.
In context, though, I think this solution is the best (but that's someone else's answer, not mine):
ftp -nv $FTPHOST <<END_FTP 2>&1 >> ftp.log | logger
...
END_FTP
Why not create a netrc file and let that do your login and put the file for you.
The netrc file will let you login and allow you to define an init macro that will make the needed directory and put the file you want over there. Most ftp commands let you specify which netrc file you'd like to use, so you could use various netrc files for various purposes.
Here's an example netrc file called my_netrc:
machine ftp_host
user ftp_user
password swordfish
macrodef init
binary
mkdir my_dir
cd my_dir
put my_file
bye
Then, you could do this:
$ ftp -v -Nmy_netrc $FTPHOST 2>&1 >> ftp.log | logger